From yarivsadan@REDACTED Sun Jun 1 00:04:56 2008 From: yarivsadan@REDACTED (Yariv Sadan) Date: Sat, 31 May 2008 15:04:56 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> Message-ID: <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> On Fri, May 30, 2008 at 7:18 AM, Joe Armstrong wrote: > Hi Yariv > > Well done - now a few questions: > > - what is Twitter? and why is it good (if it is good) - can you > explain in a couple of paragraphs what it does? It seemed to be some > kind of new way for interrupting people ... > > (this is from one who thinks that e-mail is intrusive and whose > mobile phone is usually > turned off :-) Twitter is a micro-blogging service. It allows asynchronous broadcasting of casual messages/questions/observations/brain-farts to your friends. It's not intrusive like IM because it's asynchronous. It's more like a public mailing list where you only see the messages from people you choose to follow. At first I didn't get Twitter, but once I started using it I realized it's kinda fun. > > - the reaction to your work seems to be (oh but it doesn't scale - > so Erlang must be no good - > despite the fact that I believe you expressly said it was a quick > hack and wasn't designed to > scale) What can ya do? :) > > - I might be nice to say "it doesn't scale YET" - and then sit back > and let us help > you make it VERY scalable. I like a good challenge. > > Could you try to describe exactly what Twitter is (in an as > abstract way as possible) so that old > fogies like me can wrap our brains around the problem of making it scale - > > So what does it do? How many things must it scale to (give me > numbers here - what are > talking about - scaling is not some abstract quantity - it's number of > bytes of data delivered to end-users > per gram of CO2 - how many bytes/gm of CO2 are we aiming at?) > > (aside) What does scalable mean? - I suppose the answer is some > constant cost per user. > > A more literal answer would be "doesn't break when we add more users" > - but surely the cost > verses number of users curve must be more important. > > In the context of Twitter what does the word "scalable mean" (how > about, we can handle > 100,000 users per "box" - each box costs 500$ and consumes 25Watts - > this is constant up to > 6x10^9 users) > > So if anybody who knows about this stuff can chip in with a few > figures it would be helpful > - what are the desired costs for running twitter for a few million > people - how would you like > this to scale > > (/aside) > > > /Joe Armstrong Here's how I userstand the main scaling challenge of a Twitter-like service: You have M users and each user follows an average of N other users. To render a user's timeline, you need to present the latest 20 tweets (twoorls in the new terminology :) ) from all his friends. The naive way to do it is to fetch N*20 messages, sort them, then take the first 20. This is expensive to do when N is big. Another problem is that because of the frequent updates from typical users, caching this calculation doesn't give you much because the cache gets frequently invalidated between page loads. The only solution I see is to keep a "latest twoorls" store for each user. When a friend sends a new twoorl, it (or a pointer to it) is copied to the twoorl store for all his friends. This trades space for speed, and it limits the paging you can do to see your twoorl history, but it's the only way you can scale this kind of service when N is big. Yariv From tsuraan@REDACTED Sun Jun 1 00:52:49 2008 From: tsuraan@REDACTED (tsuraan) Date: Sat, 31 May 2008 17:52:49 -0500 Subject: [erlang-questions] ioctl support? Message-ID: <84fb38e30805311552p33ef95ddxf62381b7cdde742e@mail.gmail.com> Is the UNIX ioctl call exposed anywhere in the erlang standard library? It doesn't seem to be in the file package, and I can't think of where else it would be. From matthew@REDACTED Sun Jun 1 01:00:40 2008 From: matthew@REDACTED (Matthew Dempsky) Date: Sat, 31 May 2008 16:00:40 -0700 Subject: [erlang-questions] ioctl support? In-Reply-To: <84fb38e30805311552p33ef95ddxf62381b7cdde742e@mail.gmail.com> References: <84fb38e30805311552p33ef95ddxf62381b7cdde742e@mail.gmail.com> Message-ID: On Sat, May 31, 2008 at 3:52 PM, tsuraan wrote: > Is the UNIX ioctl call exposed anywhere in the erlang standard > library? No. You'll probably need to write an Erlang driver or an external application to interface with. From pfisher@REDACTED Sun Jun 1 01:04:17 2008 From: pfisher@REDACTED (Paul Fisher) Date: Sat, 31 May 2008 18:04:17 -0500 Subject: [erlang-questions] ioctl support? In-Reply-To: <84fb38e30805311552p33ef95ddxf62381b7cdde742e@mail.gmail.com> References: <84fb38e30805311552p33ef95ddxf62381b7cdde742e@mail.gmail.com> Message-ID: <1212275057.6409.138.camel@localhost> On Sat, 2008-05-31 at 17:52 -0500, tsuraan wrote: > Is the UNIX ioctl call exposed anywhere in the erlang standard > library? It doesn't seem to be in the file package, and I can't think > of where else it would be. No. What are you trying to achieve? Most style of processing are possible because of the structure of the erlang environment or as part of specific library interfaces. -- paul From jeremy@REDACTED Sun Jun 1 02:01:26 2008 From: jeremy@REDACTED (Jeremy Wall) Date: Sat, 31 May 2008 19:01:26 -0500 Subject: [erlang-questions] A very rough start at an rss parsing module. Message-ID: <69d143cd0805311701h3914c2ob20527eff7729866@mail.gmail.com> It only does rss 2.0 right now and it doesn't support any of the optional elements yet but then it's only a days worth of work. I'm interested in any critique folks might have. I'm still just starting with erlang so I'm sure there are better ways to do some of this. -- Jeremy Wall http://jeremy.marzhillstudios.com Jeremy@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: rss.erl Type: application/octet-stream Size: 1724 bytes Desc: not available URL: From sean.hinde@REDACTED Sun Jun 1 02:19:44 2008 From: sean.hinde@REDACTED (Sean Hinde) Date: Sun, 1 Jun 2008 01:19:44 +0100 Subject: [erlang-questions] eep: multiple patterns In-Reply-To: <4841890B.7060101@san.rr.com> References: <200805141620.19559.bekesa@sch.bme.hu> <4836A517.2040909@gmail.com> <200805271436.19305.bekesa@sch.bme.hu> <6BA6A9E9-EF89-4FE6-B683-05F3866E9E09@cs.otago.ac.nz> <483D0CB6.2080103@ericsson.com> <483FF500.3020101@ericsson.com> <4841890B.7060101@san.rr.com> Message-ID: On 31 May 2008, at 18:21, Darren New wrote: > Sean Hinde wrote: >> I had thought that there was an optimisation where if the pattern >> was identical to last time the same receive was entered for that >> process then only newer messages were examined. > > I think that's only if you get multiple messages arriving without > actually triggering a receive. > >> Seems like it could be a useful optimisation in any case :-) > > If the guards didn't change and you didn't run any other receive > statements, maybe one could detect that. Figuring that out at > compile time would seem to be pretty brutal, but noticing which > receive statement ran last and keeping a list of all the guard > values at run time might work. Probably easier to just say "don't do > that." :-) If the patterns to be matched in the receive clause are all constants or unbound variable the compiler should certainly be able to detect that. In this (rather common) case it could generate special code to some funky new instructions that save the position in the message queue and skip all the unnecessary scanning. It could also as you say be done at runtime, but with some overhead. It seems on the surface to be quite do-able.. This problem (of poor performance with selective receive) seems like one of the nastiest (and least well understood ;-) ) gotchas of the current runtime. I hope while we consider tweaking the syntax we can also find time to optimise the behaviour. Sean From dnew@REDACTED Sun Jun 1 01:15:07 2008 From: dnew@REDACTED (Darren New) Date: Sat, 31 May 2008 16:15:07 -0700 Subject: [erlang-questions] Performance of matches In-Reply-To: <484126E2.5040000@adam.com.au> References: <483EDB2E.9070103@san.rr.com> <484126E2.5040000@adam.com.au> Message-ID: <4841DBFB.2000308@san.rr.com> David Lloyd wrote: >> Is either of those O(1) lookup time? > It strikes me that this is something that is eminently easy to profile... Good point. Altho compiling a function matching against one of 100,000 atoms did take 20 minutes. :-) It looks like even matching atoms[*] takes O(lg n) time. I'll have to see if ets does any better. [*] func(x1) -> 1; func(x2) -> 2; ... (Sorry you got this twice, David.) -- Darren New / San Diego, CA, USA (PST) "That's pretty. Where's that?" "It's the Age of Channelwood." "We should go there on vacation some time." From tsuraan@REDACTED Sun Jun 1 03:00:29 2008 From: tsuraan@REDACTED (tsuraan) Date: Sat, 31 May 2008 20:00:29 -0500 Subject: [erlang-questions] ioctl support? In-Reply-To: <1212275057.6409.138.camel@localhost> References: <84fb38e30805311552p33ef95ddxf62381b7cdde742e@mail.gmail.com> <1212275057.6409.138.camel@localhost> Message-ID: <84fb38e30805311800t66e47fc9jab554a0abb6f7e40@mail.gmail.com> > No. What are you trying to achieve? Most style of processing are > possible because of the structure of the erlang environment or as part > of specific library interfaces. I'm trying to determine if my cd-rom drive tray is open. I think the only way to do that in Linux is to use the CDROM_DRIVE_STATUS request (an ioctl). Writing the port to do that should be pretty easy, but I figured I'd see if there was a way to do the ioctl directly before learning how to write a port. From yinso.chen@REDACTED Sun Jun 1 03:53:04 2008 From: yinso.chen@REDACTED (YC) Date: Sat, 31 May 2008 18:53:04 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> Message-ID: <779bf2730805311853k2e7ca457k8f037e4d52a3d75a@mail.gmail.com> > You have M users and each user follows an average of N other users. To > render a user's timeline, you need to present the latest 20 tweets > (twoorls in the new terminology :) ) from all his friends. The naive > way to do it is to fetch N*20 messages, sort them, then take the first > 20. This is expensive to do when N is big. Another problem is that > because of the frequent updates from typical users, caching this > calculation doesn't give you much because the cache gets frequently > invalidated between page loads. > > The only solution I see is to keep a "latest twoorls" store for each > user. When a friend sends a new twoorl, it (or a pointer to it) is > copied to the twoorl store for all his friends. This trades space for > speed, and it limits the paging you can do to see your twoorl history, > but it's the only way you can scale this kind of service when N is > big. > It seems the solution will look similar to emails and mailing lists, which both are quite scalable. Except in this case every account is also a broadcast mailing list. Cheers, yc -------------- next part -------------- An HTML attachment was scrubbed... URL: From masterofquestions@REDACTED Sun Jun 1 04:31:52 2008 From: masterofquestions@REDACTED (db) Date: Sat, 31 May 2008 22:31:52 -0400 Subject: [erlang-questions] cacherl memcached app startup problem question In-Reply-To: <14f0e3620805310300t175e4ea0wb539e3c8f1a84786@mail.gmail.com> References: <1218d6a50805301550u50ddc883l4e07607c2e70f760@mail.gmail.com> <14f0e3620805310300t175e4ea0wb539e3c8f1a84786@mail.gmail.com> Message-ID: <1218d6a50805311931rbe47e08m26de982925a08c81@mail.gmail.com> fedora 8. On Sat, May 31, 2008 at 6:00 AM, Gleb Peregud wrote: > On Sat, May 31, 2008 at 12:50 AM, db wrote: >> I am not sure what the problem is. Mnesia directory actually gets >> created at /var/mensia/cacherl@REDACTED but there is no content. >> Is this something to do with disc_copies and any reason for using >> disc_copies rather than ram_copies? Another question, does cacherl >> work with mysql memcached engine? >> >> [a@REDACTED cacherl]# ./start.sh >> Erlang (BEAM) emulator version 5.6.1 [source] [smp:2] >> [async-threads:30] [hipe] [kernel-poll:true] >> >> Eshell V5.6.1 (abort with ^G) >> (cacherl@REDACTED)1> >> =INFO REPORT==== 30-May-2008::17:16:51 === >> Creating mnesia table ns_default >> >> =SUPERVISOR REPORT==== 30-May-2008::17:16:51 === >> Supervisor: {local,memcached} >> Context: start_error >> Reason: {{badmatch, >> {aborted, >> {bad_type,ns_default,disc_copies, >> 'cacherl@REDACTED'}}}, >> [{memcached_mnesia,init_mnesia_table,2}, >> {memcached_mnesia,init,1}, >> {gen_server,init_it,6}, >> {proc_lib,init_p,5}]} >> Offender: [{pid,undefined}, >> {name,memcached_mnesia}, >> {mfa,{memcached_mnesia,start_link,[]}}, >> {restart_type,permanent}, >> {shutdown,2000}, >> {child_type,worker}] >> >> >> =CRASH REPORT==== 30-May-2008::17:16:51 === >> crasher: >> pid: <0.69.0> >> registered_name: [] >> exception exit: {shutdown,{memcached,start,[normal,[]]}} >> in function application_master:init/4 >> initial call: application_master:init(<0.6.0>,<0.68.0>, >> {appl_data,memcached, >> [memcached,memcached_mnesia], >> undefined, >> {memcached,[]}, >> [memcached,memcached_mnesia, >> memcached_stats,memcached_util, >> memcached_purge,memcached_reader], >> [],infinity,infinity}, >> normal) >> ancestors: [<0.68.0>] >> messages: [{'EXIT',<0.70.0>,normal}] >> links: [<0.68.0>,<0.6.0>] >> dictionary: [] >> trap_exit: true >> status: running >> heap_size: 377 >> stack_size: 23 >> reductions: 87 >> neighbours: >> >> =INFO REPORT==== 30-May-2008::17:16:51 === >> application: memcached >> exited: {shutdown,{memcached,start,[normal,[]]}} >> type: temporary >> >> =INFO REPORT==== 30-May-2008::17:16:51 === >> started TCP listener on 0.0.0.0:11311 >> >> >> -- >> rk >> >> That which we persist in doing becomes easier for us to do; not that >> the nature of the thing itself is changed, but that our power to do is >> increased. >> -Ralph Waldo Emerson >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > Hello, > > What system do you use? Is this Cygwin/WinXP? Anthony Kong had exacly > the same problem on Cygwin/WinXP and as far as i know was unable to > find the source. Anthony, may you comment on this issue? Having two > similar cases it would be easier to cope with this problem. > >> Another question, does cacherl >> work with mysql memcached engine? > I've never worked with mysql memcached engine, but as an answer i'll > quote myself: > > Cacherl is fully compatible with original memcached, hence you can > just replace your memcached with cacherl and it should work out of the > box, regardless language your application is written in and libraries > it uses. > > Hence it will probably work :) > > > Best regards, > -- > Gleb Peregud > http://gleber.pl/ > > Every minute is to be grasped. > Time waits for nobody. > -- Inscription on a Zen Gong > -- rk That which we persist in doing becomes easier for us to do; not that the nature of the thing itself is changed, but that our power to do is increased. -Ralph Waldo Emerson From steven.charles.davis@REDACTED Sun Jun 1 09:23:27 2008 From: steven.charles.davis@REDACTED (Steve) Date: Sun, 1 Jun 2008 00:23:27 -0700 (PDT) Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> Message-ID: <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> On May 31, 5:04 pm, "Yariv Sadan" wrote: > ...but it's the only way you can scale this kind of service when N is > big. Hmmm, Yariv, aren't you still thinking about this in the way that Dave Smith pointed to as the heart of the issue? i.e. Dave said: "My understanding is that the reason they have such poor uptime is due to the fact that they modeled the problem as a web-app instead of a messaging system." I'm aware that you are likely a good way away from hitting any scalability problems, but some kind of tiering would seem to be appropriate if twoorl is to be "twitter done right". Yaws at the front end, definitely - but rather /RabbitMQ/ at the back end. I believe that you'd then have the flexibility to distribute/cluster as necessary to scale to the required level (whatever that may be). For sure, Twoorl is a great demo of what can be done with Erlang in an incredibly short time. I'm a relative noob to Erlang, and have learned a great deal from your blog/code/examples. Steve From steven.charles.davis@REDACTED Sun Jun 1 09:32:08 2008 From: steven.charles.davis@REDACTED (Steve) Date: Sun, 1 Jun 2008 00:32:08 -0700 (PDT) Subject: [erlang-questions] Message Receive Semantics (was eep: Multiple Patterns) In-Reply-To: <8209f740805302259n62c23462r7974fd636bf9d9db@mail.gmail.com> References: <8209f740805302259n62c23462r7974fd636bf9d9db@mail.gmail.com> Message-ID: <1175bba7-6449-493a-84ed-cbf8c52aa0a4@34g2000hsf.googlegroups.com> At the risk of sounding like a "dumb noob" (which is, actually, pretty much what I am)... ...can anyone clarify for me why you would wish to retain messages in the process mailbox that the process didn't wish to receive or act upon? It struck me from the get-go when learning about Erlang that I could not think of a use case for this mode of operation, and it always seemed to me to be a touch dangerous (and a good way to break an unsuspecting application). Yes - I'm asking why the erlang messaging works the way it does. /Steve From monch1962@REDACTED Sun Jun 1 10:41:05 2008 From: monch1962@REDACTED (David Mitchell) Date: Sun, 1 Jun 2008 18:41:05 +1000 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> Message-ID: This is a REALLY interesting discussion, but at this point it's becoming obvious that I don't know enough about Twitter... Are you suggesting that Twoorl should be architected as follows: - when they register, every user gets assigned their own RabbitMQ incoming and outgoing queues - user adds a message via Web/Yaws interface (I know, this could be SMS or something else later...) - message goes to that user's RabbitMQ incoming queue - a backend reads messages from the user's incoming queue, looks up in e.g. a Mnesia table to see who should be receiving messages from that user and whether they're connected or not. If "yes" to both, RabbitMQ then forwards the message to each of those users' outgoing queues - either the receiving users poll their outgoing queue for the forwarded message, or a COMET-type Yaws app springs to life and forwards the message to their browser (again, ignoring SMS) This seems like a reasonable approach; I'm just curious if that's what you're suggesting, or whether you've got something else in mind. Great thread, and thanks Yariv for getting this discussion going with Twoorl Regards Dave M. 2008/6/1 Steve : > > On May 31, 5:04 pm, "Yariv Sadan" wrote: >> ...but it's the only way you can scale this kind of service when N is >> big. > > Hmmm, Yariv, aren't you still thinking about this in the way that Dave > Smith pointed to as the heart of the issue? i.e. > Dave said: "My understanding is that the reason they have such poor > uptime is due to the fact that they modeled the problem as a web-app > instead of a messaging system." > > I'm aware that you are likely a good way away from hitting any > scalability problems, but some kind of tiering would seem to be > appropriate if twoorl is to be "twitter done right". Yaws at the front > end, definitely - but rather /RabbitMQ/ at the back end. I believe > that you'd then have the flexibility to distribute/cluster as > necessary to scale to the required level (whatever that may be). > > For sure, Twoorl is a great demo of what can be done with Erlang in an > incredibly short time. I'm a relative noob to Erlang, and have learned > a great deal from your blog/code/examples. > > Steve > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From pstanley1970@REDACTED Sun Jun 1 11:10:16 2008 From: pstanley1970@REDACTED (Paul Stanley) Date: Sun, 1 Jun 2008 10:10:16 +0100 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> Message-ID: On Sun, Jun 1, 2008 at 8:23 AM, Steve wrote: > > Hmmm, Yariv, aren't you still thinking about this in the way that Dave > Smith pointed to as the heart of the issue? i.e. > Dave said: "My understanding is that the reason they have such poor > uptime is due to the fact that they modeled the problem as a web-app > instead of a messaging system." This seems to be the Twitter developers' own conclusion, too (http://dev.twitter.com/2008/05/twittering-about-architecture.html): "Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency's sake, Twitter was built with technologies and practices that are more appropriate to a content management system. Over the last year and a half we've tried to make our system behave like a messaging system as much as possible, but that's introduced a great deal of complexity and unpredictability. When we're in crisis mode, adding more instrumentation to help us navigate the web of interdependencies in our current architecture is often our primary recourse. This is, clearly, not optimal." It's worth googling "twitter architecture". There's a lot of rather feverish discussion on the various issues, but some of it is illuminating. -- Paul Stanley From jay.krishnanm@REDACTED Sun Jun 1 12:47:51 2008 From: jay.krishnanm@REDACTED (Jayakrishnan M) Date: Sun, 1 Jun 2008 16:17:51 +0530 Subject: [erlang-questions] erlang module in apache cgi script not working Message-ID: <88f71dce0806010347h6c337854ic62a38191d50e5f3@mail.gmail.com> Hi, I am using Apache httpd 2 and have a shell script as a cgi script. I am trying to call an erlang module from the shell script for some processing. The problem is that the erlang code is not getting executed. The lines in the shell script which are above and below the call to erlang binary are working. I have been trying to figure out the problem for a long time, but without any success. Can anyone help me please? Thanks in advance. Jayakrishnan From ulf@REDACTED Sun Jun 1 12:59:47 2008 From: ulf@REDACTED (Ulf Wiger) Date: Sun, 1 Jun 2008 12:59:47 +0200 Subject: [erlang-questions] Message Receive Semantics (was eep: Multiple Patterns) In-Reply-To: <1175bba7-6449-493a-84ed-cbf8c52aa0a4@34g2000hsf.googlegroups.com> References: <8209f740805302259n62c23462r7974fd636bf9d9db@mail.gmail.com> <1175bba7-6449-493a-84ed-cbf8c52aa0a4@34g2000hsf.googlegroups.com> Message-ID: <8209f740806010359u53b3c2ecxe4892ebe8d1be8b@mail.gmail.com> Steve, The simplest (and most ubiquitous) case is gen_server:call(). The function gen:await_resp(...) uses pattern matching to check different messages that relate to the request sent. It touches no other messages. It's a form of scoped receive, and there are of course other ways to achieve the same thing. BR, Ulf W 2008/6/1 Steve : > At the risk of sounding like a "dumb noob" (which is, actually, pretty > much what I am)... > > ...can anyone clarify for me why you would wish to retain messages in > the process mailbox that the process didn't wish to receive or act > upon? > > It struck me from the get-go when learning about Erlang that I could > not think of a use case for this mode of operation, and it always > seemed to me to be a touch dangerous (and a good way to break an > unsuspecting application). > > Yes - I'm asking why the erlang messaging works the way it does. > > /Steve > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From chsu79@REDACTED Sun Jun 1 13:03:05 2008 From: chsu79@REDACTED (Christian S) Date: Sun, 1 Jun 2008 13:03:05 +0200 Subject: [erlang-questions] Message Receive Semantics (was eep: Multiple Patterns) In-Reply-To: <1175bba7-6449-493a-84ed-cbf8c52aa0a4@34g2000hsf.googlegroups.com> References: <8209f740805302259n62c23462r7974fd636bf9d9db@mail.gmail.com> <1175bba7-6449-493a-84ed-cbf8c52aa0a4@34g2000hsf.googlegroups.com> Message-ID: > ...can anyone clarify for me why you would wish to retain messages in > the process mailbox that the process didn't wish to receive or act > upon? Having one receive-clause retain messages doesnt mean that you have to make all do it. So it is a matter of the process only wanting to handle a certain kind of message _at this time_. A perfect example of a selective receive is in a gen_server:call, you want the reply to your request back, not the next message in the mailbox. From valentin@REDACTED Sun Jun 1 13:22:06 2008 From: valentin@REDACTED (Valentin Micic) Date: Sun, 1 Jun 2008 13:22:06 +0200 Subject: [erlang-questions] Message Receive Semantics (was eep: MultiplePatterns) References: <8209f740805302259n62c23462r7974fd636bf9d9db@mail.gmail.com> <1175bba7-6449-493a-84ed-cbf8c52aa0a4@34g2000hsf.googlegroups.com> Message-ID: <008d01c8c3d9$bcf0b3f0$6401a8c0@moneymaker2> > From: "Steve" > ...can anyone clarify for me why you would wish to retain messages in > the process mailbox that the process didn't wish to receive or act > upon? > Consider a following scenario: for every five "high" priority messages processed, I wanted to handle three "normal" and one "low" priority message(*). To implement this with a selective receive is relatively easy and concise, even for a novice programmer. To do without it, may be considerably more difficult and messy. Aside selective receive caveats due to the current realization (I really like Sean's idea for optimization), Erlang is much better with, than without it. IMO, the selective receive reflect the spirit of ERLANG. After all, you don't have to use it. > Yes - I'm asking why the erlang messaging works the way it does. So, I suppose the answer might be: because everything else works pretty much the same way. V. (*) Assuming that priority is given as a part of the message. From pstanley1970@REDACTED Sun Jun 1 14:39:19 2008 From: pstanley1970@REDACTED (Paul Stanley) Date: Sun, 1 Jun 2008 13:39:19 +0100 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> Message-ID: On Sun, Jun 1, 2008 at 9:41 AM, David Mitchell wrote: ... > Are you suggesting that Twoorl should be architected as follows: > - when they register, every user gets assigned their own RabbitMQ > incoming and outgoing queues > - user adds a message via Web/Yaws interface (I know, this could be > SMS or something else later...) > - message goes to that user's RabbitMQ incoming queue > - a backend reads messages from the user's incoming queue, looks up in > e.g. a Mnesia table to see who should be receiving messages from that > user and whether they're connected or not. If "yes" to both, RabbitMQ > then forwards the message to each of those users' outgoing queues > - either the receiving users poll their outgoing queue for the > forwarded message, or a COMET-type Yaws app springs to life and > forwards the message to their browser (again, ignoring SMS) > > This seems like a reasonable approach; I'm just curious if that's what > you're suggesting, or whether you've got something else in mind. I think this is not quite right. As I understand it, twitter messages are retrieved by someone who is "following" another user when the follower logs on. They don't have to be connected when the message is sent. In other words, the server side has to maintain (or construct) an archive of "received messages" for each user. Like email. But unlike email (which is normally longish messages to few people, this system assumes short messages which may well go to lots of people. There seem to be two options. The first (which is what Twitter originally used) is to store every outgoing message once, and construct the archive on the fly. When a user comes online and asks for messages, a server looks up who that user follows, finds the messages from each followed person (which may involve checking who is allowed to see them), arranges them and delivers them. The cost, in effort and space, of storing a message is small. The cost of retrieving them is high, and this (it seems) is where Twitter has been hitting problems. It is difficult to cache efficiently (since most users follow a different set of people, so queries are often unique) and they have had trouble scaling it. The alternative is to process outgoing messages at once, delivering copies to each "follower". It's tempting to assume this is the obviously right solution, but bear in mind that it means more heavy weight processing on send, and that it also means a great deal of redundant storage. Even if each message is only 200 bytes long, including metadata, if 10 people are following a message that still means more than 1K of "unnecessary" storage, and that's before you consider the need for copies to ensure reliability. As I understand it, some users are followed by thousands of people ... you can see where that is going. Tempting as it is to say "throw more disks at it", I don't think that's an altogether elegant answer: after all, even if storage doesn't cost much it does cost something. One solution, which Yariv proposed above, is to limit the number of messages in each queue, flushing old messages. But as I understand it, there are many users who like to be able to go back more than 20 messages. So that solves the problem at the cost of desired function. Perhaps (probably?) the key is to find a way of storing very lightweight pointers to messages, which can be appear redundantly in many queues/archives without too much wastage, but without arbitrary limits on archive size. The devil is probably in the detail with that, though. In particular, unless you make them very lightweight you may still have a painfully wastefully bloated storage requirement and you have to be happy that you have a blazingly efficient way of retrieving the messages from these pointers. (I feel this discussion, fascinating as I find it, is not very Erlang-related (at least not directly). I hope it's worth having though, because these basic issues are very interesting ... at least I find them so!) -- Paul Stanley From steven.charles.davis@REDACTED Sun Jun 1 15:05:45 2008 From: steven.charles.davis@REDACTED (Steve) Date: Sun, 1 Jun 2008 06:05:45 -0700 (PDT) Subject: [erlang-questions] Message Receive Semantics (was eep: MultiplePatterns) In-Reply-To: <008d01c8c3d9$bcf0b3f0$6401a8c0@moneymaker2> References: <8209f740805302259n62c23462r7974fd636bf9d9db@mail.gmail.com> <1175bba7-6449-493a-84ed-cbf8c52aa0a4@34g2000hsf.googlegroups.com> <008d01c8c3d9$bcf0b3f0$6401a8c0@moneymaker2> Message-ID: Many thanks all - I consider myself fully "clarified" on this issue. Also - forewarned is forearmed - I actually do have a number of high volume "priority queues" in my current application effort - hence my initial interest in this thread. I admit that I was being a touch disingenuous in phrasing my question but it seemed the briefest way to get to the point - i.e. ask the dumbest relevant question possible. I was avoiding using pattern matching on receive exactly because of this nagging issue about "dead messages" - but maybe I'll review this approach given what's been said. I do have a concern that e.g. if you scan through and so end up with a box full of low priority messages and then a new high priority message arrives... that the scan will have to go through all the old messages to get a match to the new high priority one... but I think I'll leave my concerns about that to when the app is working correctly and I can do some genuine performance measurements - unless anyone has some rule- of-thumb guidelines/gotchas I should be aware of before I commit to this path? Regs, Steve From steven.charles.davis@REDACTED Sun Jun 1 16:08:00 2008 From: steven.charles.davis@REDACTED (Steve) Date: Sun, 1 Jun 2008 07:08:00 -0700 (PDT) Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> Message-ID: <1059230d-ef2a-4ad1-b8cf-3cfad8b17864@s50g2000hsb.googlegroups.com> On Jun 1, 3:41 am, "David Mitchell" wrote: > I'm just curious if that's what > you're suggesting, or whether you've got something else in mind. Actually, I haven't fully thought it all through in detail (I'm supposed to be 100% on my own projects, lol) but I had in mind something like... ...an MQ backend with maybe a publish and subscribe queue for each user where the web app instances/sms gateways act as clients to collect the tweets for presentation/forwarding to the current follower/ user... ...I also was thinking of a peer to peer queue between MQ instances that would allow for separating the messaging volume out (to allow unlimited scaling) into messaging "domains" or regions across multiple instances... ... And that the user accounts (far less volume) could be probably stored in a single RDBMS *cluster* (think oracle replication), and possibly the account data could hold appropriate routing info to the users' "messaging domain"... ...all very vague but seems to me to be the right direction to go. Regs, Steve From erlang@REDACTED Sun Jun 1 16:10:22 2008 From: erlang@REDACTED (Joe Armstrong) Date: Sun, 1 Jun 2008 16:10:22 +0200 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> Message-ID: <9b08084c0806010710y1eb4e077k2888a7d14d37f347@mail.gmail.com> It *is* an interesting discussion - not about Twitter - but about architectures. There seem to be some implicit assumptions here: Let's suppose that twoorl services are accessed through a *single* name (www.twooerl.org) or something. Let's assume we have 20 (for the sake of argument) back-end machines (could be hundreds or thousands though)( I guess scalable means that we can just add/remove backend machines without breaking things :-) The first step must be to associate 20 IP addresses with this single name - some kind of DNS load balancing should be possible. (how do you do this?????)(is this DNS round robin????)(does this scale to thousands of machines???) The user tries to connect to www.twooerl.org and looks up the address in DNS the result is one of these IP addresses. The user connects to one of theses IP addresses and and requests data for "joe" - they are immediately redirected to the machine having joe's data. The simplest way to find a machine would be to use the idea in chord. For a machine with IP 123.45.12.34 we create a tuple {"123.45.12.34", Md5("123.45.12.34")} we do this for all machines and sort my the Md5 sum so we get a list of twenty machines, say: [{"122.34.45.67", Md1}, {"223.56.1.23", Md2}, ... ] Which machine is "joe" stored on? - to find this we compute md5("joe") and find the first machine in the list whose Md5 sum is greater than md5("joe"). The initial machine would perform this computation and redirect to the machine with joe's data. The initial machine can also check the liveness this second machine - and if it is unresponsive redirect to a machine containing a replica of joe's data (which could this be? - imagine the machines arranged in a circle and redirect to the machine nearest to 180 degrees away from the original machine) A problem occurs if the original machine is dead - (ie the one that DNS said was the address associated with www.twoo.erl) - if the back-end machines are in pairs that monitor each other then I guess a gratuitous arp can fix the problem and reassign the the IP address of the failing machine to a new machine (which must now take over the role of the first machine) In all of this the fundamental problem seems to be that if the server fails then we have to do a lot of messing around to keep the server on a fixed IP address. It would be a zillion times easier if the *client* did all this messing around. If the client knew about N server address it could automatically redirect to a different server if the primary server failed. if the client knew about N machines and performed the md5(Key) calculation to find the correct machine then the problem would be very easy to solve with no messing around in the server. (( The fact that DNS has multiple addresses make writing a resolver really easy, if one DNS server fails just try anothert )) Now if we're talking "standard browsers" then accessing multiple sites is painful. Javascript and flash plugins etc. are very restrictive in the sites that they can open sockets to (for good reasons) I think only the originating site is permitted. If we could persuade the users to install a small proxy one their machines then all these problems would go away - a standard browser could talk to a proxy on localhost and this could talk to the multiple back ends. What appears to be a tricky problem in scaling things if we have to keep the back-end servers on fixed addresses seems a lot easier if the clients have some limited intelligence. The next step is (ovf course) to allow all the clients to collectively behave as if they were as server - I think therefore that the problem is really one about the base level of a P2P architecture and not about a twitter architecture per se. If we did have a simple proxy component that allowed messaging to multiple sites then this and many other problems would be easily soluble. We might imaging a proxy that that was programmable: It presents a menu like this: {allow, ServiceName, at, Addr1, Addr2, Addr3, .....} (( example {allow, twitter, at, www.a.b, www.p.q, www.c.d} - this means that the proxy can open "twitter" sessions to the three "trusted" machines in the list then the web browser could access the proxy that could talk to the trusted machines - the trusted machine should just redirect to other machines, until finally the desired machines are found. )) Would the fact that you have to install a proxy server limit deployment of a service? - probably. Also it opens up a new bag of worms (security) - for good reasons browsers do not allow plugins to access multiple sites (only the originating sites). I suppose therefore, that the inner cluster that provides the service would have a full P2P structure and that the service would be accessed by DNS round robin with IP failover to handle the errors. I suspect that architecture like this are being used in some Erlang systems (the details might vary) If anybody would like to post code and go into the details of how to rig systems for DNS load balancing (or whatever it's called) and for IP monitoring and fail-over then we could get to the interesting part of building the application) (( the next bit will be to look at the limits of scaling - still nobody has talked numbers - how far can we press a two-tier system - with say 20 name servers in the front-end that *only* do redirects - this bit should be very fast )) (( By combining twitter with IRC we might make the option of installing a proxy more attractive - The irc "server" for a group G should really me the client that first created the group G - If G drops out the second machine in the group could become the server and so on. Really twitter is like having one IRC grrup per person. many people can join but only the owner can write to it. What's the difference (architectually) . )) Cheers /Joe Armstrong On Sun, Jun 1, 2008 at 10:41 AM, David Mitchell wrote: > This is a REALLY interesting discussion, but at this point it's > becoming obvious that I don't know enough about Twitter... > > Are you suggesting that Twoorl should be architected as follows: > - when they register, every user gets assigned their own RabbitMQ > incoming and outgoing queues > - user adds a message via Web/Yaws interface (I know, this could be > SMS or something else later...) > - message goes to that user's RabbitMQ incoming queue > - a backend reads messages from the user's incoming queue, looks up in > e.g. a Mnesia table to see who should be receiving messages from that > user and whether they're connected or not. If "yes" to both, RabbitMQ > then forwards the message to each of those users' outgoing queues > - either the receiving users poll their outgoing queue for the > forwarded message, or a COMET-type Yaws app springs to life and > forwards the message to their browser (again, ignoring SMS) > > This seems like a reasonable approach; I'm just curious if that's what > you're suggesting, or whether you've got something else in mind. > > Great thread, and thanks Yariv for getting this discussion going with Twoorl > > Regards > > Dave M. > > 2008/6/1 Steve : >> >> On May 31, 5:04 pm, "Yariv Sadan" wrote: >>> ...but it's the only way you can scale this kind of service when N is >>> big. >> >> Hmmm, Yariv, aren't you still thinking about this in the way that Dave >> Smith pointed to as the heart of the issue? i.e. >> Dave said: "My understanding is that the reason they have such poor >> uptime is due to the fact that they modeled the problem as a web-app >> instead of a messaging system." >> >> I'm aware that you are likely a good way away from hitting any >> scalability problems, but some kind of tiering would seem to be >> appropriate if twoorl is to be "twitter done right". Yaws at the front >> end, definitely - but rather /RabbitMQ/ at the back end. I believe >> that you'd then have the flexibility to distribute/cluster as >> necessary to scale to the required level (whatever that may be). >> >> For sure, Twoorl is a great demo of what can be done with Erlang in an >> incredibly short time. I'm a relative noob to Erlang, and have learned >> a great deal from your blog/code/examples. >> >> Steve >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From per.melin@REDACTED Sun Jun 1 16:58:02 2008 From: per.melin@REDACTED (Per Melin) Date: Sun, 1 Jun 2008 16:58:02 +0200 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <9b08084c0806010710y1eb4e077k2888a7d14d37f347@mail.gmail.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <9b08084c0806010710y1eb4e077k2888a7d14d37f347@mail.gmail.com> Message-ID: 2008/6/1 Joe Armstrong : > A problem occurs if the original machine is dead - (ie the one that > DNS said was the address associated > with www.twoo.erl) - if the back-end machines are in pairs that > monitor each other then I guess > a gratuitous arp can fix the problem and reassign the the IP address > of the failing machine to a > new machine (which must now take over the role of the first machine) This is a problem that every web site faces once it outgrows a single server, and is usually solved with a dedicated off-the-shelf load balancer (with a "high enough" MTBF) that detects and excludes failing servers. These are turn-key solutions that work for the rest of the web, including banks and others with higher demands than something like Twitter has. > The user connects to one of theses IP addresses and and requests data > for "joe" - they are immediately redirected to the machine having > joe's data. Here is where Twitter goes wrong, I think. The data, and the processing, is not distributed. All of it is in a single MySQL master server, with two slaves for read access. It's easy to see how that can't scale. From patrickdlogan@REDACTED Sun Jun 1 17:01:30 2008 From: patrickdlogan@REDACTED (Patrick Logan) Date: Sun, 1 Jun 2008 08:01:30 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone Message-ID: "Yaws at the front end, definitely - but rather /RabbitMQ/ at the back end." I'm not sure why RabbitMQ would be a better choice than XMPP / ejabberd for a text messaging system? In addition to that XMPP can federate with existing IM systems, an XMPP-based "Twoorl" could federate itself easily across multiple organizations. How easy is it to federate across organizational boundaries AMQP or RabbitMQ in particular? I mean assuming if Twoorl does get to this scale, does he want to continue paying for the entire infrastructure? From harveyd@REDACTED Sun Jun 1 17:03:02 2008 From: harveyd@REDACTED (Dale Harvey) Date: Sun, 1 Jun 2008 16:03:02 +0100 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <9b08084c0806010710y1eb4e077k2888a7d14d37f347@mail.gmail.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <9b08084c0806010710y1eb4e077k2888a7d14d37f347@mail.gmail.com> Message-ID: Flash sockets can open links to external sites, it requires the same verification as local sites now do since (v9.0.16.0?) which means when you try to .connect, flash will setup a temporary connection, send a and expects an xml snippet that looks something like http://api.flickr.com/crossdomain.xml. An IP lookup does sound nicer than a full http proxy, As far as I know you cant do cross domain POSTs with js though, GET works fine with json return. 2008/6/1 Joe Armstrong : > It *is* an interesting discussion - not about Twitter - but about > architectures. > > There seem to be some implicit assumptions here: > > Let's suppose that twoorl services are accessed through a *single* > name (www.twooerl.org) or something. > > Let's assume we have 20 (for the sake of argument) back-end machines > (could be hundreds or thousands though)( I guess scalable means that > we can just add/remove backend machines without breaking > things :-) > > The first step must be to associate 20 IP addresses with this single > name - some kind of DNS load balancing should be possible. (how do you > do this?????)(is this DNS round robin????)(does this scale to > thousands of machines???) > > The user tries to connect to www.twooerl.org and looks up the address > in DNS the result is one of these IP addresses. > > The user connects to one of theses IP addresses and and requests data > for "joe" - they are immediately redirected to the machine having > joe's data. > > The simplest way to find a machine would be to use the idea in chord. > > For a machine with IP 123.45.12.34 we create a tuple > > {"123.45.12.34", Md5("123.45.12.34")} we do this for all machines and > sort my the Md5 sum > > > so we get a list of twenty machines, say: > > [{"122.34.45.67", Md1}, {"223.56.1.23", Md2}, ... ] > > Which machine is "joe" stored on? - to find this we compute md5("joe") > and find the > first machine in the list whose Md5 sum is greater than md5("joe"). > > The initial machine would perform this computation and redirect to the > machine with joe's data. > > The initial machine can also check the liveness this second machine - > and if it is unresponsive > redirect to a machine containing a replica of joe's data (which could > this be? - imagine the > machines arranged in a circle and redirect to the machine nearest to > 180 degrees away from the > original machine) > > A problem occurs if the original machine is dead - (ie the one that > DNS said was the address associated > with www.twoo.erl) - if the back-end machines are in pairs that > monitor each other then I guess > a gratuitous arp can fix the problem and reassign the the IP address > of the failing machine to a > new machine (which must now take over the role of the first machine) > > In all of this the fundamental problem seems to be that if the server > fails then we have to do a lot > of messing around to keep the server on a fixed IP address. > > It would be a zillion times easier if the *client* did all this > messing around. If the client > knew about N server address it could automatically redirect to a > different server if the primary server > failed. if the client knew about N machines and performed the md5(Key) > calculation to find the correct machine then the problem would be very > easy to solve with no messing around in the server. > > (( The fact that DNS has multiple addresses make writing a resolver > really easy, if one DNS server > fails just try anothert )) > > Now if we're talking "standard browsers" then accessing multiple sites > is painful. Javascript and flash > plugins etc. are very restrictive in the sites that they can open > sockets to (for good reasons) > I think only the originating site is permitted. > > If we could persuade the users to install a small proxy one their > machines then all these problems would > go away - a standard browser could talk to a proxy on localhost and > this could talk to the multiple > back ends. > > What appears to be a tricky problem in scaling things if we have to > keep the back-end servers on fixed > addresses seems a lot easier if the clients have some limited intelligence. > > The next step is (ovf course) to allow all the clients to collectively > behave as if they were as server - > I think therefore that the problem is really one about the base level > of a P2P architecture and not > about a twitter architecture per se. > > If we did have a simple proxy component that allowed messaging to > multiple sites then this and many other > problems would be easily soluble. > > We might imaging a proxy that that was programmable: > > It presents a menu like this: > > {allow, ServiceName, at, Addr1, Addr2, Addr3, .....} > > (( example {allow, twitter, at, www.a.b, www.p.q, www.c.d} > - this means that the proxy can open "twitter" sessions to the > three "trusted" machines in the list > > then the web browser could access the proxy that could talk to the > trusted machines - > the trusted machine should just redirect to other machines, until > finally the desired machines are found. )) > > Would the fact that you have to install a proxy server limit > deployment of a service? - probably. > Also it opens up a new bag of worms (security) - for good reasons > browsers do not allow plugins > to access multiple sites (only the originating sites). > > I suppose therefore, that the inner cluster that provides the service > would have a full P2P structure > and that the service would be accessed by DNS round robin with IP > failover to handle the errors. > > I suspect that architecture like this are being used in some Erlang > systems (the details might vary) > > If anybody would like to post code and go into the details of how to > rig systems for DNS load balancing > (or whatever it's called) and for IP monitoring and fail-over then we > could get to the interesting > part of building the application) > > (( the next bit will be to look at the limits of scaling - still > nobody has talked numbers - > how far can we press a two-tier system - with say 20 name servers > in the front-end that *only* do > redirects - this bit should be very fast )) > > (( By combining twitter with IRC we might make the option of > installing a proxy more attractive - > The irc "server" for a group G should really me the client that > first created the group G - > If G drops out the second machine in the group could become the > server and so on. > Really twitter is like having one IRC grrup per person. many > people can join but only the > owner can write to it. What's the difference (architectually) . > )) > > Cheers > > /Joe Armstrong > > > > On Sun, Jun 1, 2008 at 10:41 AM, David Mitchell > wrote: > > This is a REALLY interesting discussion, but at this point it's > > becoming obvious that I don't know enough about Twitter... > > > > Are you suggesting that Twoorl should be architected as follows: > > - when they register, every user gets assigned their own RabbitMQ > > incoming and outgoing queues > > - user adds a message via Web/Yaws interface (I know, this could be > > SMS or something else later...) > > - message goes to that user's RabbitMQ incoming queue > > - a backend reads messages from the user's incoming queue, looks up in > > e.g. a Mnesia table to see who should be receiving messages from that > > user and whether they're connected or not. If "yes" to both, RabbitMQ > > then forwards the message to each of those users' outgoing queues > > - either the receiving users poll their outgoing queue for the > > forwarded message, or a COMET-type Yaws app springs to life and > > forwards the message to their browser (again, ignoring SMS) > > > > This seems like a reasonable approach; I'm just curious if that's what > > you're suggesting, or whether you've got something else in mind. > > > > Great thread, and thanks Yariv for getting this discussion going with > Twoorl > > > > Regards > > > > Dave M. > > > > 2008/6/1 Steve : > >> > >> On May 31, 5:04 pm, "Yariv Sadan" wrote: > >>> ...but it's the only way you can scale this kind of service when N is > >>> big. > >> > >> Hmmm, Yariv, aren't you still thinking about this in the way that Dave > >> Smith pointed to as the heart of the issue? i.e. > >> Dave said: "My understanding is that the reason they have such poor > >> uptime is due to the fact that they modeled the problem as a web-app > >> instead of a messaging system." > >> > >> I'm aware that you are likely a good way away from hitting any > >> scalability problems, but some kind of tiering would seem to be > >> appropriate if twoorl is to be "twitter done right". Yaws at the front > >> end, definitely - but rather /RabbitMQ/ at the back end. I believe > >> that you'd then have the flexibility to distribute/cluster as > >> necessary to scale to the required level (whatever that may be). > >> > >> For sure, Twoorl is a great demo of what can be done with Erlang in an > >> incredibly short time. I'm a relative noob to Erlang, and have learned > >> a great deal from your blog/code/examples. > >> > >> Steve > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://www.erlang.org/mailman/listinfo/erlang-questions > >> > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Sun Jun 1 17:20:53 2008 From: rvirding@REDACTED (Robert Virding) Date: Sun, 1 Jun 2008 17:20:53 +0200 Subject: [erlang-questions] how: Building Core Erlang abstract syntax trees In-Reply-To: <4841A212.5070800@it.uu.se> References: <4841A212.5070800@it.uu.se> Message-ID: <3dbc6d1c0806010820x50aa653fo32946bd4996b405a@mail.gmail.com> 2008/5/31 Richard Carlsson : > Christian S wrote: > > One can use compile:forms/2 to compile core erlang abstract syntax > > trees directly, but what is the suggested way to construct these > > trees? > > > > There is lib/compile/src/core_parse.* (and especially the records in > > the hrl file), and there is the 'cerl' module which has good edoc. The > > module 'cerl' has node-constructors for the core erlang ast. > > > > Looking at LFE, it doesnt make use of cerl, it hits the core erlang > > records directly. Is this an oversight of Robert Virding or the more > > practical thing to do? > > This is for historical reasons. When we started working on Core Erlang, > Robert (and others) preferred working directly with records, while I > preferred using a "proper" abstract datatype (even though it meant > that I couldn't decompose structures using pattern matching). Also, > we had some different ideas regarding how some things such as literals > and variables ought to be represented. So you'll find that the central > parts of the compiler use the records defined in core_parse.hrl, while > cerl_inline and everything in lib/hipe/cerl uses the cerl.erl ADT (which > can be converted to/from the record format). > Also for what I was doing it was just as easy to use records directly. I do not actually "work" with the Core code, I only build it and pass it directly into the compiler. Actually this is one Big Win in generating Erlang code at this level: - I get to generate a simple functional language which has most of what I need* - I get to use all the optimisations in the compiler as they work on Core, and later, code - the things I lose from the earlier passes I either don't need or can do much easier (in the next release I have list comprehensions, it's the same algorithm but in LFE it's a macro and you can actually "see" what it is doing) So again from my point of view there is no win in using cerl over the records. At I least I don't know of any. > Also, Core Erlang allows for annotations, what is the benefits one > > could have from adding these? Static type analysis or can the compiler > > make use of them? > One useful feature is tagging clauses as 'compiler_generated' so the compiler won't complain that they can never be reached. * What is missing are non-recursive local functions. The reason they are missing is that they aren't need for compiling Erlang. Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From steven.charles.davis@REDACTED Sun Jun 1 17:27:38 2008 From: steven.charles.davis@REDACTED (Steve) Date: Sun, 1 Jun 2008 08:27:38 -0700 (PDT) Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: Message-ID: <1f17f366-cda6-4e81-96e7-6a3316494721@56g2000hsm.googlegroups.com> On Jun 1, 10:01 am, "Patrick Logan" wrote: > "Yaws at the front end, definitely - but rather /RabbitMQ/ at the back end." > I'm not sure why RabbitMQ would be a better choice than XMPP / > ejabberd for a text messaging system? Well, I've not really done my "due diligence" on this, but here goes anyway - the first considerations that come to mind with XMPP/ejabberd is support for... multiple recipients, offline delivery, clients as systems (web servers, sms gateways) not user apps (browsers/phones). I could probably add to the list. > > In addition to that XMPP can federate with existing IM systems, an > XMPP-based "Twoorl" could federate itself easily across multiple > organizations. I can't see how this applies here.. you see, the MQ suggestion was for an additional back-end tier... I wasn't bothering to address the front end interface(s) -- maybe I wasn't clear enough on that? > How easy is it to federate across organizational boundaries AMQP or > RabbitMQ in particular? I mean assuming if Twoorl does get to this > scale, does he want to continue paying for the entire infrastructure? I'm not sure I understand the question here in this context. The suggestion was assuming that you run the entire service a la Twitter. /regs From rvirding@REDACTED Sun Jun 1 17:40:00 2008 From: rvirding@REDACTED (Robert Virding) Date: Sun, 1 Jun 2008 17:40:00 +0200 Subject: [erlang-questions] Leex - a lexical anaylzer generator, released In-Reply-To: <43A4755C-221D-4DD0-8EBD-8D173E00515D@gmail.com> References: <3dbc6d1c0805240839k24b5a91cq71346591fe0faed8@mail.gmail.com> <0E07146E-57CA-47A1-9DD6-856DC2B7B477@gmail.com> <3dbc6d1c0805271656l2fb4ca0fyb13d1ac7f2d10f24@mail.gmail.com> <9924E672-5F09-4BA6-94D4-103F99894222@gmail.com> <3dbc6d1c0805291405w5dd5b82iafdadf9b83155285@mail.gmail.com> <43A4755C-221D-4DD0-8EBD-8D173E00515D@gmail.com> Message-ID: <3dbc6d1c0806010840o7e96ba1di7001a5a1fd82442a@mail.gmail.com> 2008/5/30 Ben Hood <0x6e6562@REDACTED>: > One feature that I miss from other CC toolkits is being able to declare > your regexs to be case insensitive. > > Here's an example from a JavaCC grammar: > > /* Reserved Words */ > TOKEN [IGNORE_CASE] : > { > < NOT : "NOT"> > | < AND : "AND"> > | < OR : "OR"> > | < BETWEEN : "BETWEEN"> > | < LIKE : "LIKE"> > | < ESCAPE : "ESCAPE"> > | < IN : "IN"> > | < IS : "IS"> > | < TRUE : "TRUE" > > | < FALSE : "FALSE" > > | < NULL : "NULL" > > | < XPATH : "XPATH" > > | < XQUERY : "XQUERY" > > } > > I ended having to put patterns in like > > between|BETWEEN > > to match lower and upper cases alike. > > I know that you are bound by what the regexp module can give you though. I know I know. This is a feature which is lacking in the regexp module as well. :-) Maybe in a future release (of both). Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From harveyd@REDACTED Sun Jun 1 17:44:53 2008 From: harveyd@REDACTED (Dale Harvey) Date: Sun, 1 Jun 2008 16:44:53 +0100 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <1f17f366-cda6-4e81-96e7-6a3316494721@56g2000hsm.googlegroups.com> References: <1f17f366-cda6-4e81-96e7-6a3316494721@56g2000hsm.googlegroups.com> Message-ID: http://www.process-one.net/en/blogs/article/introducing_the_xmpp_application_server/ is also pretty topical read, Blaine Cook (ex twitter dev) mentioned it "Scaling Twitter as a messaging platform is pretty easy. See Micka?l R?mond's poston the subject. Scaling the archival, and massive infrastructure concerns (think billions of authenticated polling requests per month) are not, no matter what platform you're on. Particularly when you need to take complex privacy concerns into account." 2008/6/1 Steve : > On Jun 1, 10:01 am, "Patrick Logan" wrote: > > "Yaws at the front end, definitely - but rather /RabbitMQ/ at the back > end." > > I'm not sure why RabbitMQ would be a better choice than XMPP / > > ejabberd for a text messaging system? > > Well, I've not really done my "due diligence" on this, but here goes > anyway - the first considerations that come to mind with XMPP/ejabberd > is support for... multiple recipients, offline delivery, clients as > systems (web servers, sms gateways) not user apps (browsers/phones). I > could probably add to the list. > > > > > In addition to that XMPP can federate with existing IM systems, an > > XMPP-based "Twoorl" could federate itself easily across multiple > > organizations. > > I can't see how this applies here.. you see, the MQ suggestion was for > an additional back-end tier... I wasn't bothering to address the front > end interface(s) -- maybe I wasn't clear enough on that? > > > How easy is it to federate across organizational boundaries AMQP or > > RabbitMQ in particular? I mean assuming if Twoorl does get to this > > scale, does he want to continue paying for the entire infrastructure? > > I'm not sure I understand the question here in this context. The > suggestion was assuming that you run the entire service a la Twitter. > > /regs > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From per.melin@REDACTED Sun Jun 1 17:48:23 2008 From: per.melin@REDACTED (Per Melin) Date: Sun, 1 Jun 2008 17:48:23 +0200 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> Message-ID: 2008/6/1 Paul Stanley : > The alternative is to process outgoing messages at once, delivering > copies to each "follower". Twitter has features that makes it different from any other messaging service I can think of. Some examples: Anyone can go to a users page and see all their posts. So everyone needs a special follower called "everyone else". Unless the user has checked the "protect my updates" box in the settings, in which case only approved followers can view the posts. And even if *my* posts are public, one of them may be a reply to a non-public post from someone else, and I believe that in this case Twitter filters out even the public reply. Unless of course you're logged in when you're viewing my posts and are also approved to view the protected post that I was replying to. I can put @paul in a message to show that it's directed to you. Users have the option to filter out @-messages that are directed to people they themselves are not following. If Joe is following me, but not you, he may or may be shown a post from me with @paul in it. So at some point in the chain, probably every time you show a post, you need to pass through several filters, some of which may require that you examine the settings of other users (which in a distributed system may live on another node). Disclaimer: I'm guilty of some speculation here, since I've only lurked on Twitter, and don't fully know the ins and outs of the functionality it offers. That will change now. I'll start posting, at least for a while. My page: http://twitter.com/pmelin From jay@REDACTED Mon Jun 2 01:08:56 2008 From: jay@REDACTED (Jay Nelson) Date: Sun, 1 Jun 2008 16:08:56 -0700 Subject: [erlang-questions] Message Receive Semantics (was eep: MultiplePatterns) Message-ID: Valentin wrote: > Consider a following scenario: for every five "high" priority messages > processed, I wanted to handle three "normal" and one "low" priority > message(*). To implement this with a selective receive is relatively easy > and concise, even for a novice programmer. To do without it, may be > considerably more difficult and messy. I think priority is one of the harder cases to get right, actually. The beginner will end up blocked waiting for the 5th high priority message when only 4 are on the queue, or will end up delaying high priority messages while they scan for low priority ones. So here is the concrete challenge: 1) Send a message to a linked process: a) {request, Msg} 2) The other process should generate 20 or so messages randomly and then quit 3) Always handle messages in the following order: a) 5 messages of {reply, high, Msg} b) 3 messages of {reply, normal, Msg} c) 1 message of {reply, low, Msg} d) 'EXIT' message Of course, there may not be 5 remaining high messages or 3 remaining normal messages so you need to deal with these end cases. In doing so, make sure you don't just end up polling every 100 milliseconds. I've simplified the problem by only having one process that can generate replies, so that the selective receive doesn't become even more complicated. I challenge any beginner or intermediate erlang programmer to write and post the code for both processes and a description of your results. All the advanced erlang people can help point out the issues that will have to be addressed when you change the initial code approach. At least, that is my bet. Valentin may be right. Let's see some code! In another post, I'll give an alternative problem that is easier and shows the usefulness of selective receive. In general, I would argue that erlang's approach to message is geared towards handling multiple conversations or sessions on a single channel, and that handling priorities is a pathological case for the simple message queue approach that erlang uses. It is far easier to implement priorities by unloading the whole queue into a process which implements a priority queue and then doling out messages according to your own priority logic. jay From jay@REDACTED Mon Jun 2 02:09:28 2008 From: jay@REDACTED (Jay Nelson) Date: Sun, 1 Jun 2008 17:09:28 -0700 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) Message-ID: <7FD49941-69F7-4D4E-B0EE-F4D1104059C3@duomark.com> I wrote: > >Whenever you have disjoint receive statements, you need to > >take care that there is a technique for emptying unexpected > >messages. Edward Fine accidentally replied only to me directly: > Is this a good place to use the catch-all, or is there a better > technique? I ask this as a newcomer to Erlang. (This posting also gives an alternative example to Valentin's priority problem suggestion) Consider a case where you are doing a scatter / gather algorithm to spread processing across nodes or across different processing algorithms. To make it concrete, suppose we have a database with 5 different tables and we need to collect information from each table to assemble into a single view to the user. The standard approach is to use the DB capability to join the tables. This introduces a single point access problem since the database server is doing all the work while the initiating process waits. Instead we put each of the tables in a different DB, flat file or ets table. Then we create a process for each one that provides caching and an access interface using messages. They may end up on the same machine or on 5 different machines, but we will get parallelism on the I/O and possibly on the cache and assembly processing (if there are multiple cores or multiple machines in the case of cache and assembly). What does the code look like? [Assume getQueries(UserId) generates a list of queries that are related to the database information we would like to display and that the length of this list matches the number of DB processes we have. ] doUserQuery(UserId) -> Queries = getQueries(UserId), QueryRef = make_ref(), [Pid ! {getData, QueryRef, UserId, Query} || {Pid, Query} <- lists:zip(DbPids, Queries)], Responses = collect_responses(QueryRef), display_db_info(Responses), erlang:send_after(1000, self(), {cleanup, QueryRef}). This is a pretty hokey approach -- you would want something better than a 1 second delay to tell you whether to eliminate old messages from the queue, but it is a concrete example to describe why you would want to use selective receive and what to do to make sure it doesn't cause you a problem. collectResponses(QueryRef) -> collectResponses(QueryRef, []). collectResponses(QueryRef, Responses) -> receive {responseData, QueryRef, _UserId, Results} -> collectResponses(QueryRef, [Responses | Results]) after 100 -> Responses end. Again, my hokey example collects results as long as they are present or no new ones show up for 100 milliseconds. What we have so far is a single request message sent to 5 processes and a function which implements selective receive to collect only the messages that are in response to the initial request from a variety of responders (hopefully all, but not if some are slow to respond). What happens if we have a slow responding database, but it does actually produce results after 1/2 second. It was too slow to be collected but it puts messages on the queue anyway. If we have no mechanism to clear them, they will build up and cause things to gradually slow down. So at some higher level we need the following code: main() -> receive %% Throw away late arriving results from a previous request {cleanup, QueryRef} -> dumpOldResults(QueryRef); {userRequest, UserId} -> doUserQuery(UserId) end, main(). dumpOldQueryResults(QueryRef) -> receive {responseData, QueryRef, _UserId, _Results} -> dumpOldQueryResults(QueryRef) after 0 -> ok end. In the main function, we give priority to cleaning up old messages. This will keep the queue short, however, it ensures a full queue scan for every user request. As long as the queue is short, that won't hurt us. Dumping old messages just cycles as fast as it can accepting messages that have our unique token and ignoring the rest of the data in the message. If there are no clean up messages remaining, we than accept a new user request (which will necessarily cause the message queue to grow for a short period) and display the results. What did we see? Selective receive used in 3 different ways: 1) To collect the results of a request (a two-way session conversation) 2) To handle self notifications for maintenance + user requests 3) To handle old messages from an expired session It turns out the {cleanup, QueryRef} message is not necessary in the above example and we can just consume all {responseData, ...} messages inside main(), but it depends on how new requests are placed on the queue and whether timing allows two requests to be interleaved in the results set (you don't want to remove all the responseData for a pending request that has not had time to collect results yet). Structuring as above gave more explicit different uses of selective receive. The problem remaining in the code above is that there is no "catch all" clause. Do we worry about that? It depends on how the system evolves. If you interface to a known protocol and you have covered all the messages supported via selective receive, then you could do without a catch all. If your system is evolving or there are other processes or programmers who might inject new message types, you need a catch all in the main/0 function (although you have to be careful not consume something that should stay on the queue). I have not tried this code, nor have I typed it into a erl prompt, so I can't guarantee it even compiles. Mostly it should give you ideas about ways to use selective receive. What if we didn't have selective receive? I see two choices: 1) Start a thread and open a new socket to the databases for each user request. Maintain the conversations as independent channels. 2) Create a hash table of messages received related to each request. This requires managing the conversation correlations yourself. Both of these approaches are much more code than selective receive requires and the complexity of concepts does not increase, so selective receive is a better approach and a useful feature of erlang. Is there a better way to manage the conversations rather than the whole cleanup back channel messaging? If you can spawn a new process for each request, the responses will go to privately owned message queues. When enough responses, or enough time has passed, the newly spawned request process returns its results and terminates. Any messages stuck on the queue are eliminated. Any future messages are silently discarded since there is no process to receive them. If the backend DB process were monitoring the request process, it could even interrupt its response to discard the results rather than waiting for processing to complete and pass them on to a non-existent process. With erlang, there are many architectural choices when you consider the uses of messaging and selective receive. jay From erlang-questions_efine@REDACTED Mon Jun 2 05:35:37 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Sun, 1 Jun 2008 23:35:37 -0400 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <7FD49941-69F7-4D4E-B0EE-F4D1104059C3@duomark.com> References: <7FD49941-69F7-4D4E-B0EE-F4D1104059C3@duomark.com> Message-ID: <6c2563b20806012035g3e4e7568ld1a89aa6527f14a4@mail.gmail.com> Jay, Thanks for a very detailed and informative response. Although it obviously depends on circumstances, I feel that, given Erlang's extremely fast process creation time and small process size, I would first consider your last option, namely, to create an individual process per request, and use an ETS table to coordinate responses. If there are very many responses to be collected for each request, I would intuitively imagine in my "Erlang newbie fog" that using an ETS table with its constant-time performance and no-garbage-collection characteristics would be better on average than using selective receive, which I understand has to do a linear scan and move unprocessed messages to another area. Of course, intuition often does not stand up to the reality of performance measurements, so it would be interesting to see a benchmark of the various architectural options you have described, perhaps as a function of response time vs. request rate. Regards, Edwin Fine On Sun, Jun 1, 2008 at 8:09 PM, Jay Nelson wrote: > I wrote: > > > >Whenever you have disjoint receive statements, you need to > > >take care that there is a technique for emptying unexpected > > >messages. > > Edward Fine accidentally replied only to me directly: > > > Is this a good place to use the catch-all, or is there a better > > technique? I ask this as a newcomer to Erlang. > > (This posting also gives an alternative example to Valentin's > priority problem suggestion) > > Consider a case where you are doing a scatter / gather algorithm > to spread processing across nodes or across different processing > algorithms. To make it concrete, suppose we have a database > with 5 different tables and we need to collect information from each > table to assemble into a single view to the user. > > The standard approach is to use the DB capability to join the tables. > This introduces a single point access problem since the database > server is doing all the work while the initiating process waits. > > Instead we put each of the tables in a different DB, flat file or ets > table. > Then we create a process for each one that provides caching and > an access interface using messages. They may end up on the same > machine or on 5 different machines, but we will get parallelism on > the I/O and possibly on the cache and assembly processing (if there > are multiple cores or multiple machines in the case of cache and > assembly). > > What does the code look like? [Assume getQueries(UserId) generates > a list of queries that are related to the database information we would > like to display and that the length of this list matches the number of > DB processes we have. ] > > doUserQuery(UserId) -> > Queries = getQueries(UserId), > QueryRef = make_ref(), > [Pid ! {getData, QueryRef, UserId, Query} || {Pid, Query} <- > lists:zip(DbPids, Queries)], > Responses = collect_responses(QueryRef), > display_db_info(Responses), > erlang:send_after(1000, self(), {cleanup, QueryRef}). > > > This is a pretty hokey approach -- you would want something better > than a 1 second delay to tell you whether to eliminate old messages > from the queue, but it is a concrete example to describe why you > would want to use selective receive and what to do to make sure it > doesn't cause you a problem. > > collectResponses(QueryRef) -> > collectResponses(QueryRef, []). > > collectResponses(QueryRef, Responses) -> > receive > {responseData, QueryRef, _UserId, Results} -> > collectResponses(QueryRef, [Responses | Results]) > after 100 -> Responses > end. > > Again, my hokey example collects results as long as they are present > or no new ones show up for 100 milliseconds. > > What we have so far is a single request message sent to 5 processes > and a function which implements selective receive to collect only the > messages that are in response to the initial request from a variety of > responders (hopefully all, but not if some are slow to respond). > > What happens if we have a slow responding database, but it does > actually produce results after 1/2 second. It was too slow to be > collected > but it puts messages on the queue anyway. If we have no mechanism > to clear them, they will build up and cause things to gradually slow > down. > > So at some higher level we need the following code: > > main() -> > receive > %% Throw away late arriving results from a previous request > {cleanup, QueryRef} -> dumpOldResults(QueryRef); > {userRequest, UserId} -> doUserQuery(UserId) > end, > main(). > > dumpOldQueryResults(QueryRef) -> > receive > {responseData, QueryRef, _UserId, _Results} -> > dumpOldQueryResults(QueryRef) > after 0 -> ok > end. > > In the main function, we give priority to cleaning up old messages. > This will keep the queue short, however, it ensures a full queue > scan for every user request. As long as the queue is short, that > won't hurt us. Dumping old messages just cycles as fast as it can > accepting messages that have our unique token and ignoring the > rest of the data in the message. If there are no clean up messages > remaining, we than accept a new user request (which will necessarily > cause the message queue to grow for a short period) and display the > results. > > What did we see? Selective receive used in 3 different ways: > > 1) To collect the results of a request (a two-way session conversation) > 2) To handle self notifications for maintenance + user requests > 3) To handle old messages from an expired session > > It turns out the {cleanup, QueryRef} message is not necessary in > the above example and we can just consume all {responseData, ...} > messages inside main(), but it depends on how new requests are > placed on the queue and whether timing allows two requests to > be interleaved in the results set (you don't want to remove all the > responseData for a pending request that has not had time to collect > results yet). Structuring as above gave more explicit different uses > of selective receive. > > The problem remaining in the code above is that there is no > "catch all" clause. Do we worry about that? It depends on how the > system evolves. If you interface to a known protocol and you have > covered all the messages supported via selective receive, then > you could do without a catch all. If your system is evolving or there > are other processes or programmers who might inject new message > types, you need a catch all in the main/0 function (although you have > to be careful not consume something that should stay on the queue). > > I have not tried this code, nor have I typed it into a erl prompt, so I > can't guarantee it even compiles. Mostly it should give you ideas > about ways to use selective receive. > > What if we didn't have selective receive? I see two choices: > > 1) Start a thread and open a new socket to the databases for each > user request. Maintain the conversations as independent channels. > > 2) Create a hash table of messages received related to each request. > This requires managing the conversation correlations yourself. > > Both of these approaches are much more code than selective receive > requires and the complexity of concepts does not increase, so selective > receive is a better approach and a useful feature of erlang. > > Is there a better way to manage the conversations rather than the whole > cleanup back channel messaging? > > If you can spawn a new process for each request, the responses will > go to privately owned message queues. When enough responses, or > enough time has passed, the newly spawned request process returns > its results and terminates. Any messages stuck on the queue are > eliminated. Any future messages are silently discarded since there is > no process to receive them. If the backend DB process were monitoring > the request process, it could even interrupt its response to discard the > results rather than waiting for processing to complete and pass them > on to a non-existent process. > > With erlang, there are many architectural choices when you consider > the uses of messaging and selective receive. > > jay > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay@REDACTED Mon Jun 2 06:01:18 2008 From: jay@REDACTED (Jay Nelson) Date: Sun, 1 Jun 2008 21:01:18 -0700 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <6c2563b20806012035g3e4e7568ld1a89aa6527f14a4@mail.gmail.com> References: <7FD49941-69F7-4D4E-B0EE-F4D1104059C3@duomark.com> <6c2563b20806012035g3e4e7568ld1a89aa6527f14a4@mail.gmail.com> Message-ID: <294F2A0A-489B-4022-B10D-7C05E585C469@duomark.com> On Jun 1, 2008, at 8:35 PM, Edwin Fine wrote: > Jay, > > Thanks for a very detailed and informative response. Although it > obviously depends on circumstances, I feel that, given Erlang's > extremely fast process creation time and small process size, I > would first consider your last option, namely, to create an > individual process per request, and use an ETS table to coordinate > responses. If there are very many responses to be collected for > each request, I would intuitively imagine in my "Erlang newbie fog" > that using an ETS table with its constant-time performance and no- > garbage-collection characteristics would be better on average than > using selective receive, which I understand has to do a linear scan > and move unprocessed messages to another area. Of course, intuition > often does not stand up to the reality of performance measurements, > so it would be interesting to see a benchmark of the various > architectural options you have described, perhaps as a function of > response time vs. request rate. If you spawn a separate process for each, there is no need for an ets table. Just have the process send the results back "en masse". Dying PID's final message: Caller ! {responseData, QueryRef, AllTheDataAssembledAsNeeded} The caller's main loop can just: receive {responseData, QueryRef, Results} -> do_something(Results) end. If you need to pass it back to another process, just arrange: QueryRef = {make_ref(), UltimatePidToSendResults} Then the receive pattern above can become: {responseData, {Ref, EndPid}, Results} -> EndPid ! {response, self(), Results} In erlang you find that you lose code where in other languages you must add code. You don't check for errors, just code like it will succeed. Don't reconstruct structured data as a hash table or tree when the message can be tagged and returned to you with the correct classification as you knew it to start with. jay From bjorn@REDACTED Mon Jun 2 09:32:48 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 02 Jun 2008 09:32:48 +0200 Subject: [erlang-questions] eep: multiple patterns In-Reply-To: References: <200805141620.19559.bekesa@sch.bme.hu> <4836A517.2040909@gmail.com> <200805271436.19305.bekesa@sch.bme.hu> <6BA6A9E9-EF89-4FE6-B683-05F3866E9E09@cs.otago.ac.nz> <483D0CB6.2080103@ericsson.com> <483FF500.3020101@ericsson.com> Message-ID: Sean Hinde writes: > You are right. > > I had thought that there was an optimisation where if the pattern was > identical to last time the same receive was entered for that process > then only newer messages were examined. No, there is only an optimization within the same receive: if the receive is waiting because none of the messages in the message queue matched the pattern, when new messages are sent to the process, only those messages will be matched. > Seems like it could be a useful optimisation in any case :-) Yes, but I think it would be a lot of work in both the compiler and the run-time system. The trick is to avoid slowing down receives in general when selective receives are not used. /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From mats.cronqvist@REDACTED Mon Jun 2 09:40:06 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Mon, 02 Jun 2008 09:40:06 +0200 Subject: [erlang-questions] erlang-questions Digest, Vol 12, Issue 98 In-Reply-To: <8f24f4b10805300742r3cda023bi23f0d6fe3b398590@mail.gmail.com> References: <8f24f4b10805300742r3cda023bi23f0d6fe3b398590@mail.gmail.com> Message-ID: <4843A3D6.804@gmail.com> John Haugeland wrote: > Forgive my being naive. [...] > > loop(State) -> > receive RcvState -> if > isa_quit(RcvState) -> quit(); > isa_connect(RcvState) -> loop(conn(State)); > isa_disconnect(RcvState) -> loop(disc(State)); > isa_show_about(RcvState) -> loop(show_about(State)); > isa_hide_about(RcvState) -> loop(hide_about(State)); > isa_data(RcvState) -> loop(update(State, RcvState)); > X -> io:fwrite("got ~p~n", [X]), loop(State) > end end. > > Everything's still named and abstracted, easier to maintain, update > and repair, but this doesn't require new syntax or compiler > alterations. So, what am I missing? that it won't compile? if lists:member(a,[a,b])->ok end. ** 1: illegal guard expression ** mats From bjorn@REDACTED Mon Jun 2 09:45:50 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 02 Jun 2008 09:45:50 +0200 Subject: [erlang-questions] how: Building Core Erlang abstract syntax trees In-Reply-To: <3dbc6d1c0806010820x50aa653fo32946bd4996b405a@mail.gmail.com> References: <4841A212.5070800@it.uu.se> <3dbc6d1c0806010820x50aa653fo32946bd4996b405a@mail.gmail.com> Message-ID: > 2008/5/31 Richard Carlsson : > > > > This is for historical reasons. When we started working on Core Erlang, > > Robert (and others) preferred working directly with records, while I > > preferred using a "proper" abstract datatype (even though it meant > > that I couldn't decompose structures using pattern matching). Also, > > we had some different ideas regarding how some things such as literals > > and variables ought to be represented. So you'll find that the central > > parts of the compiler use the records defined in core_parse.hrl, while > > cerl_inline and everything in lib/hipe/cerl uses the cerl.erl ADT (which > > can be converted to/from the record format). In a future release we will probably unify the the Core Erlang formats, so that the cerl functions can be used directly on the records. Also, already in R12B, some of the semantic differences between the Core Erlang formats have been removed. For instance, literals are now represented in the same way (that was needed to implement the constant/literal pool in R12B). /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From mikael@REDACTED Mon Jun 2 09:50:29 2008 From: mikael@REDACTED (Mikael Lixenstrand) Date: Mon, 2 Jun 2008 09:50:29 +0200 Subject: [erlang-questions] gen_sctp multi-homing In-Reply-To: References: <483E92A2.9040606@gmail.com> Message-ID: Still haven't figured out what the problem might be. I try another time here I tried this in the erlang shell with a running program with gen_sctp:open([{ip, {10,80,250,112}},{ip, {10,70,128,96}} ,{port,33333}] ++ ?sctp_options), {ok, Socket} = gen_sctp:open([{ip, {10,30,40,50}}]). {ok, Assoc1} =gen_sctp:connect(Socket,{10,80,250,112},33333,[], 6000). {ok,{sctp_assoc_change,comm_up,0,10,10,1}} {ok, Asso21} = gen_sctp:connect(Socket,{10,70,128,96},33333,[], 6000). =ERROR REPORT==== 2-Jun-2008::08:52:19 === Error in process <0.38.0> on node 'asd@REDACTED' with exit value: {{badmatch,{error,eisconn}},[{erl_eval,expr,3}]} ** exited: {{badmatch,{error,eisconn}},[{erl_eval,expr,3}]} ** Someone have any idea what the problem might be? I'm using R11B-5. 2008/5/29, Mikael Lixenstrand : > > I try this but get Error Reason: eisconn (The socket is already connected.) > > Could i have done anything else wrong? > > gen_sctp:open(lists:merge(IP,?sctp_client_options)) of > {ok, Socket} -> > AssocList = lists:foldl(fun(IP, AccList) -> > case gen_sctp:connect(Socket, IP, Port,[], > ?sctp_connect_timeout) of > {ok, Assoc} -> > [Assoc | AccList]; > {error, Reason} -> > AccList > end > end, [], IPList), > > thanks for the help > > > > 2008/5/29, Serge Aleynikov : >> >> You can call gen_sctp:connect/4 multiple times. Note that this call >> doesn't create new sockets but establishes a new association on a given >> socket. >> >> Serge >> >> Mikael Lixenstrand wrote: >> >>> I have some questions regarding gen_sctp and multihoming. It?s possible >>> to >>> create a socket to multiple local peers but i haven't figured out how to >>> connect a socket to multiple IPs on another host. >>> >>> Is this possible with gen_sctp or would i need to implement something >>> using >>> more sockets? >>> I would like to have one socket with multiple associations, one for each >>> peer on remote host. >>> >>> /mikael >>> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >> >> > > > -- > **************************** > Mikael Lixenstrand > Sl?ttadammsgatan 9 a 12 > 417 27 G?teborg > mob : 0707 593809 > icq :169264851 > msn: lixen@REDACTED > **************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From hakan@REDACTED Mon Jun 2 11:52:00 2008 From: hakan@REDACTED (Hakan Mattsson) Date: Mon, 2 Jun 2008 11:52:00 +0200 (CEST) Subject: [erlang-questions] Mnesia load and unload tables at runtime? In-Reply-To: References: Message-ID: On Sat, 31 May 2008, Jong Hian Zin wrote: JHZ> Is there a way that I can load a Mnesia ram_copies table during JHZ> runtime from file dumped by dump_tables, and "unload" it when I JHZ> dont need it, in order to save ram? No. The mnesia:dump_tables/1 function is an ancient function that ought to be avoided. It is kept in the API due to backwards compatibility reasons. You can however take a backup of a subset of your tables, empty them (or delete them) and later restore the tables when you need them. /H?kan --- H?kan Mattsson (uabhams) Erlang/OTP, Ericsson From raimo+erlang-questions@REDACTED Mon Jun 2 12:10:08 2008 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Mon, 2 Jun 2008 12:10:08 +0200 Subject: [erlang-questions] Password reminders Message-ID: <20080602101008.GA6851@erix.ericsson.se> The erlang.org mailing list's configurations have changed to not send password reminders any more, since it is a security risc to do so. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From mats.cronqvist@REDACTED Mon Jun 2 12:59:11 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Mon, 02 Jun 2008 12:59:11 +0200 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> Message-ID: <4843D27F.3080406@gmail.com> Ulf Wiger wrote: > I would really like to discourage people from avoiding > selective receive because it's "expensive". It can be > expensive on very large message queues, but this is > a pretty rare error condition, and fairly easily observable. > i think the "issue" of how the emu deals with huge in-queues is pretty uninteresting. in my personal experience, every single time this has come up the real problem has turned out to be lack of proper flow control (typically using {active,true} sockets). having 100k messages in an in-queue is not a realistic use case. the fact that this is not, afaik, particularly well documented is of course a problem. mats From saleyn@REDACTED Mon Jun 2 14:11:32 2008 From: saleyn@REDACTED (Serge Aleynikov) Date: Mon, 02 Jun 2008 08:11:32 -0400 Subject: [erlang-questions] gen_sctp multi-homing In-Reply-To: References: <483E92A2.9040606@gmail.com> Message-ID: <4843E374.8080807@gmail.com> Mikael, I suspect that the issue is that the SCTP implementation in the OS kernel doesn't allow for this. Traditionally in multi-homing setup if a client is multi-homed, it notifies the server about all of its IP addresses in the INIT messages' parameter list during connection establishment. Likewise, the server provides all of its IP addresses to the client in the INIT-ACK message. Therefore the client is only required to know one IP address of the server to establish a multi-homed connection. So I believe I confused you with my previous wrongful email. The way to have a multi-homed setup would be to open a socket on the server side using: {ok, S} = gen_sctp:open([{ip, IP1}, {ip, IP2},{port,Port}, ...]), ok = gen_sctp:listen(S, true) Open a multi-homed client socket: {ok, S} = gen_sctp:open([{ip, {10,80,250,112}}, {ip, {10,70,128,96}},{port,33333}, ...]), And then on the client do: {ok, Assoc} = gen_sctp:connect(S, Host, Port, Options), You should be able to verify with eithereal/sharkwire that the INIT and INIT-ACK indeed carry all the intended IPs. At the set-up of an association one of the IP addresses from the returned list is selected as initial primary path that used as default for transmitting data until there's a connectivity issue, in which case SCTP fails over to the alternative path. Regards, Serge Mikael Lixenstrand wrote: > Still haven't figured out what the problem might be. I try another time here > > I tried this in the erlang shell with a running program with > gen_sctp:open([{ip, {10,80,250,112}},{ip, {10,70,128,96}} ,{port,33333}] ++ > ?sctp_options), > > {ok, Socket} = gen_sctp:open([{ip, {10,30,40,50}}]). > {ok, Assoc1} =gen_sctp:connect(Socket,{10,80,250,112},33333,[], 6000). > {ok,{sctp_assoc_change,comm_up,0,10,10,1}} > > {ok, Asso21} = gen_sctp:connect(Socket,{10,70,128,96},33333,[], 6000). > > =ERROR REPORT==== 2-Jun-2008::08:52:19 === > Error in process <0.38.0> on node 'asd@REDACTED' with exit value: > {{badmatch,{error,eisconn}},[{erl_eval,expr,3}]} > > ** exited: {{badmatch,{error,eisconn}},[{erl_eval,expr,3}]} ** > > Someone have any idea what the problem might be? > I'm using R11B-5. > > 2008/5/29, Mikael Lixenstrand : >> I try this but get Error Reason: eisconn (The socket is already connected.) >> >> Could i have done anything else wrong? >> >> gen_sctp:open(lists:merge(IP,?sctp_client_options)) of >> {ok, Socket} -> >> AssocList = lists:foldl(fun(IP, AccList) -> >> case gen_sctp:connect(Socket, IP, Port,[], >> ?sctp_connect_timeout) of >> {ok, Assoc} -> >> [Assoc | AccList]; >> {error, Reason} -> >> AccList >> end >> end, [], IPList), >> >> thanks for the help >> >> >> >> 2008/5/29, Serge Aleynikov : >>> You can call gen_sctp:connect/4 multiple times. Note that this call >>> doesn't create new sockets but establishes a new association on a given >>> socket. >>> >>> Serge >>> >>> Mikael Lixenstrand wrote: >>> >>>> I have some questions regarding gen_sctp and multihoming. It?s possible >>>> to >>>> create a socket to multiple local peers but i haven't figured out how to >>>> connect a socket to multiple IPs on another host. >>>> >>>> Is this possible with gen_sctp or would i need to implement something >>>> using >>>> more sockets? >>>> I would like to have one socket with multiple associations, one for each >>>> peer on remote host. >>>> >>>> /mikael >>>> >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>> >>> >> >> -- >> **************************** >> Mikael Lixenstrand >> Sl?ttadammsgatan 9 a 12 >> 417 27 G?teborg >> mob : 0707 593809 >> icq :169264851 >> msn: lixen@REDACTED >> **************************** > From spkhaira@REDACTED Mon Jun 2 14:52:52 2008 From: spkhaira@REDACTED (Sant Parkash Singh) Date: Mon, 2 Jun 2008 18:22:52 +0530 Subject: [erlang-questions] Newbie Orber-ssl question Message-ID: <5ab5b8820806020552q3b6f1209m493cab8c7600103b@mail.gmail.com> Hello all, I am very new to Erlang and want to write clients in Erlang for my C++ CORBA servers which are working on SSLIOP (SSL over IIOP). I am trying to initialize Orber with security on windows machine (XP). For this I followed the steps mentioned in Orber User Guide ... but facing some problems ... I am trying the following commands from Erlang shell - 1) mnesia:start(). ok 2) corba:orb_init([{domain, "MyRAMSingleNodeORB"}, {secure,ssl}, {ssl_server_verify,0}, {ssl_client_cacertfile,"C:/WINDOWS/AvayaCerts/ProactiveContactCA.pem"}, {ssl_client_certfile,"C:/WINDOWS/AvayaCerts/corbaServer_cert.pem"}, {ssl_client_keyfile,"C:/WINDOWS/AvayaCerts/corbaServer_key.pem"}, {ssl_server_certfile,"C:/WINDOWS/AvayaCerts/corbaServer_cert.pem"}, {ssl_server_cacertfile, "C:/WINDOWS/AvayaCerts/ProactiveContactCA.pem"}, {ssl_server_keyfile, "C:/WINDOWS/AvayaCerts/corbaServer_key.pem"}]). ok 3) orber:install([node()], [{ifr_storage_type, ram_copies}]). ok 4) orber:start(). {error,"Unable to start Orber. Is the listen port vacant?"} =INFO REPORT==== 2-Jun-2008::17:58:57 === application: orber exited: {shutdown,{orber,start,[normal,[]]}} type: temporary Can anyone please help me with this? Many thanks in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nick@REDACTED Mon Jun 2 15:25:51 2008 From: nick@REDACTED (Niclas Eklund) Date: Mon, 2 Jun 2008 15:25:51 +0200 (MEST) Subject: [erlang-questions] Newbie Orber-ssl question In-Reply-To: <5ab5b8820806020552q3b6f1209m493cab8c7600103b@mail.gmail.com> Message-ID: Hello! Add this option {orber_debug_level, 10} to the corba:orb_init(..) function to get more detailed debug information and send the result. Best regards, Niclas (Erlang/OTP) On Mon, 2 Jun 2008, Sant Parkash Singh wrote: > Hello all, > > I am very new to Erlang and want to write clients in Erlang for my C++ CORBA > servers which are working on SSLIOP (SSL over IIOP). > > I am trying to initialize Orber with security on windows machine (XP). For > this I followed the steps mentioned in Orber User Guide ... > but facing some problems ... > > I am trying the following commands from Erlang shell - > > 1) mnesia:start(). > > ok > > 2) corba:orb_init([{domain, "MyRAMSingleNodeORB"}, {secure,ssl}, > {ssl_server_verify,0}, > {ssl_client_cacertfile,"C:/WINDOWS/AvayaCerts/ProactiveContactCA.pem"}, > {ssl_client_certfile,"C:/WINDOWS/AvayaCerts/corbaServer_cert.pem"}, > {ssl_client_keyfile,"C:/WINDOWS/AvayaCerts/corbaServer_key.pem"}, > {ssl_server_certfile,"C:/WINDOWS/AvayaCerts/corbaServer_cert.pem"}, > {ssl_server_cacertfile, "C:/WINDOWS/AvayaCerts/ProactiveContactCA.pem"}, > {ssl_server_keyfile, "C:/WINDOWS/AvayaCerts/corbaServer_key.pem"}]). > > ok > > 3) orber:install([node()], [{ifr_storage_type, ram_copies}]). > > ok > > 4) orber:start(). > > {error,"Unable to start Orber. Is the listen port vacant?"} > > =INFO REPORT==== 2-Jun-2008::17:58:57 === > application: orber > exited: {shutdown,{orber,start,[normal,[]]}} > type: temporary > > > > Can anyone please help me with this? > Many thanks in advance. > From chandrashekhar.mullaparthi@REDACTED Mon Jun 2 15:31:05 2008 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Mon, 2 Jun 2008 14:31:05 +0100 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <4843D27F.3080406@gmail.com> References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <4843D27F.3080406@gmail.com> Message-ID: 2008/6/2 Mats Cronqvist : > Ulf Wiger wrote: > > I would really like to discourage people from avoiding > > selective receive because it's "expensive". It can be > > expensive on very large message queues, but this is > > a pretty rare error condition, and fairly easily observable. > > > > i think the "issue" of how the emu deals with huge in-queues is pretty > uninteresting. > in my personal experience, every single time this has come up the real > problem has turned out to be lack of proper flow control (typically > using {active,true} sockets). having 100k messages in an in-queue is not > a realistic use case. > the fact that this is not, afaik, particularly well documented is of > course a problem. > This is true - but if one has no prior experience of this situation, it is hard to understand why a system is behaving sluggishly. What will be nice is having an option, as Ulf suggested earlier, to have bounded message queues (kill the process if the message queue length exceeds a certain value). That way, flow control problems will be more readily visible to users. In real life situations, when a process gets into this state, the only way to fix it is to kill that process as it will probably never catch up. This has been discussed before: http://www.erlang.org/pipermail/erlang-questions/2006-January/018364.html It also fits in well with the "Let it crash" philosophy. cheers Chandru -------------- next part -------------- An HTML attachment was scrubbed... URL: From masterofquestions@REDACTED Mon Jun 2 16:55:56 2008 From: masterofquestions@REDACTED (db) Date: Mon, 2 Jun 2008 10:55:56 -0400 Subject: [erlang-questions] cacherl memcached app startup problem question In-Reply-To: <14f0e3620805310300t175e4ea0wb539e3c8f1a84786@mail.gmail.com> References: <1218d6a50805301550u50ddc883l4e07607c2e70f760@mail.gmail.com> <14f0e3620805310300t175e4ea0wb539e3c8f1a84786@mail.gmail.com> Message-ID: <1218d6a50806020755w263cb8a5o49f489ed2cc8b53c@mail.gmail.com> I changed the disc_copies to ram_copies. Seem to work now. I have another question about tcp listener on 0.0.0.0:11311 =INFO REPORT==== 30-May-2008::17:16:51 === started TCP listener on 0.0.0.0:11311 Why is tcp listener on 0.0.0.0:11311 even though cacherl's memcached is on XX.XX.XX.XX:11311? -- rk That which we persist in doing becomes easier for us to do; not that the nature of the thing itself is changed, but that our power to do is increased. -Ralph Waldo Emerson From masterofquestions@REDACTED Mon Jun 2 17:13:57 2008 From: masterofquestions@REDACTED (db) Date: Mon, 2 Jun 2008 11:13:57 -0400 Subject: [erlang-questions] cacherl memcached app startup problem question In-Reply-To: <1218d6a50806020755w263cb8a5o49f489ed2cc8b53c@mail.gmail.com> References: <1218d6a50805301550u50ddc883l4e07607c2e70f760@mail.gmail.com> <14f0e3620805310300t175e4ea0wb539e3c8f1a84786@mail.gmail.com> <1218d6a50806020755w263cb8a5o49f489ed2cc8b53c@mail.gmail.com> Message-ID: <1218d6a50806020813u6d5550e6oba58925cc8cc4b45@mail.gmail.com> Ignore the previous message I was too hasty. On Mon, Jun 2, 2008 at 10:55 AM, db wrote: > I changed the disc_copies to ram_copies. Seem to work now. I have > another question about tcp listener on 0.0.0.0:11311 > > =INFO REPORT==== 30-May-2008::17:16:51 === > started TCP listener on 0.0.0.0:11311 > > Why is tcp listener on 0.0.0.0:11311 even though cacherl's memcached > is on XX.XX.XX.XX:11311? > > > -- > rk > > That which we persist in doing becomes easier for us to do; not that > the nature of the thing itself is changed, but that our power to do is > increased. > -Ralph Waldo Emerson > -- rk That which we persist in doing becomes easier for us to do; not that the nature of the thing itself is changed, but that our power to do is increased. -Ralph Waldo Emerson From mberrow1@REDACTED Mon Jun 2 16:35:47 2008 From: mberrow1@REDACTED (Mike Berrow) Date: Mon, 2 Jun 2008 07:35:47 -0700 Subject: [erlang-questions] Ruby and Erlang in the cloud Message-ID: <07b501c8c4bd$f5bc23b0$6401a8c0@rubicon> At RailsConf 2008 this weekend Ezra Zygmuntowicz of EngineYard (Merb, Rubinius) announced Vertebra, an Ejabberd-based cloud-control framework. It created quite a stir. Vertebra slides are here http://tinyurl.com/6gyfhd The diagram on Vertebra slide 42 shows ruby/erlang/ejabberd/mnesia integration A month ago, EngineYard posted that they were looking for "one or two kick ass Erlang programmers" http://brainspl.at/articles/2008/05/01/engine-yard-needs-erlang-programmers -- Mike Berrow From rvirding@REDACTED Mon Jun 2 17:45:40 2008 From: rvirding@REDACTED (Robert Virding) Date: Mon, 2 Jun 2008 17:45:40 +0200 Subject: [erlang-questions] LFE presentations Message-ID: <3dbc6d1c0806020845g12596bd7qeb3b5ddd6329a674@mail.gmail.com> Here are the overheads to the presentation I gave at a recent Erlang Stockholm Usergroup meeting: http://forum.trapexit.org/viewtopic.php?p=43941#43941 Francesco filmed it, but I don't know what has become of the film. Those interested will have to hound him for it. :-) Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.thilani@REDACTED Mon Jun 2 17:46:23 2008 From: a.thilani@REDACTED (Thilani Abeysinghe) Date: Mon, 2 Jun 2008 21:16:23 +0530 Subject: [erlang-questions] use erlSoap package in Erlang Message-ID: Hi All I want to use ErlSoap package for sending SOAP messages using Erlang. Can you please tell me how to configure the Erlsoap package.I use windows platform Thanks in Advance Regards Thilani Computer Science and Engineering University of Moratuwa Sri Lanka http://www.thilani.blogspot.com http://www.cgmax.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From stonecypher@REDACTED Mon Jun 2 19:09:28 2008 From: stonecypher@REDACTED (John Haugeland) Date: Mon, 2 Jun 2008 11:09:28 -0600 Subject: [erlang-questions] erlang-questions Digest, Vol 12, Issue 98 In-Reply-To: <4d08db370806020040h462dfadbw9763c11f92e71e5f@mail.gmail.com> References: <8f24f4b10805300742r3cda023bi23f0d6fe3b398590@mail.gmail.com> <4d08db370806020040h462dfadbw9763c11f92e71e5f@mail.gmail.com> Message-ID: <8f24f4b10806021009u12c56b6btc92da5bdc747b2f8@mail.gmail.com> > > 1. You don't tried to compile it (illegal guard expression). > Yeah, well, it's easily fixed. I'm at work and I don't have an erlang shell here. > 2. It can't be selective receive. > Why not? isa_foo(Bar) -> case Bar of { foo, _, ThingToSelect } -> { true, ThingToSelect }; _ -> false end. etc - John -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul-trapexit@REDACTED Mon Jun 2 19:17:27 2008 From: paul-trapexit@REDACTED (Paul Mineiro) Date: Mon, 2 Jun 2008 10:17:27 -0700 (PDT) Subject: [erlang-questions] how: prefix matching on external term format Message-ID: i'm now ready to attack my main use case for a disk-based ordered_set erlang term store, namely, prefix matching on tuple/list keys. i want to be able to look up a match head such as { Bound, '_' } or [ Bound | '_' ] very quickly. (this is what we do all day at my job and we are currently stuck with ets and we'd rather store some of this data on disk). my tree consists of erlang terms in external format and sorted according to erlang term order (via erl_compare_ext). unfortunately, tokyocabinet does not export a search interface[1], only a prefix-matching operation and a range operation. so i've envisioned the following choices: 0. constructing the bytes corresponding to the prefix of the external encoding up to the first variable encountered doesn't work; the prefix of an externally formatted term is not a valid encoding (ever?), and erl_compare_ext can potentially march along doing uninitialized reads (erl_compare_ext doesn't take a length parameter for the buffers, it's very trusting of the input). 1. there is no smallest or largest erlang term, unfortunately. however i can pick a "very small" and "very large" erlang term, replace variables in the match head with such terms, query the tree using range, and then search at both boundaries in case "very small" and "very large" are not small or large enough (in the typical case, this will just involve looking at one more record, so shouldn't be that bad). 2. i can fork erl_compare_ext and include a custom version in my code. i could take two unused tags and call them the smallest and largest erlang term, then query the tree using range (or even prefix, since i can make my custom erl_compare_ext safe to use on partial external term encodings terminated by the special smallest or largest term tag; this would make constructing the query cheaper). of course this means the driver will break everytime new type tags are added :( ... unless these special tags are officially added to the external format to make these kind of manipulations easier :) i'm interested in better ideas. for instance, my cursory inspection of the ets implementation suggests using a search interface coupled with C implementation of match specs to direct the search. if this is the way to go, i could contact the tokyocabinet maintainer and ask for a search interface. -- p [1] by search interface, i mean, a way to search the tree according to a user-specified function (which of course has to respect the existing ordering of keys in the tree, in this case erlang term order). In an artificial world, only extremists live naturally. -- Paul Graham From vladdu55@REDACTED Mon Jun 2 23:24:32 2008 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Mon, 2 Jun 2008 23:24:32 +0200 Subject: [erlang-questions] [ANN] Erlide 0.3.54 Message-ID: <95be1d3b0806021424r3feb8ed8wd0047132386da3e7@mail.gmail.com> Hi all, A new release of Erlide is available. We have a lot of improvements, mainly regarding: - stability - performance (compiling, parsing) - navigation (go to function/macro definition/included file) - indenting code - preferences for many settings were added - several bugs were fixed - I'm sure I'm forgetting some nice features too, but on the other hand mentioning everything would shrink the pleasure of the discovery :-) Update site is at http://erlide.sourceforge.net/update or you can download from http://sourceforge.net/project/platformdownload.php?group_id=58889&sel_platform=4078 If you encounter problems, go to Window->Preferences->Erlang and press the "" button. Fill the description of the issue and the logs from the latest session will be attached to a trouble report that you can retrieve from your home directory (the file is named _.txt). Please attach this file when reporting (I'll try to make it submit it automatically, but there are some OS issues) Enjoy! /Vlad -- .......__o .......\<, ....( )/ ( )... -------------- next part -------------- An HTML attachment was scrubbed... URL: From dnew@REDACTED Mon Jun 2 23:15:02 2008 From: dnew@REDACTED (Darren New) Date: Mon, 02 Jun 2008 14:15:02 -0700 Subject: [erlang-questions] Performance of matches In-Reply-To: <4841DBFB.2000308@san.rr.com> References: <483EDB2E.9070103@san.rr.com> <484126E2.5040000@adam.com.au> <4841DBFB.2000308@san.rr.com> Message-ID: <484462D6.8000001@san.rr.com> Darren New wrote: > David Lloyd wrote: > >> Is either of those O(1) lookup time? > > It strikes me that this is something that is eminently easy to profile... > > Good point. Actually, I was asking a slightly more complex question. It's only easy to profile if the compiler doesn't have special cases that make some things faster. For example, Haskell has enough smarts to notice some operations never look at the unmodified value after you modify it, and use that to avoid copying the whole collection (for example) when you modify one element. Much the same way I've read Erlang avoids copying binaries when you append to them in certain circumstances. So, basically, I was trying to ask the question "is there any O(1) data structure in Erlang, such as a hashtable or an array equivalent?" Maybe it just hasn't clicked for me, is all. Good to know the process dictionary is actually a hashtable, but there's only one per process and they don't nest, so that's pretty limited. Plus there are recommendations to avoid it, for some reason that isn't explained. -- Darren New / San Diego, CA, USA (PST) "That's pretty. Where's that?" "It's the Age of Channelwood." "We should go there on vacation some time." From masterofquestions@REDACTED Tue Jun 3 03:19:04 2008 From: masterofquestions@REDACTED (db) Date: Mon, 2 Jun 2008 21:19:04 -0400 Subject: [erlang-questions] cacherl memcached app startup problem question In-Reply-To: <1218d6a50806020813u6d5550e6oba58925cc8cc4b45@mail.gmail.com> References: <1218d6a50805301550u50ddc883l4e07607c2e70f760@mail.gmail.com> <14f0e3620805310300t175e4ea0wb539e3c8f1a84786@mail.gmail.com> <1218d6a50806020755w263cb8a5o49f489ed2cc8b53c@mail.gmail.com> <1218d6a50806020813u6d5550e6oba58925cc8cc4b45@mail.gmail.com> Message-ID: <1218d6a50806021819m354f65vddcdd659d40b90f4@mail.gmail.com> Any reason why memcached_client doesn't use a gen_server? Seems like I have to manually keep track of the states. -- rk That which we persist in doing becomes easier for us to do; not that the nature of the thing itself is changed, but that our power to do is increased. -Ralph Waldo Emerson From rapsey@REDACTED Tue Jun 3 06:12:46 2008 From: rapsey@REDACTED (Rapsey) Date: Tue, 3 Jun 2008 06:12:46 +0200 Subject: [erlang-questions] Performance of matches In-Reply-To: <484462D6.8000001@san.rr.com> References: <483EDB2E.9070103@san.rr.com> <484126E2.5040000@adam.com.au> <4841DBFB.2000308@san.rr.com> <484462D6.8000001@san.rr.com> Message-ID: <97619b170806022112x73792e29y5a612cb966049305@mail.gmail.com> Actually there is an array: http://erlang.org/doc/man/array.html It was added in R12B I think. On Mon, Jun 2, 2008 at 11:15 PM, Darren New wrote: > Darren New wrote: > > David Lloyd wrote: > > >> Is either of those O(1) lookup time? > > > It strikes me that this is something that is eminently easy to > profile... > > > > Good point. > > Actually, I was asking a slightly more complex question. It's only easy > to profile if the compiler doesn't have special cases that make some > things faster. For example, Haskell has enough smarts to notice some > operations never look at the unmodified value after you modify it, and > use that to avoid copying the whole collection (for example) when you > modify one element. Much the same way I've read Erlang avoids copying > binaries when you append to them in certain circumstances. > > So, basically, I was trying to ask the question "is there any O(1) data > structure in Erlang, such as a hashtable or an array equivalent?" Maybe > it just hasn't clicked for me, is all. > > Good to know the process dictionary is actually a hashtable, but there's > only one per process and they don't nest, so that's pretty limited. Plus > there are recommendations to avoid it, for some reason that isn't > explained. > > -- > Darren New / San Diego, CA, USA (PST) > "That's pretty. Where's that?" > "It's the Age of Channelwood." > "We should go there on vacation some time." > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.kleberg@REDACTED Tue Jun 3 07:01:55 2008 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Tue, 03 Jun 2008 07:01:55 +0200 Subject: [erlang-questions] Performance of matches In-Reply-To: <484462D6.8000001@san.rr.com> References: <483EDB2E.9070103@san.rr.com> <484126E2.5040000@adam.com.au> <4841DBFB.2000308@san.rr.com> <484462D6.8000001@san.rr.com> Message-ID: <1212469315.14971.18.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> Greetings, Reasons to avoid the process dictionary: 1 It is using side effects which makes your code harder to understand/reason about. 2 It is per process which makes your code harder to test. bengt On Mon, 2008-06-02 at 14:15 -0700, Darren New wrote: > Darren New wrote: > > David Lloyd wrote: > > >> Is either of those O(1) lookup time? > > > It strikes me that this is something that is eminently easy to profile... > > > > Good point. > > Actually, I was asking a slightly more complex question. It's only easy > to profile if the compiler doesn't have special cases that make some > things faster. For example, Haskell has enough smarts to notice some > operations never look at the unmodified value after you modify it, and > use that to avoid copying the whole collection (for example) when you > modify one element. Much the same way I've read Erlang avoids copying > binaries when you append to them in certain circumstances. > > So, basically, I was trying to ask the question "is there any O(1) data > structure in Erlang, such as a hashtable or an array equivalent?" Maybe > it just hasn't clicked for me, is all. > > Good to know the process dictionary is actually a hashtable, but there's > only one per process and they don't nest, so that's pretty limited. Plus > there are recommendations to avoid it, for some reason that isn't > explained. > From nick@REDACTED Tue Jun 3 07:52:04 2008 From: nick@REDACTED (Nick Gerakines) Date: Mon, 2 Jun 2008 22:52:04 -0700 Subject: [erlang-questions] Debugging float math in geohashing module Message-ID: I've taken an interest in the concept of the geohash [1] and wrote a simple module [2] to encode/decode latitude/longitude pairs into these geohashes. The modules is loosely based on Geo::Hash [3] but with a number of obvious changes. So far encoding works perfectly and the same test cases from the Geo::Hash perl module pass in my module for encode operations. The decode functionality is different however. When I attempt to decode a geohash into a latitude/longitude pair the end result is slightly* off. When I say slightly off I mean that the float precision is incorrect from the latitude/longitude that was initially given. I think it has something to do with the way the floats are being divided and rounded in the process. I'm really having a rough time working around this, any help is welcome. I've tried (unsuccessfully) to work around it using big int tuples ({N, E}) but that didn't go so well. [1] http://en.wikipedia.org/wiki/Geohash [2] http://github.com/ngerakines/erlang_geohash/tree/master/geohash.erl [3] http://search.cpan.org/~andya/Geo-Hash/lib/Geo/Hash.pm # Nick Gerakines From w.a.de.jong@REDACTED Tue Jun 3 08:19:43 2008 From: w.a.de.jong@REDACTED (Willem de Jong) Date: Tue, 3 Jun 2008 08:19:43 +0200 Subject: [erlang-questions] use erlSoap package in Erlang In-Reply-To: References: Message-ID: <407d9ef80806022319u2c036e85gaa3803e17d3e8e1@mail.gmail.com> Hi Thilani, Do you need to implement a client (sending a request and receiving a response) or a server (receiving a request and sending a response)? For a server the code could look something like this: -module(minimal_soap_server). -compile(export_all). -include("sms.hrl"). % .hrl file generated by erlsom start() -> mod_soap:start(), ok = mod_soap:add_callback("/parlay_sms", "parlayx_sms_send_service_3_1.wsdl", fun(_I, _Header, Body) -> handler(Body) end). handler([#'p:sendSms'{addresses = To, message = Text}]) -> {ok, send(To, Text)}. send(To, Text) -> [#'p:sendSmsResponse'{'result' = "sending " ++ Text ++ " to: " ++ To}]. As you can see, you need to have a WSDL. You need to install erlsoap and erlsom. That should be relatively easy: download the files, put them in the right place and compile them. I think you also need to provide a simple configuration file for erlsoap. Let me know if you need more information. A simple client might look like this: -module(minimal_client). -export([sms/2]). -include("sms.hrl"). sms(To, Text) -> Wsdl = yaws_soap_lib:initModel("file://parlayx_sms_send_service_3_1.wsdl "), Sms = #'p:sendSms'{addresses = [To], senderName = "Sender", message = Text}, yaws_soap_lib:call(Wsdl, "sendSms", [], [Sms]). This uses the soap client code provided with yaws, but the erlsoap client is essentially the same. Again you will need a WSDL, and erlsom has to be installed. Good luck, Willem On 6/2/08, Thilani Abeysinghe wrote: > > Hi All > I want to use ErlSoap package for sending SOAP messages using Erlang. > Can you please tell me how to configure the Erlsoap package.I use > windows > platform > Thanks in Advance > > Regards > Thilani > > Computer Science and Engineering > University of Moratuwa > Sri Lanka > http://www.thilani.blogspot.com > http://www.cgmax.blogspot.com > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mats.cronqvist@REDACTED Tue Jun 3 08:28:22 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Tue, 03 Jun 2008 08:28:22 +0200 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <4843D27F.3080406@gmail.com> Message-ID: <4844E486.4030406@gmail.com> Chandru wrote: > 2008/6/2 Mats Cronqvist >: > > Ulf Wiger wrote: > > I would really like to discourage people from avoiding > > selective receive because it's "expensive". It can be > > expensive on very large message queues, but this is > > a pretty rare error condition, and fairly easily observable. > > > > i think the "issue" of how the emu deals with huge in-queues is > pretty > uninteresting. > in my personal experience, every single time this has come up the > real > problem has turned out to be lack of proper flow control (typically > using {active,true} sockets). having 100k messages in an in-queue > is not > a realistic use case. > the fact that this is not, afaik, particularly well documented is of > course a problem. > > > This is true - but if one has no prior experience of this situation, > it is hard to understand why a system is behaving sluggishly. What > will be nice is having an option, as Ulf suggested earlier, to have > bounded message queues (kill the process if the message queue length > exceeds a certain value). That way, flow control problems will be more > readily visible to users. true enough. mats From a.thilani@REDACTED Tue Jun 3 08:36:01 2008 From: a.thilani@REDACTED (Thilani Abeysinghe) Date: Tue, 3 Jun 2008 12:06:01 +0530 Subject: [erlang-questions] use erlSoap package in Erlang In-Reply-To: <407d9ef80806022319u2c036e85gaa3803e17d3e8e1@mail.gmail.com> References: <407d9ef80806022319u2c036e85gaa3803e17d3e8e1@mail.gmail.com> Message-ID: At the moment can you please guide me how to configure these erlsom and erlsoap packages. Normally in erlang shell I can invoke erlang modules like module_name:fundction(parameters). How to configure above packages with the existing erlang system. Thanks in advance for your kind reply Regards Thilani Computer Science and Engineering University of Moratuwa Sri Lanka http://www.thilani.blogspot.com http://www.cgmax.blogspot.com On Tue, Jun 3, 2008 at 11:49 AM, Willem de Jong wrote: > Hi Thilani, > > Do you need to implement a client (sending a request and receiving a > response) or a server (receiving a request and sending a response)? > > For a server the code could look something like this: > > -module(minimal_soap_server). > -compile(export_all). > > -include("sms.hrl"). % .hrl file generated by erlsom > > start() -> > mod_soap:start(), > ok = mod_soap:add_callback("/parlay_sms", > "parlayx_sms_send_service_3_1.wsdl", > fun(_I, _Header, Body) -> handler(Body) end). > > handler([#'p:sendSms'{addresses = To, > message = Text}]) -> > {ok, send(To, Text)}. > > send(To, Text) -> > [#'p:sendSmsResponse'{'result' = "sending " ++ Text ++ " to: " ++ To}]. > As you can see, you need to have a WSDL. > > You need to install erlsoap and erlsom. That should be relatively easy: > download the files, put them in the right place and compile them. I think > you also need to provide a simple configuration file for erlsoap. Let me > know if you need more information. > > A simple client might look like this: > > -module(minimal_client). > -export([sms/2]). > > -include("sms.hrl"). > sms(To, Text) -> > Wsdl = yaws_soap_lib:initModel("file://parlayx_sms_send_service_3_1.wsdl > "), > Sms = #'p:sendSms'{addresses = [To], > senderName = "Sender", > message = Text}, > yaws_soap_lib:call(Wsdl, "sendSms", [], [Sms]). > > This uses the soap client code provided with yaws, but the erlsoap client > is essentially the same. Again you will need a WSDL, and erlsom has to be > installed. > > Good luck, > Willem > > On 6/2/08, Thilani Abeysinghe wrote: > >> Hi All >> I want to use ErlSoap package for sending SOAP messages using Erlang. >> Can you please tell me how to configure the Erlsoap package.I use >> windows >> platform >> Thanks in Advance >> >> Regards >> Thilani >> >> Computer Science and Engineering >> University of Moratuwa >> Sri Lanka >> http://www.thilani.blogspot.com >> http://www.cgmax.blogspot.com >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan556@REDACTED Tue Jun 3 09:12:23 2008 From: johan556@REDACTED (Johan Holmberg) Date: Tue, 3 Jun 2008 09:12:23 +0200 Subject: [erlang-questions] Relocating Erlang installation on Windows? In-Reply-To: <52482.1212082932@snookles.snookles.com> References: <351ce2d70805211142v3ad7df8fj127ab5c2fc08261a@mail.gmail.com> <52482.1212082932@snookles.snookles.com> Message-ID: <351ce2d70806030012i3c846706jc30aea6b173da705@mail.gmail.com> On Thu, May 29, 2008 at 7:42 PM, Scott Lystig Fritchie wrote: > Johan Holmberg wrote: > > jh> I wonder if there is some way of getting the installation "position > jh> independent"? > > Not that I'm aware of. (I'm just catching up on my mailing list reading > and noticed that no one (?) has replied to your query.) > Thanks for your answer (yes, you were the first to answer). I think I understand what your script is doing. I have tried similar things on Windows, but there other files contain the "absolute paths" (e.g. "erl.ini"). Erlang seems to handle this stuff in totally different ways on Unix and Windows: on Unix with symlinks and shell script wrappers, and on Windows by reading the "erl.ini" files. Doesn't your script still produce an installation that is not position independent, just relocated to another place? (since one absolute path is replaced with a new one in the scripts). I would like to take this one step further to be able to move the directory tree to a new place and have it work "out of the box". For my Windows Erlang I decided to change the C source code reading the "erl.ini" file. So now I can use a symbolic value in my "erl.ini" files like this: [erlang] Bindir=%EXE_DIR%\\..\\erts-5.6.2\\bin Progname=erl Rootdir=%EXE_DIR%\\.. When "erl.exe" is started, it will replace %EXE_DIR% with the actual directory of the "erl.exe" binary. This feels like a kludge, and I had to rebuild Erlang. But it seems to work ... > We use a shell script to relocate builds of R11B-5 on Linux, which is > included below. It mostly works, but since we don't use "escript" and > some of the other scripts, I don't know if it does 100% the right thing > with all the scripts. It also doesn't do anything with Windows-only > files. Hope it helps. > > -Scott Since I found the same "absolute paths" as you, I now believe my changes have a chance of working. But like you, I'm not 100% sure. /Johan From richardc@REDACTED Tue Jun 3 09:20:35 2008 From: richardc@REDACTED (Richard Carlsson) Date: Tue, 03 Jun 2008 09:20:35 +0200 Subject: [erlang-questions] Performance of matches In-Reply-To: <97619b170806022112x73792e29y5a612cb966049305@mail.gmail.com> References: <483EDB2E.9070103@san.rr.com> <484126E2.5040000@adam.com.au> <4841DBFB.2000308@san.rr.com> <484462D6.8000001@san.rr.com> <97619b170806022112x73792e29y5a612cb966049305@mail.gmail.com> Message-ID: <4844F0C3.3020404@it.uu.se> Rapsey wrote: > Actually there is an array: > http://erlang.org/doc/man/array.html > > It was added in R12B I think. Yes, but it is a purely functional data structure, and still has O(log n) access time, although with a much better constant factor than dict or gb_trees. /Richard From exta7@REDACTED Tue Jun 3 10:29:04 2008 From: exta7@REDACTED (Zvi) Date: Tue, 3 Jun 2008 01:29:04 -0700 (PDT) Subject: [erlang-questions] Re locating Erlang installation on Windows? In-Reply-To: <351ce2d70806030012i3c846706jc30aea6b173da705@mail.gmail.com> References: <351ce2d70805211142v3ad7df8fj127ab5c2fc08261a@mail.gmail.com> <52482.1212082932@snookles.snookles.com> <351ce2d70806030012i3c846706jc30aea6b173da705@mail.gmail.com> Message-ID: <17618659.post@talk.nabble.com> Hi Johan, can you give me the pointers, how to rebuild Erlang from source on Windows? Thanks in Advance Zvi Johan Holmberg-2 wrote: > > On Thu, May 29, 2008 at 7:42 PM, Scott Lystig Fritchie > wrote: >> Johan Holmberg wrote: >> >> jh> I wonder if there is some way of getting the installation "position >> jh> independent"? >> >> Not that I'm aware of. (I'm just catching up on my mailing list reading >> and noticed that no one (?) has replied to your query.) >> > > Thanks for your answer (yes, you were the first to answer). > > I think I understand what your script is doing. I have tried similar > things on Windows, but there other files contain the "absolute paths" > (e.g. "erl.ini"). Erlang seems to handle this stuff in totally > different ways on Unix and Windows: on Unix with symlinks and shell > script wrappers, and on Windows by reading the "erl.ini" files. > > Doesn't your script still produce an installation that is not position > independent, just relocated to another place? (since one absolute > path is replaced with a new one in the scripts). I would like to take > this one step further to be able to move the directory tree to a new > place and have it work "out of the box". > > For my Windows Erlang I decided to change the C source code reading > the "erl.ini" file. So now I can use a symbolic value in my "erl.ini" > files like this: > > [erlang] > Bindir=%EXE_DIR%\\..\\erts-5.6.2\\bin > Progname=erl > Rootdir=%EXE_DIR%\\.. > > When "erl.exe" is started, it will replace %EXE_DIR% with the actual > directory of the "erl.exe" binary. This feels like a kludge, and I had > to rebuild Erlang. But it seems to work ... > >> We use a shell script to relocate builds of R11B-5 on Linux, which is >> included below. It mostly works, but since we don't use "escript" and >> some of the other scripts, I don't know if it does 100% the right thing >> with all the scripts. It also doesn't do anything with Windows-only >> files. Hope it helps. >> >> -Scott > > Since I found the same "absolute paths" as you, I now believe my > changes have a chance of working. But like you, I'm not 100% sure. > > /Johan > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > -- View this message in context: http://www.nabble.com/Relocating-Erlang-installation-on-Windows--tp17376315p17618659.html Sent from the Erlang Questions mailing list archive at Nabble.com. From als@REDACTED Tue Jun 3 10:38:13 2008 From: als@REDACTED (Anthony Shipman) Date: Tue, 3 Jun 2008 18:38:13 +1000 Subject: [erlang-questions] ASN.1 and RELATIVE-OID Message-ID: <200806031838.13764.als@iinet.net.au> I've found that the asn1 system in Erlang, up to 12B2, does not implement RELATIVE-OID. Are there any plans to develop the asn1 system further? -- Anthony Shipman Mamas don't let your babies als@REDACTED grow up to be outsourced. From gleber.p@REDACTED Tue Jun 3 11:02:40 2008 From: gleber.p@REDACTED (Gleb Peregud) Date: Tue, 3 Jun 2008 11:02:40 +0200 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <4844E486.4030406@gmail.com> References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <4843D27F.3080406@gmail.com> <4844E486.4030406@gmail.com> Message-ID: <14f0e3620806030202g6695c9edj1d49a95acbf92b5b@mail.gmail.com> On Tue, Jun 3, 2008 at 8:28 AM, Mats Cronqvist wrote: > Chandru wrote: >> 2008/6/2 Mats Cronqvist > >: >> >> Ulf Wiger wrote: >> > I would really like to discourage people from avoiding >> > selective receive because it's "expensive". It can be >> > expensive on very large message queues, but this is >> > a pretty rare error condition, and fairly easily observable. >> > >> >> i think the "issue" of how the emu deals with huge in-queues is >> pretty >> uninteresting. >> in my personal experience, every single time this has come up the >> real >> problem has turned out to be lack of proper flow control (typically >> using {active,true} sockets). having 100k messages in an in-queue >> is not >> a realistic use case. >> the fact that this is not, afaik, particularly well documented is of >> course a problem. >> >> >> This is true - but if one has no prior experience of this situation, >> it is hard to understand why a system is behaving sluggishly. What >> will be nice is having an option, as Ulf suggested earlier, to have >> bounded message queues (kill the process if the message queue length >> exceeds a certain value). That way, flow control problems will be more >> readily visible to users. > > true enough. > > mats > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > What > will be nice is having an option, as Ulf suggested earlier, to have > bounded message queues (kill the process if the message queue length > exceeds a certain value). +1 P.S. Sorry Mats for sending this only to You previously -- Gleb Peregud http://gleber.pl/ Every minute is to be grasped. Time waits for nobody. -- Inscription on a Zen Gong From vladdu55@REDACTED Tue Jun 3 11:07:38 2008 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Tue, 3 Jun 2008 11:07:38 +0200 Subject: [erlang-questions] Allowed node names Message-ID: <95be1d3b0806030207y32fad332g390ea226b73e9fb4@mail.gmail.com> Hi, I couldn't find any documentation regarding which node names are allowed, so I had to trace it down in the emulator code. Node names must conform to the following regexp: [a-zA-Z0-9_-]+ Maybe it should be added to the docs. regards, Vlad -- .......__o .......\<, ....( )/ ( )... -------------- next part -------------- An HTML attachment was scrubbed... URL: From avindev@REDACTED Tue Jun 3 11:09:29 2008 From: avindev@REDACTED (Arbow) Date: Tue, 3 Jun 2008 17:09:29 +0800 Subject: [erlang-questions] Bugs in tsung 1.2.2 ts_utils.erl Message-ID: I found tsung 1.2.2 also can't start at Erlang R12B2, so I found this bug: in ./src/ts_utils.erl: check_httpd_old_version()-> case erlang:system_info(version) of [$5,$.,Maj] when Maj > $5 -> false; [$5,$.,Maj, $.,Min] when ( Maj > $5 ) or ((Maj == $5) and Min > $3) -> false; _ -> true end. it return true at R12B2, when it change to [$5,$.,Maj, $.,Min] when ( Maj > $5 ) or ((Maj == $5) and (Min > $3) ) -> false; it works From vychodil.hynek@REDACTED Tue Jun 3 11:17:07 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Tue, 3 Jun 2008 11:17:07 +0200 Subject: [erlang-questions] erlang-questions Digest, Vol 12, Issue 98 In-Reply-To: <8f24f4b10806021009u12c56b6btc92da5bdc747b2f8@mail.gmail.com> References: <8f24f4b10805300742r3cda023bi23f0d6fe3b398590@mail.gmail.com> <4d08db370806020040h462dfadbw9763c11f92e71e5f@mail.gmail.com> <8f24f4b10806021009u12c56b6btc92da5bdc747b2f8@mail.gmail.com> Message-ID: <4d08db370806030217k337cfbbao69afdb397c2291e2@mail.gmail.com> 2008/6/2 John Haugeland : > 1. You don't tried to compile it (illegal guard expression). >> > > Yeah, well, it's easily fixed. I'm at work and I don't have an erlang > shell here. > Well, then fix it. Little hint: Mission impossible. > > > > >> 2. It can't be selective receive. >> > > Why not? > > isa_foo(Bar) -> case Bar of { foo, _, ThingToSelect } -> { true, > ThingToSelect }; _ -> false end. > > etc > Are you sure you do know what "selective receive" means? > > > > > > - John > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hinde@REDACTED Tue Jun 3 12:11:55 2008 From: sean.hinde@REDACTED (Sean Hinde) Date: Tue, 3 Jun 2008 11:11:55 +0100 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <4843D27F.3080406@gmail.com> Message-ID: On 2 Jun 2008, at 14:31, Chandru wrote: > 2008/6/2 Mats Cronqvist : > Ulf Wiger wrote: > > I would really like to discourage people from avoiding > > selective receive because it's "expensive". It can be > > expensive on very large message queues, but this is > > a pretty rare error condition, and fairly easily observable. > > > > i think the "issue" of how the emu deals with huge in-queues is > pretty > uninteresting. > in my personal experience, every single time this has come up the > real > problem has turned out to be lack of proper flow control (typically > using {active,true} sockets). having 100k messages in an in-queue is > not > a realistic use case. > the fact that this is not, afaik, particularly well documented is of > course a problem. > > This is true - but if one has no prior experience of this situation, > it is hard to understand why a system is behaving sluggishly. What > will be nice is having an option, as Ulf suggested earlier, to have > bounded message queues (kill the process if the message queue length > exceeds a certain value). That way, flow control problems will be > more readily visible to users. In real life situations, when a > process gets into this state, the only way to fix it is to kill that > process as it will probably never catch up. This has been discussed > before: http://www.erlang.org/pipermail/erlang-questions/2006-January/018364.html > > It also fits in well with the "Let it crash" philosophy. I respectfully disagree. It is nigh on impossible to predict where there might be some error that leads to a large queue, and this would lead to "defensive programming" where every process has a short max length. This would result in random crashes and loss of data for those uncommon situations in an generally well designed system where there might be a legitimate short term peak in queue lengths. We already have a mechanism to restart if a queue grows too large (actually 2 - process_info monitoring, and out of memory !) cheers, Sean From chsu79@REDACTED Tue Jun 3 12:38:07 2008 From: chsu79@REDACTED (Christian S) Date: Tue, 3 Jun 2008 12:38:07 +0200 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <4843D27F.3080406@gmail.com> Message-ID: On Tue, Jun 3, 2008 at 12:11 PM, Sean Hinde wrote: > I respectfully disagree. It is nigh on impossible to predict where > there might be some error that leads to a large queue, and this would > lead to "defensive programming" where every process has a short max > length. This would result in random crashes and loss of data for those > uncommon situations in an generally well designed system where there > might be a legitimate short term peak in queue lengths. > > We already have a mechanism to restart if a queue grows too large > (actually 2 - process_info monitoring, and out of memory !) Maybe more-queued-than-a-set-threshold could be made into a traceable event? What happened to the thread about creating a dtrace provider for erlang? From sean.hinde@REDACTED Tue Jun 3 13:13:08 2008 From: sean.hinde@REDACTED (Sean Hinde) Date: Tue, 3 Jun 2008 12:13:08 +0100 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <4843D27F.3080406@gmail.com> Message-ID: On 3 Jun 2008, at 11:38, Christian S wrote: > On Tue, Jun 3, 2008 at 12:11 PM, Sean Hinde > wrote: >> I respectfully disagree. It is nigh on impossible to predict where >> there might be some error that leads to a large queue, and this would >> lead to "defensive programming" where every process has a short max >> length. This would result in random crashes and loss of data for >> those >> uncommon situations in an generally well designed system where there >> might be a legitimate short term peak in queue lengths. >> >> We already have a mechanism to restart if a queue grows too large >> (actually 2 - process_info monitoring, and out of memory !) > > Maybe more-queued-than-a-set-threshold could be made into a > traceable event? Could be nice yes. I sill think it would also be much better if the system didn't slow to a crawl if queues grow large - this is the effect that almost guarantees the need for a restart. To quote Chandru "In real life situations, when a process gets into this state, the only way to fix it is to kill that process as it will *probably never catch up*" (emphasis mine). Both slowdown effects (GC copying and selective receive repeated searching) seem quite amenable to smart optimisations. > What happened to the thread about creating a dtrace provider for > erlang? I was left with the impression someone went away to start implementing stuff .. Cheers, Sean From chandrashekhar.mullaparthi@REDACTED Tue Jun 3 13:30:47 2008 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Tue, 3 Jun 2008 12:30:47 +0100 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <4843D27F.3080406@gmail.com> Message-ID: 2008/6/3 Sean Hinde : > > On 2 Jun 2008, at 14:31, Chandru wrote: > > 2008/6/2 Mats Cronqvist : >> Ulf Wiger wrote: >> > I would really like to discourage people from avoiding >> > selective receive because it's "expensive". It can be >> > expensive on very large message queues, but this is >> > a pretty rare error condition, and fairly easily observable. >> > >> >> i think the "issue" of how the emu deals with huge in-queues is pretty >> uninteresting. >> in my personal experience, every single time this has come up the real >> problem has turned out to be lack of proper flow control (typically >> using {active,true} sockets). having 100k messages in an in-queue is not >> a realistic use case. >> the fact that this is not, afaik, particularly well documented is of >> course a problem. >> >> This is true - but if one has no prior experience of this situation, it is >> hard to understand why a system is behaving sluggishly. What will be nice is >> having an option, as Ulf suggested earlier, to have bounded message queues >> (kill the process if the message queue length exceeds a certain value). That >> way, flow control problems will be more readily visible to users. In real >> life situations, when a process gets into this state, the only way to fix it >> is to kill that process as it will probably never catch up. This has been >> discussed before: >> http://www.erlang.org/pipermail/erlang-questions/2006-January/018364.html >> >> It also fits in well with the "Let it crash" philosophy. >> > > I respectfully disagree. It is nigh on impossible to predict where there > might be some error that leads to a large queue, and this would lead to > "defensive programming" where every process has a short max length. This > would result in random crashes and loss of data for those uncommon > situations in an generally well designed system where there might be a > legitimate short term peak in queue lengths. > > We already have a mechanism to restart if a queue grows too large (actually > 2 - process_info monitoring, and out of memory !) > > I agree it is nearly impossible to predict this - but what options does a programmer have without this bounded queue facility. 1. Introduce message queue monitoring for every process which is potentially long lived, which imho is extra boiler plate code which reduces readability of core functionality. Also there will be different ways of doing it depending on how your process is structured (gen_fsm, gen_server, gen_event, pure erlang...). If all that one does upon detecting this condition is clear the message queue by discarding messages, or terminate the process, wouldn't it be good to have this built-in? 2. have another process which monitors the entire system - which is not very scalable when you have hundreds of thousands of processes. 3. Wait for the system to crash in live and then figure out what happened. cheers Chandru -------------- next part -------------- An HTML attachment was scrubbed... URL: From chandrashekhar.mullaparthi@REDACTED Tue Jun 3 14:03:11 2008 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Tue, 3 Jun 2008 13:03:11 +0100 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <95be1d3b0806030456y49753eedp89f58cbd84403972@mail.gmail.com> References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <4843D27F.3080406@gmail.com> <95be1d3b0806030456y49753eedp89f58cbd84403972@mail.gmail.com> Message-ID: 2008/6/3 Vlad Dumitrescu : > HI, > > 2008/6/3 Chandru : > >> 2008/6/3 Sean Hinde : >> >>> On 2 Jun 2008, at 14:31, Chandru wrote: >>> >>> I respectfully disagree. It is nigh on impossible to predict where there >>> might be some error that leads to a large queue, and this would lead to >>> "defensive programming" where every process has a short max length. This >>> would result in random crashes and loss of data for those uncommon >>> situations in an generally well designed system where there might be a >>> legitimate short term peak in queue lengths. >>> >>> We already have a mechanism to restart if a queue grows too large >>> (actually 2 - process_info monitoring, and out of memory !) >>> >>> >> I agree it is nearly impossible to predict this - but what options does a >> programmer have without this bounded queue facility. >> >> 1. Introduce message queue monitoring for every process which is >> potentially long lived, which imho is extra boiler plate code which reduces >> readability of core functionality. Also there will be different ways of >> doing it depending on how your process is structured (gen_fsm, gen_server, >> gen_event, pure erlang...). If all that one does upon detecting this >> condition is clear the message queue by discarding messages, or terminate >> the process, wouldn't it be good to have this built-in? >> >> 2. have another process which monitors the entire system - which is not >> very scalable when you have hundreds of thousands of processes. >> >> 3. Wait for the system to crash in live and then figure out what >> happened. >> >> > An alternative to the crash-when-queue-is-full solution could be that a > user-defined function gets called first. This function should use > application-specific knowledge to clean up the queue with the least amount > of disturbance for the application. The process should crash only if this > cleanup is still not enough or if the function sets a new higher threshold > for the queue length. The function could also turn on debug mode so that the > reason for the many messages is found as it happens. > I agree. This would be useful too. cheers Chandru -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Tue Jun 3 14:14:28 2008 From: erlang@REDACTED (Dominic Williams) Date: Tue, 3 Jun 2008 08:14:28 -0400 (EDT) Subject: [erlang-questions] Mocking in Erlang In-Reply-To: <69d143cd0805261846g46973594gbf58798c4b256330@mail.gmail.com> References: <69d143cd0805231746p7f8ed4cdl91e0f6fdd7da8025@mail.gmail.com> <34687.217.128.75.198.1211802071.squirrel@www.geekisp.com> <69d143cd0805261846g46973594gbf58798c4b256330@mail.gmail.com> Message-ID: Hi Jeremy, > what is the typical way to handle the case where your module > uses ibrowse say. > > is the best practice in testing just to not mock ibrowse? or > do you typically just isolate the ibrowse calls in a small > untested function and test everything else as normal? or is > there some other trick I dont know about yet? This is one of the reasons that I do not subscribe to the "traditional" Erlang practise of writing interface modules that hide message passing behind function calls. I like to expose the message passing and have the message protocol /be/ the official interface. Testing and mocking become very much easier. However, when faced with existing applications that use the interface module approach (such as, in your case, ibrowse) , I resort to one of the following to perform mocking: - define my own set of messages (usually a small subset of the applications full functionality) and introduce a bridge process - invert the dependency: avoid side-effects and don't call out to ibrowse directly from the module that needs to be tested: have it return lists of "ibrowse command" terms, and let the caller pass these on to a dumb (untested) module that makes the actual ibrowse calls given these terms - create higher-order functions: always pass into the functions of the module you want to test (that makes the ibrowse calls) the functions it should use to make the calls. When there are too many functions to make this practical, either pass in a module, or pass in a single function whose first argument could be the ibrowse function name... - use parameterised modules, with the name of the module to be used to make ibrowse calls being a parameter. I am driven to adopt such approaches in order to achieve good testability, but in my experience they turn out to be good designs in themselves, with more flexibility and modularity. Regards, Dominic Williams http://dominicwilliams.net ---- From sean.hinde@REDACTED Tue Jun 3 14:32:28 2008 From: sean.hinde@REDACTED (Sean Hinde) Date: Tue, 3 Jun 2008 13:32:28 +0100 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <4843D27F.3080406@gmail.com> Message-ID: On 3 Jun 2008, at 12:30, Chandru wrote: > > We already have a mechanism to restart if a queue grows too large > (actually 2 - process_info monitoring, and out of memory !) > > > I agree it is nearly impossible to predict this - but what options > does a programmer have without this bounded queue facility. Well, I guess, mostly you need to have a design that doesn't lead to massive queue build up under sustained overload :-). This might mean input load regulation, or tweaking the process structure (the logger process problem). The system is unlikely to be performing to spec during this whole period of queue build up followed by cyclic restart - it doesn't really matter if the system restarts because it runs out of memory or cyclic restarts one process inside. It is still an outage for customers of the system. All you need to know is that it has crashed and why, so you can fix the bug. The erl_crash dump will tell you about the huge message queue. > 1. Introduce message queue monitoring for every process which is > potentially long lived, which imho is extra boiler plate code which > reduces readability of core functionality. Also there will be > different ways of doing it depending on how your process is > structured (gen_fsm, gen_server, gen_event, pure erlang...). If all > that one does upon detecting this condition is clear the message > queue by discarding messages, or terminate the process, wouldn't it > be good to have this built-in? Another option - fix the system so that it doesn't get into that state. > 2. have another process which monitors the entire system - which > is not very scalable when you have hundreds of thousands of processes. > > 3. Wait for the system to crash in live and then figure out what > happened. Exactly. It is a bad bug that leads to such queue build up. Crashing is fine in this case, and probably preferable to lingering onwards silently failing to provide service. Cheers, Sean From chandrashekhar.mullaparthi@REDACTED Tue Jun 3 14:49:36 2008 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Tue, 3 Jun 2008 13:49:36 +0100 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <4843D27F.3080406@gmail.com> Message-ID: 2008/6/3 Sean Hinde : > > On 3 Jun 2008, at 12:30, Chandru wrote: > > >> We already have a mechanism to restart if a queue grows too large >> (actually 2 - process_info monitoring, and out of memory !) >> >> >> I agree it is nearly impossible to predict this - but what options does a >> programmer have without this bounded queue facility. >> > > Well, I guess, mostly you need to have a design that doesn't lead to > massive queue build up under sustained overload :-). This might mean input > load regulation, or tweaking the process structure (the logger process > problem). Ofcourse :-) But as you say, sometimes it is hard to predict it so the design probably didn't cater for it. > The system is unlikely to be performing to spec during this whole period of > queue build up followed by cyclic restart - it doesn't really matter if the > system restarts because it runs out of memory or cyclic restarts one process > inside. It is still an outage for customers of the system. > > All you need to know is that it has crashed and why, so you can fix the > bug. The erl_crash dump will tell you about the huge message queue. I have seen erlang nodes die a few times without producing an erl_crash.dump. Sometimes it is because Ops got impatient and brutally killed all erlang related processes. Even if you did allow the system to run out of memory, for a system with a lot of memory, it will take a long time. All the while, the system will not be responding as it should be. 1. Introduce message queue monitoring for every process which is potentially > long lived, which imho is extra boiler plate code which reduces readability > of core functionality. Also there will be different ways of doing it > depending on how your process is structured (gen_fsm, gen_server, gen_event, > pure erlang...). If all that one does upon detecting this condition is clear > the message queue by discarding messages, or terminate the process, wouldn't > it be good to have this built-in? > > Another option - fix the system so that it doesn't get into that state. I'm all for fixing the system - all I'm asking for is facilities to detect this with less pain. >> 3. Wait for the system to crash in live and then figure out what >> happened. >> > 2. have another process which monitors the entire system - which is not > very scalable when you have hundreds of thousands of processes. > > Exactly. It is a bad bug that leads to such queue build up. Crashing is > fine in this case, and probably preferable to lingering onwards silently > failing to provide service. Exactly my point. I guess we both agree that it should crash. The disagreement seems to be about *when and how* it should crash.I would prefer that the process in question crash because in all probability, it's callers have timedout and not expecting a response any way. cheers Chandru -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Tue Jun 3 14:58:06 2008 From: jeremy@REDACTED (Jeremy Wall) Date: Tue, 3 Jun 2008 07:58:06 -0500 Subject: [erlang-questions] Mocking in Erlang In-Reply-To: References: <69d143cd0805231746p7f8ed4cdl91e0f6fdd7da8025@mail.gmail.com> <34687.217.128.75.198.1211802071.squirrel@www.geekisp.com> <69d143cd0805261846g46973594gbf58798c4b256330@mail.gmail.com> Message-ID: <69d143cd0806030558y424e2923ld0c212a08f6eefe1@mail.gmail.com> An Interesting approach. I'm going to have to try this later. On Tue, Jun 3, 2008 at 7:14 AM, Dominic Williams < erlang-dated-1212927270.714ba1@REDACTED> wrote: > Hi Jeremy, > > > what is the typical way to handle the case where your module > > uses ibrowse say. > > > > is the best practice in testing just to not mock ibrowse? or > > do you typically just isolate the ibrowse calls in a small > > untested function and test everything else as normal? or is > > there some other trick I dont know about yet? > > This is one of the reasons that I do not subscribe to the > "traditional" Erlang practise of writing interface modules that > hide message passing behind function calls. I like to expose the > message passing and have the message protocol /be/ the official > interface. Testing and mocking become very much easier. > > However, when faced with existing applications that use the > interface module approach (such as, in your case, ibrowse) , I > resort to one of the following to perform mocking: > > - define my own set of messages (usually a small subset of the > applications full functionality) and introduce a bridge process > > - invert the dependency: avoid side-effects and don't call out to > ibrowse directly from the module that needs to be tested: have > it return lists of "ibrowse command" terms, and let the caller > pass these on to a dumb (untested) module that makes the actual > ibrowse calls given these terms > > - create higher-order functions: always pass into the functions > of the module you want to test (that makes the ibrowse calls) > the functions it should use to make the calls. When there are > too many functions to make this practical, either pass in a > module, or pass in a single function whose first argument could > be the ibrowse function name... > > - use parameterised modules, with the name of the module to be > used to make ibrowse calls being a parameter. > > I am driven to adopt such approaches in order to achieve good > testability, but in my experience they turn out to be good > designs in themselves, with more flexibility and modularity. > > Regards, > > Dominic Williams > http://dominicwilliams.net > > ---- > > -- Jeremy Wall http://jeremy.marzhillstudios.com Jeremy@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hinde@REDACTED Tue Jun 3 15:14:57 2008 From: sean.hinde@REDACTED (Sean Hinde) Date: Tue, 3 Jun 2008 14:14:57 +0100 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <4843D27F.3080406@gmail.com> Message-ID: On 3 Jun 2008, at 13:49, Chandru wrote: > > 2008/6/3 Sean Hinde : > > > Well, I guess, mostly you need to have a design that doesn't lead to > massive queue build up under sustained overload :-). This might mean > input load regulation, or tweaking the process structure (the logger > process problem). > > Ofcourse :-) But as you say, sometimes it is hard to predict it so > the design probably didn't cater for it. All telecom systems are soak tested at X times overload on all external interfaces before going into service right :-) Although not all web systems perhaps !! > Another option - fix the system so that it doesn't get into that > state. > > I'm all for fixing the system - all I'm asking for is facilities to > detect this with less pain. If it is just detection you are after then have a process that calls process_info to get the queue length of all processes in the system once per minute and raise an alarm if any are above a threshold. That is not much overhead at all, and can be done without introducing new features. > Exactly my point. I guess we both agree that it should crash. The > disagreement seems to be about *when and how* it should crash.I > would prefer that the process in question crash because in all > probability, it's callers have timedout and not expecting a response > any way. Either way in all likelihood the same fault will manifest itself again within a few seconds. I can't help but imagine the proposed feature misused in all sorts of quite disgusting ways. Shudder! Sean From foolishewe@REDACTED Tue Jun 3 15:32:08 2008 From: foolishewe@REDACTED (Foolish Ewe) Date: Tue, 3 Jun 2008 13:32:08 +0000 Subject: [erlang-questions] Configuring The Erlang Debugger and Distel Message-ID: Hello All: Can anyone point me to a detailed and correct description of how to properly configure the Erlang Debugger? My host name is set via dhcp. I'm having some difficulty getting the node name as returned by node() to work correctly, I'm seeing nonode@REDACTED, and have tried futzing with my /etc/hosts and my ~/.hosts.erlang but I'm so new to Erlang and have no real elisp skills to speak of. I'm using xemacs 21.5.28 under OpenSuse 10.3. I hand iinstalled distel using the subversion repository. My host name is set by dhcp, but here is what I'd liike to be able to do: 1) I can't attach to a node to debug, can I let my default node name to be something like mynode@REDACTED? 2) I'm guessing debugging information is not compiled in by default. If so can I have Distel allow me to set compiler options (i.e. debug_info) to compiling in debugging information or have an extra option in the compile 3) Do I need to do something with cookies? 4) Should the debugging support in Distel default to connecting to the xemacs buffer with an active Erlang shell? Could this be automated? Thanks: Bill M. _________________________________________________________________ Instantly invite friends from Facebook and other social networks to join you on Windows Live? Messenger. https://www.invite2messenger.net/im/?source=TXT_EML_WLH_InviteFriends From bertil.karlsson@REDACTED Tue Jun 3 11:00:20 2008 From: bertil.karlsson@REDACTED (Bertil Karlsson) Date: Tue, 03 Jun 2008 11:00:20 +0200 Subject: [erlang-questions] ASN.1 and RELATIVE-OID In-Reply-To: <200806031838.13764.als@iinet.net.au> References: <200806031838.13764.als@iinet.net.au> Message-ID: <48450824.7060703@ericsson.com> Yes, extended asn1-functionality, among other things the RELATIVE-OID type, is under development. There are some more things to do before a patch-release, unfortunately that has to wait due to other more prioritized work. I hope we will be able to release the extensions of the asn1- compiler in the late summer or early autumn. /Bertil Karlsson Anthony Shipman wrote: > I've found that the asn1 system in Erlang, up to 12B2, does not implement > RELATIVE-OID. Are there any plans to develop the asn1 system further? > From erlang-questions_efine@REDACTED Tue Jun 3 16:51:19 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Tue, 3 Jun 2008 10:51:19 -0400 Subject: [erlang-questions] Allowed node names In-Reply-To: <95be1d3b0806030207y32fad332g390ea226b73e9fb4@mail.gmail.com> References: <95be1d3b0806030207y32fad332g390ea226b73e9fb4@mail.gmail.com> Message-ID: <6c2563b20806030751v51830b49p216daaf4b31e856@mail.gmail.com> Gosh, Vlad, how can you say it is not documented? I mean, it clearly states in the comments to the Erlang C source code in erts/emulator/beam/dist.c that ** ** n must be a valid node name: string of ([a-z][A-Z][0-9]_-)+ ** It only took me 20 minutes to track this down. Don't you know, Real Programmers Don't Need No Steenking Documentation, They Just Read The Code ;-) *(Disclaimer: In case anyone inadvertently takes this rant literally and gets ticked off, it was written with heavy irony and tongue firmly in cheek).* 2008/6/3 Vlad Dumitrescu : > Hi, > > I couldn't find any documentation regarding which node names are allowed, > so I had to trace it down in the emulator code. > > Node names must conform to the following regexp: [a-zA-Z0-9_-]+ > > Maybe it should be added to the docs. > regards, > Vlad > > -- > .......__o > .......\<, > ....( )/ ( )... > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang-questions_efine@REDACTED Tue Jun 3 16:58:50 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Tue, 3 Jun 2008 10:58:50 -0400 Subject: [erlang-questions] Allowed node names In-Reply-To: <95be1d3b0806030753j6a421acfjd3efbdde7e4c7caf@mail.gmail.com> References: <95be1d3b0806030207y32fad332g390ea226b73e9fb4@mail.gmail.com> <6c2563b20806030751v51830b49p216daaf4b31e856@mail.gmail.com> <95be1d3b0806030753j6a421acfjd3efbdde7e4c7caf@mail.gmail.com> Message-ID: <6c2563b20806030758p124e3df0k7c9e29f9c925cb1c@mail.gmail.com> The "steenking comment" is right above the code of is_node_name() :) Ed On Tue, Jun 3, 2008 at 10:53 AM, Vlad Dumitrescu wrote: > Well, I admit I didn't find that, but got to the actual implementation of > is_node_name(), which is actually reading the code, not some stinking > comment :-) > > /Vlad > > > On Tue, Jun 3, 2008 at 4:51 PM, Edwin Fine > wrote: > >> Gosh, Vlad, how can you say it is not documented? I mean, it clearly >> states in the comments to the Erlang C source code in >> erts/emulator/beam/dist.c that >> >> ** >> ** n must be a valid node name: string of ([a-z][A-Z][0-9]_-)+ >> ** >> >> It only took me 20 minutes to track this down. Don't you know, Real >> Programmers Don't Need No Steenking Documentation, They Just Read The Code >> ;-) >> >> *(Disclaimer: In case anyone inadvertently takes this rant literally and >> gets ticked off, it was written with heavy irony and tongue firmly in >> cheek).* >> >> >> 2008/6/3 Vlad Dumitrescu : >> >>> Hi, >>> >>> I couldn't find any documentation regarding which node names are allowed, >>> so I had to trace it down in the emulator code. >>> >>> Node names must conform to the following regexp: [a-zA-Z0-9_-]+ >>> >>> Maybe it should be added to the docs. >>> regards, >>> Vlad >>> >>> -- >>> .......__o >>> .......\<, >>> ....( )/ ( )... >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anders.nygren@REDACTED Tue Jun 3 17:02:31 2008 From: anders.nygren@REDACTED (Anders Nygren) Date: Tue, 3 Jun 2008 10:02:31 -0500 Subject: [erlang-questions] Allowed node names In-Reply-To: <6c2563b20806030751v51830b49p216daaf4b31e856@mail.gmail.com> References: <95be1d3b0806030207y32fad332g390ea226b73e9fb4@mail.gmail.com> <6c2563b20806030751v51830b49p216daaf4b31e856@mail.gmail.com> Message-ID: 2008/6/3 Edwin Fine : > Gosh, Vlad, how can you say it is not documented? I mean, it clearly states > in the comments to the Erlang C source code in erts/emulator/beam/dist.c > that > > ** > ** n must be a valid node name: string of ([a-z][A-Z][0-9]_-)+ > ** > > It only took me 20 minutes to track this down. Don't you know, Real > Programmers Don't Need No Steenking Documentation, They Just Read The Code Oh, come on, every Really Real Programmer(tm) knows that You can't trust the comments. Only a noob would believe the lies that are in the comments. /Anders :) > ;-) > > (Disclaimer: In case anyone inadvertently takes this rant literally and gets > ticked off, it was written with heavy irony and tongue firmly in cheek). > > > 2008/6/3 Vlad Dumitrescu : >> >> Hi, >> >> I couldn't find any documentation regarding which node names are allowed, >> so I had to trace it down in the emulator code. >> >> Node names must conform to the following regexp: [a-zA-Z0-9_-]+ >> >> Maybe it should be added to the docs. >> regards, >> Vlad >> >> -- >> .......__o >> .......\<, >> ....( )/ ( )... >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From erlang-questions_efine@REDACTED Tue Jun 3 17:34:34 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Tue, 3 Jun 2008 11:34:34 -0400 Subject: [erlang-questions] Allowed node names In-Reply-To: References: <95be1d3b0806030207y32fad332g390ea226b73e9fb4@mail.gmail.com> <6c2563b20806030751v51830b49p216daaf4b31e856@mail.gmail.com> Message-ID: <6c2563b20806030834g3ead2993y2b1b32ad4cdbffc6@mail.gmail.com> LOL! That's why I wrote "They Just Read The *Code*". It just so happens that in this case, the code and the comments are ACTUALLY congruent!!! Amazing. Ed On Tue, Jun 3, 2008 at 11:02 AM, Anders Nygren wrote: > 2008/6/3 Edwin Fine : > > Gosh, Vlad, how can you say it is not documented? I mean, it clearly > states > > in the comments to the Erlang C source code in erts/emulator/beam/dist.c > > that > > > > ** > > ** n must be a valid node name: string of ([a-z][A-Z][0-9]_-)+ > > ** > > > > It only took me 20 minutes to track this down. Don't you know, Real > > Programmers Don't Need No Steenking Documentation, They Just Read The > Code > > Oh, come on, every Really Real Programmer(tm) knows that You can't > trust the comments. > Only a noob would believe the lies that are in the comments. > > /Anders :) > > > ;-) > > > > (Disclaimer: In case anyone inadvertently takes this rant literally and > gets > > ticked off, it was written with heavy irony and tongue firmly in cheek). > > > > > > 2008/6/3 Vlad Dumitrescu : > >> > >> Hi, > >> > >> I couldn't find any documentation regarding which node names are > allowed, > >> so I had to trace it down in the emulator code. > >> > >> Node names must conform to the following regexp: [a-zA-Z0-9_-]+ > >> > >> Maybe it should be added to the docs. > >> regards, > >> Vlad > >> > >> -- > >> .......__o > >> .......\<, > >> ....( )/ ( )... > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://www.erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.polvorin@REDACTED Tue Jun 3 17:39:21 2008 From: pablo.polvorin@REDACTED (Pablo Polvorin) Date: Tue, 3 Jun 2008 12:39:21 -0300 Subject: [erlang-questions] Bugs in tsung 1.2.2 ts_utils.erl In-Reply-To: References: Message-ID: <1ffe809c0806030839o649ac5adsa20e4a2eb56b0916@mail.gmail.com> Yes, this had been reported in the tsung mail list, it's fixed in svn. 2008/6/3, Arbow : > I found tsung 1.2.2 also can't start at Erlang R12B2, so I found this bug: > > in ./src/ts_utils.erl: > > check_httpd_old_version()-> > case erlang:system_info(version) of > [$5,$.,Maj] when Maj > $5 -> > false; > [$5,$.,Maj, $.,Min] when ( Maj > $5 ) or ((Maj == $5) and Min > $3) > -> > false; > _ -> > true > end. > > it return true at R12B2, when it change to > > [$5,$.,Maj, $.,Min] when ( Maj > $5 ) or ((Maj == $5) and (Min >> $3) ) -> > false; > it works > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- -- pablo http://ppolv.wordpress.com ---- From jay@REDACTED Tue Jun 3 17:47:26 2008 From: jay@REDACTED (Jay Nelson) Date: Tue, 3 Jun 2008 08:47:26 -0700 Subject: [erlang-questions] Fwd: Message Receive Semantics (was eep:MultiplePatterns) References: <004a01c8c47e$9a355910$6501a8c0@moneymaker2> Message-ID: <3225392D-B203-45C1-9BB2-9AB5130F3987@duomark.com> Valentin accidentally replied directly to me: > From: "Valentin Micic" > Date: June 2, 2008 12:02:15 AM PDT > To: "Jay Nelson" > Subject: Re: [erlang-questions] Message Receive Semantics (was > eep:MultiplePatterns) > > Hi Jay, > > Back in 2000, when I just started with Erlang (had a extensive > commercial experience in procedural languages, though), one of the > first problems I tried to solve had to do with priority handling. > So I wrote something very ugly, but someting that reflected a > paridgm I had at the time -- I remember writing something like this: > > loop_high( 0 ) -> loop_normal( 3 ); > loop_high( N ) -> > receive > {high, Msg} -> > process_msg( Msg ), > loop_high( N-1 ) > after 0 -> receive > {normal, Msg}-> > process_msg( Msg ), > loop_high( N ) > after 0 -> receive > { low, Msg} -> > process_msg( Msg ), > loop_high( N ) > after 0 -> receive > ANY -> > case process_msg( ANY ) of > stop -> exit( normal ); > _ -> loop_high( N ) > end > after 250 -> loop_high( N ) > end > end > end > end . > loop_normal( 0 ) -> loop_low( 1 ); > loop_normal( N ) -> > recevie > {low, Msg} -> > process_msg( Msg ), > loop_normal( N-1 ) > after 0 -> receive > {high, Msg} -> > process_msg( Msg ), > loop_normal( N ) > after 0 -> ... > .... > .... > end > end . > loop_low( 0 ) -> loop_high( 5 ); > loop_low( N ) -> ... > ... > ... > > The above code would do the trick, however, being ugly as it may -- > would eventually trigger refactoring to the tune of: > > loop_high( 0 ) -> loop_normal( 3 ); > loop_hihg(N) -> > receive > {high, Msg} -> > process_msg( Msg ), > loop_high( N-1 ); > {normal, Msg} -> > process_msg( Msg ), > loop_high( N ); > {low, Msg} -> > process_msg( Msg ), > loop_high(N); > ANY -> > case process_msg( ANY ) of > stop -> exit( normal ); > _ -> loop_high( N ) > end > after 1000 -> loop_high( N ) > end. > loop_normal( 0 ) -> loop_low( 1 ); > loop_normal(N) -> > receive > {normal, Msg} -> > process_msg( Msg ), > loop_normal(N-1); > {high, Msg} -> > process_msg( Msg ), > loop_normal( N); > {low, Msg} -> > process_msg( Msg ), > loop_normal(N); > ANY -> > case process_msg( ANY ) of > stop -> exit( normal ); > _ -> loop_normal( N ) > end > after 1000 -> loop_normal( N ) > end. > loop_low( 0 ) -> loop_high( 5 ); > loop_low(N ) -> ..... > .... > .... > > What I'm trying to say, I guess -- it does not really matter if one > is novice or not. We all know how to think, and only through > thinking we can arrive to a point where we could articulate the > "right" kind of questions. Nothing wrong in making a few mistakes > in a process. > > BTW, how easy one may make a mistake depends largely on how the > problem was stated. If one says: > > > > 3) Always handle messages in the following order: > a) 5 messages of {reply, high, Msg} > b) 3 messages of {reply, normal, Msg} > c) 1 message of {reply, low, Msg} > d) 'EXIT' message > > > > one should not get surprised if the resulting code blocks until 5 > high priority messages have been processed. However, if one > indicates that preference should be given to high priority > messages, than normal, followed by low priority message, utilising > a ratio of 5:3:1, and stipulate that should there be no higher > priority message, lower one should be processed -- one may close a > semantic gap, and thus prevent a novice from making a mistake. All > of that in far better English, of course ;-). > > > V. > > ----- Original Message ----- From: "Jay Nelson" > To: > Sent: Monday, June 02, 2008 1:08 AM > Subject: Re: [erlang-questions] Message Receive Semantics (was > eep:MultiplePatterns) > > >> Valentin wrote: >> >> > Consider a following scenario: for every five "high" priority >> messages >> > processed, I wanted to handle three "normal" and one "low" priority >> > message(*). To implement this with a selective receive is >> relatively easy >> > and concise, even for a novice programmer. To do without it, may be >> > considerably more difficult and messy. >> >> >> I think priority is one of the harder cases to get right, actually. >> The beginner >> will end up blocked waiting for the 5th high priority message when >> only >> 4 are on the queue, or will end up delaying high priority messages >> while >> they scan for low priority ones. >> >> So here is the concrete challenge: >> >> 1) Send a message to a linked process: >> a) {request, Msg} >> 2) The other process should generate 20 or so messages randomly >> and then quit >> 3) Always handle messages in the following order: >> a) 5 messages of {reply, high, Msg} >> b) 3 messages of {reply, normal, Msg} >> c) 1 message of {reply, low, Msg} >> d) 'EXIT' message >> >> Of course, there may not be 5 remaining high messages or 3 remaining >> normal messages so you need to deal with these end cases. In >> doing so, >> make sure you don't just end up polling every 100 milliseconds. I've >> simplified the problem by only having one process that can generate >> replies, so that the selective receive doesn't become even more >> complicated. >> >> I challenge any beginner or intermediate erlang programmer to write >> and post the code for both processes and a description of your >> results. >> All the advanced erlang people can help point out the issues that >> will >> have to be addressed when you change the initial code approach. >> >> At least, that is my bet. Valentin may be right. Let's see some >> code! >> In another post, I'll give an alternative problem that is easier and >> shows the usefulness of selective receive. >> >> In general, I would argue that erlang's approach to message is geared >> towards handling multiple conversations or sessions on a single >> channel, >> and that handling priorities is a pathological case for the simple >> message >> queue approach that erlang uses. It is far easier to implement >> priorities >> by unloading the whole queue into a process which implements a >> priority >> queue and then doling out messages according to your own priority >> logic. >> >> jay >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions > From chsu79@REDACTED Tue Jun 3 18:19:05 2008 From: chsu79@REDACTED (Christian S) Date: Tue, 3 Jun 2008 18:19:05 +0200 Subject: [erlang-questions] Mocking in Erlang In-Reply-To: References: <69d143cd0805231746p7f8ed4cdl91e0f6fdd7da8025@mail.gmail.com> <34687.217.128.75.198.1211802071.squirrel@www.geekisp.com> <69d143cd0805261846g46973594gbf58798c4b256330@mail.gmail.com> Message-ID: On Tue, Jun 3, 2008 at 2:14 PM, Dominic Williams wrote: > - invert the dependency: avoid side-effects and don't call out to > ibrowse directly from the module that needs to be tested: have > it return lists of "ibrowse command" terms, and let the caller > pass these on to a dumb (untested) module that makes the actual > ibrowse calls given these terms I have some thoughts spinning in my mind about dependency injection in Erlang. This became rather long: In Java dependency injection is a quite materalised/working idea. It works like this; rather than creating instances of the explicitly named classes you need, you instead refer to the classes by an interface, and you expect to have the interface implementing objects passed to you in your constructor. So what is done is that you have a "Location", a "Primate" and a "Consumable" interface. You have a separate specification that maps implementing classes to interfaces: Jungle implements Location, Gorilla implements Primate, Banana implements Consumable. Now, this buys something (at the cost of making an already very overly verbose Java even more verbose) and that is the ability to supply Mock objects. One can create a Gorilla and pass it your mock objects in the constructor: a Beer implements Consumable, and a Cage implements Location. This provides you with a setting for safe testing of the Gorilla under known conditions; having the Consume object throw weird exceptions in the consume method. Maybe finding out that the gorilla itself throws a "weird exception" after consuming a few beers. I have also observed how it leads to better designs in themselfes, like you have Dominic: This decoupling also makes it easier to add future features. Just imagine version 2008 where you have City implements Location, Human implements Primate and Latte implements Consumable. You can get there in small increments, changing one interface implementation a time. (Perhaps by testing how your Primate likes Beer :). ==Back to Erlang== =processes= We don't need interfaces for Erlang processes the way Java does, since process protocols are (still?) only informal, you can send them whatever message you want. So that is easier for us than Java. However, we can suffer from hard-coding the function used in spawning a process. If our gorilla process has a spawn_link(fun banana:start() end) for the Consumable, then it is going to be difficult to have it use our mock Consumable. One can prepare for testable code, such as : * having the Module:init(Args) method in gen_servers look for an alternative fun used when spawning and store that fun in the State parameter, but fall back to (fun banana:start() end). * depend on (the still unofficial, seemingly staying so) parameterized modules to supply overriding module names when testing * if possible, avoid creating processes and having them be sent in spawn arguments or in messages Either way adds some "cruft" to code, but why expect that the additional requirement of testability should come without any cost? =external calls= This is a bit trickier. It is the problem where you want to mock ibrowse like the example earlier in the thread. What is bad is the need to explicitly compile for testing. Eunit's separate-compile-for-testing is actually bad, but I see why it is required there. Lets try to avoid it. Is there any input from dialyzer people and compiler optimization people on how you would prefer having a mock implementation of ibrowse injected in a module without recompiling it? Lets say one would want a few testcases: tests that guarantees that an ibrowse request always signal various remote status codes (404 not found, 501 internal error,...), and not having an actual request go out through ibrowse and externally to a simulating web server. Such a fixture is too expensive to set up if your goal is to have 10000 tests like it, tests that a programmer can run in a minute to assure himself that things are still working before a commit. > - use parameterised modules, with the name of the module to be > used to make ibrowse calls being a parameter. This is quite attractive, I much prefer this to the "oo-surrogate" people try to use parameterized modules for. Although, again, parameterized modules are still an unofficial feature. One can have -module(spider_modules). spider_server() -> spider_server:new(ibrowse,...). -module(spider_server, [IBrowse, ...]). handle_call({spider, URL}, _, S0) -> ..., Data = IBrowse:request(URL), ..., {ok, Report, S1}. Then always start with gen_server:start_link(Modules:spider_server(), Args, []) in code. Allowing one to supply another value for Module during testing. How does one maximize usefulness from dialyzer here? I want the code analyzed as if IBrowse has the value ibrowse. ==summary== Erlang is in a good position for writing testable code. There is just a lack of focus and best practices for it. There is more than one way to do it. Right now it takes experience rather than "book smart" to know how. From lestat@REDACTED Tue Jun 3 18:28:16 2008 From: lestat@REDACTED (Tamas Nagy) Date: Tue, 03 Jun 2008 17:28:16 +0100 Subject: [erlang-questions] Erlang Testing and Tools Survey Message-ID: <48457120.5080509@elte.hu> Dear Erlang User, As part of our involvement in the Protest Project consortium, an EU funded research project on property based testing and test driven development, we are compiling a survey on tools developed, supported and maintained by the Erlang community. You can read more about it on the Protest Project website (http://www.protest-project.eu). We appreciate your time in helping us. The link to the survey is here: http://FreeOnlineSurveys.com/rendersurvey.asp?sid=49wp7cnhy77u38q447214 If you prefer, you can be anonymous. We would appreciate a response by Monday the 9th. Filling it in should not take longer than 5 minutes. Its results will be presented at Erlang conferences and workshops and made public on the protest website. If you have any queries, do not hesitate to contact Tamas on tamas@REDACTED Thank You, Francesco and Tamas -- http://www.erlang-consulting.com From per.melin@REDACTED Tue Jun 3 19:54:43 2008 From: per.melin@REDACTED (Per Melin) Date: Tue, 3 Jun 2008 19:54:43 +0200 Subject: [erlang-questions] Mocking in Erlang In-Reply-To: References: <69d143cd0805231746p7f8ed4cdl91e0f6fdd7da8025@mail.gmail.com> <34687.217.128.75.198.1211802071.squirrel@www.geekisp.com> <69d143cd0805261846g46973594gbf58798c4b256330@mail.gmail.com> Message-ID: 2008/6/3 Christian S : > However, we can suffer from hard-coding the function used in spawning > a process. If our gorilla process has a spawn_link(fun banana:start() > end) for the Consumable, then it is going to be difficult to have it > use our mock Consumable. Not necessarily. Since modules aren't linked the gorilla module will call *whatever* module that calls itself banana at the time the test runs, right? So what if your test setup purges the real banana module and loads a mock banana with code:load_abs/1? It won't allow you to use a generic consumable module though. > IBrowse:request(URL) I think that calling a function with a variable module like that is still slower than a normal qualified function call. I did a quick test now and (unless I messed up) it was about 5x slower. In most cases that won't matter, but it's something to keep in mind. From w.a.de.jong@REDACTED Tue Jun 3 20:10:24 2008 From: w.a.de.jong@REDACTED (Willem de Jong) Date: Tue, 3 Jun 2008 20:10:24 +0200 Subject: [erlang-questions] use erlSoap package in Erlang In-Reply-To: References: <407d9ef80806022319u2c036e85gaa3803e17d3e8e1@mail.gmail.com> Message-ID: <407d9ef80806031110i857a809o41f0c3395ce12a7b@mail.gmail.com> Hello Thilani, Unfortunately erlsoap and erlsom are not part of the standard erlang distribution, so you will have to download and compile them. To do that for the first time can be a bit of a puzzle, but it isn't particularly difficult. Both projects are hosted on sourceforge. You will find erlsoap under http://www.sourceforge.net/projects/erlsoap and erlsom under http://www.sourceforge.net/projects/erlsom To install erlsom you have to download the tar.gz file (use the 'download erlsom' button), unzip it (you may need to download another tool for that...), put the files in the right location (probably something like "C:\Program Files\erl5.6.1\lib\erlsom-1.2.1") and compile them (start the erlang shell, cd to the right directory and compile them ( make:all([{outdir, "../ebin"}]). ). It is explained in a bit more detail in the erlsom documentation. There is no tar.gz file for erlsoap, so you will have to get the files from SVN. You will need a SVN client for that. TortoiseSVN is probably a good choice. Once you have downloaded the files you need to put them in the right location (C:\Program Files\erl5.6.1\lib\erlsoap-0.4.3) and compile them. Note that you also have to make sure that a couple of .xsd files and a .conf file are in the 'priv' (directory C:\Program Files\erl5.6.1\lib\erlsoap-0.4.3\priv) Good luck... Willem On Tue, Jun 3, 2008 at 8:36 AM, Thilani Abeysinghe wrote: > At the moment can you please guide me how to configure these erlsom and > erlsoap packages. > > Normally in erlang shell I can invoke erlang modules like > > module_name:fundction(parameters). > > How to configure above packages with the existing erlang system. > > Thanks in advance for your kind reply > > Regards > Thilani > > Computer Science and Engineering > University of Moratuwa > Sri Lanka > http://www.thilani.blogspot.com > http://www.cgmax.blogspot.com > > On Tue, Jun 3, 2008 at 11:49 AM, Willem de Jong > wrote: > >> Hi Thilani, >> >> Do you need to implement a client (sending a request and receiving a >> response) or a server (receiving a request and sending a response)? >> >> For a server the code could look something like this: >> >> -module(minimal_soap_server). >> -compile(export_all). >> >> -include("sms.hrl"). % .hrl file generated by erlsom >> >> start() -> >> mod_soap:start(), >> ok = mod_soap:add_callback("/parlay_sms", >> "parlayx_sms_send_service_3_1.wsdl", >> fun(_I, _Header, Body) -> handler(Body) end). >> >> handler([#'p:sendSms'{addresses = To, >> message = Text}]) -> >> {ok, send(To, Text)}. >> >> send(To, Text) -> >> [#'p:sendSmsResponse'{'result' = "sending " ++ Text ++ " to: " ++ To}]. >> As you can see, you need to have a WSDL. >> >> You need to install erlsoap and erlsom. That should be relatively easy: >> download the files, put them in the right place and compile them. I think >> you also need to provide a simple configuration file for erlsoap. Let me >> know if you need more information. >> >> A simple client might look like this: >> >> -module(minimal_client). >> -export([sms/2]). >> >> -include("sms.hrl"). >> sms(To, Text) -> >> Wsdl = yaws_soap_lib:initModel(" >> file://parlayx_sms_send_service_3_1.wsdl"), >> Sms = #'p:sendSms'{addresses = [To], >> senderName = "Sender", >> message = Text}, >> yaws_soap_lib:call(Wsdl, "sendSms", [], [Sms]). >> >> This uses the soap client code provided with yaws, but the erlsoap client >> is essentially the same. Again you will need a WSDL, and erlsom has to be >> installed. >> >> Good luck, >> Willem >> >> On 6/2/08, Thilani Abeysinghe wrote: >> >>> Hi All >>> I want to use ErlSoap package for sending SOAP messages using Erlang. >>> Can you please tell me how to configure the Erlsoap package.I use >>> windows >>> platform >>> Thanks in Advance >>> >>> Regards >>> Thilani >>> >>> Computer Science and Engineering >>> University of Moratuwa >>> Sri Lanka >>> http://www.thilani.blogspot.com >>> http://www.cgmax.blogspot.com >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan556@REDACTED Tue Jun 3 20:22:33 2008 From: johan556@REDACTED (Johan Holmberg) Date: Tue, 3 Jun 2008 20:22:33 +0200 Subject: [erlang-questions] Re locating Erlang installation on Windows? In-Reply-To: <17618659.post@talk.nabble.com> References: <351ce2d70805211142v3ad7df8fj127ab5c2fc08261a@mail.gmail.com> <52482.1212082932@snookles.snookles.com> <351ce2d70806030012i3c846706jc30aea6b173da705@mail.gmail.com> <17618659.post@talk.nabble.com> Message-ID: <351ce2d70806031122i2556a266n3cc5460928ea6c93@mail.gmail.com> On Tue, Jun 3, 2008 at 10:29 AM, Zvi wrote: > > Hi Johan, > > can you give me the pointers, how to rebuild Erlang from source on Windows? > > Thanks in Advance > Zvi > You can find the source at: http://www.erlang.org/download.html http://www.erlang.org/download/otp_src_R12B-2.tar.gz (current release) Once you have downloaded and unpacked the tar-file, read the file "README.win32" in the top directory. The file is quite long. I was a bit scared at first, but the instructions actully work. I haven't read the whole document. The important things are to download & install a lot of dependencies, and issue the right sequence of commands. The only thing I had initial troubles with was the fact that both Cygwin and Visual Studio have a "link.exe" command. The one from Visual Studio should be used (I happened to have the other first in my PATH). I hope this helps. /Johan From chsu79@REDACTED Tue Jun 3 20:57:03 2008 From: chsu79@REDACTED (Christian S) Date: Tue, 3 Jun 2008 20:57:03 +0200 Subject: [erlang-questions] Mocking in Erlang In-Reply-To: References: <69d143cd0805231746p7f8ed4cdl91e0f6fdd7da8025@mail.gmail.com> <34687.217.128.75.198.1211802071.squirrel@www.geekisp.com> <69d143cd0805261846g46973594gbf58798c4b256330@mail.gmail.com> Message-ID: On Tue, Jun 3, 2008 at 7:54 PM, Per Melin wrote: > Not necessarily. Since modules aren't linked the gorilla module will > call *whatever* module that calls itself banana at the time the test > runs, right? So what if your test setup purges the real banana module > and loads a mock banana with code:load_abs/1? It won't allow you to > use a generic consumable module though. True. And something I would like even better is if I could generate that module in a few lines close to the eunit test function. (Cant be impossible to create a module directly from a list of atoms and funs. Doesnt need to be fast.) A downside is that the module namespace is node-wide, so you cant safely run your tests in parallell on the same node. This is a reason I consider this to be a very dirty way to slip in the mock. >> IBrowse:request(URL) > > I think that calling a function with a variable module like that is > still slower than a normal qualified function call. > > I did a quick test now and (unless I messed up) it was about 5x > slower. In most cases that won't matter, but it's something to keep in > mind. Not a show-stopper, but bad news enough. You dont want it to be so slow that the hairiest code most in need of tests avoid testability in the name of speed. A 5x slowdown was a lot worse than my guesstimate. :/ JIT compilers get a lot of their benefits from memoizing lookups like these, it would be sweet if BEAM could do that too. Module parameters stay the same once the module "instance" has been created. From stonecypher@REDACTED Tue Jun 3 21:23:03 2008 From: stonecypher@REDACTED (John Haugeland) Date: Tue, 3 Jun 2008 13:23:03 -0600 Subject: [erlang-questions] erlang-questions Digest, Vol 12, Issue 98 In-Reply-To: <4d08db370806030217k337cfbbao69afdb397c2291e2@mail.gmail.com> References: <8f24f4b10805300742r3cda023bi23f0d6fe3b398590@mail.gmail.com> <4d08db370806020040h462dfadbw9763c11f92e71e5f@mail.gmail.com> <8f24f4b10806021009u12c56b6btc92da5bdc747b2f8@mail.gmail.com> <4d08db370806030217k337cfbbao69afdb397c2291e2@mail.gmail.com> Message-ID: <8f24f4b10806031223v51d1c8a3rc4091762d2c4f94b@mail.gmail.com> > > Are you sure you do know what "selective receive" means? > Hahah. Yes, sorry, long day, was tired. I actually started writing code as an example today before I realized I had switched to looking at the code for the wrong discussion. Ok, the selective receive point is well taken. I suppose if I wanted to be a dick about it, I could say something like "spawn a transforming process", but at that point it gets rather silly and wasteful. You've won this battle, Commander Vychodil. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Tue Jun 3 22:19:43 2008 From: rvirding@REDACTED (Robert Virding) Date: Tue, 3 Jun 2008 22:19:43 +0200 Subject: [erlang-questions] Performance of matches In-Reply-To: <4844F0C3.3020404@it.uu.se> References: <483EDB2E.9070103@san.rr.com> <484126E2.5040000@adam.com.au> <4841DBFB.2000308@san.rr.com> <484462D6.8000001@san.rr.com> <97619b170806022112x73792e29y5a612cb966049305@mail.gmail.com> <4844F0C3.3020404@it.uu.se> Message-ID: <3dbc6d1c0806031319o46b37831sa11f6ced9770aa23@mail.gmail.com> 2008/6/3 Richard Carlsson : > Rapsey wrote: > > Actually there is an array: > > http://erlang.org/doc/man/array.html > > > > It was added in R12B I think. > > Yes, but it is a purely functional data structure, > and still has O(log n) access time, although with a > much better constant factor than dict or gb_trees. The only O(1) data structure I know of in Erlang are ETS and dict, though as Richard says the constant factor is higher. Which method you choose is of course entirely dependent on your problem. Fortunately the interfaces to dict, array are gb_trees sufficiently alike so that you can hide them in macros to facilitate testing and comparing. You can do it with ets but it takes a little more fiddling. Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From richardc@REDACTED Tue Jun 3 23:17:08 2008 From: richardc@REDACTED (Richard Carlsson) Date: Tue, 03 Jun 2008 23:17:08 +0200 Subject: [erlang-questions] Performance of matches In-Reply-To: <3dbc6d1c0806031319o46b37831sa11f6ced9770aa23@mail.gmail.com> References: <483EDB2E.9070103@san.rr.com> <484126E2.5040000@adam.com.au> <4841DBFB.2000308@san.rr.com> <484462D6.8000001@san.rr.com> <97619b170806022112x73792e29y5a612cb966049305@mail.gmail.com> <4844F0C3.3020404@it.uu.se> <3dbc6d1c0806031319o46b37831sa11f6ced9770aa23@mail.gmail.com> Message-ID: <4845B4D4.3020905@it.uu.se> Robert Virding wrote: > The only O(1) data structure I know of in Erlang are ETS and dict, And it should be noted that the O(1) is just for the read operations. I'm not entirely sure, but it looks like dict is actually O(n) for writes, although it is not noticeable until n gets pretty large. (Actually, it could be a good idea to reimplement the dict module using the same underlying data structure as the array module.) ETS is probably the closest approximation to O(1) read/write that you'll get in Erlang, but it has another source of overhead: data is always copied in and out of the ETS tables, so you don't want to use it for storing large data structures (unless they are accessed infrequently). /Richard From erlang@REDACTED Wed Jun 4 00:06:18 2008 From: erlang@REDACTED (Dominic Williams) Date: Wed, 04 Jun 2008 00:06:18 +0200 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <9b08084c0806010710y1eb4e077k2888a7d14d37f347@mail.gmail.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <9b08084c0806010710y1eb4e077k2888a7d14d37f347@mail.gmail.com> Message-ID: <4845C05A.2050406@dominicwilliams.net> Hi Joe, > It would be a zillion times easier if the *client* did > all this messing around. If the client knew about N > server address it could automatically redirect to a > different server if the primary server failed. if the > client knew about N machines and performed the md5(Key) > calculation to find the correct machine then the problem > would be very easy to solve with no messing around in the > server. Indeed, and I've used this strategy in the past on systems in which we were responsible for both client and server. It has one problem, though, which is if the network is not completely reliable and can become partitioned - different clients can end up using different servers, resulting in an inconsistent state. The way around this - leader election - works nicely in small systems but surely it doesn't scale to millions of clients? > The next step is (ovf course) to allow all the clients to > collectively behave as if they were as server - I think > therefore that the problem is really one about the base > level of a P2P architecture and not about a twitter > architecture per se. I agree - but P2P architectures come with their own lot problems (firewalls, heterogenous clients, the legal issues related to innocent users storing someone else's illegal data, etc.) So I think you're right to stress that we probably need to talk actual numbers, and think about a "sufficiently scalable" architecture rather than an infinitely scalable one. Regards, Dominic Williams http://dominicwilliams.net ---- From pfisher@REDACTED Wed Jun 4 00:16:15 2008 From: pfisher@REDACTED (Paul Fisher) Date: Tue, 03 Jun 2008 17:16:15 -0500 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <4845C05A.2050406@dominicwilliams.net> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <9b08084c0806010710y1eb4e077k2888a7d14d37f347@mail.gmail.com> <4845C05A.2050406@dominicwilliams.net> Message-ID: <1212531375.7281.5.camel@localhost> On Wed, 2008-06-04 at 00:06 +0200, Dominic Williams wrote: > Hi Joe, > > > It would be a zillion times easier if the *client* did > > all this messing around. If the client knew about N > > server address it could automatically redirect to a > > different server if the primary server failed. if the > > client knew about N machines and performed the md5(Key) > > calculation to find the correct machine then the problem > > would be very easy to solve with no messing around in the > > server. > > Indeed, and I've used this strategy in the past on systems > in which we were responsible for both client and server. > > It has one problem, though, which is if the network is not > completely reliable and can become partitioned - different > clients can end up using different servers, resulting in an > inconsistent state. The way around this - leader election - > works nicely in small systems but surely it doesn't scale to > millions of clients? This is why systems that use hashing to distribute load like Amazon's dynamo also employ a "gossip protocol" to maintain group consistency and perform failure detection. Even though it is not stated more specifically in the paper, it is a reasonable guess, base on the background of their CTO (Werner Vogels), that this employs some form of virtual synchrony. The node receiving the call from a client (which may be behind in group membership and partition knowledge) will set it straight. -- paul From ulf@REDACTED Wed Jun 4 00:45:17 2008 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 4 Jun 2008 00:45:17 +0200 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <4843D27F.3080406@gmail.com> Message-ID: <8209f740806031545n65cf7d87rb31b26d2752ae86@mail.gmail.com> 2008/6/3 Sean Hinde : > > If it is just detection you are after then have a process that calls > process_info to get the queue length of all processes in the system > once per minute and raise an alarm if any are above a threshold. That > is not much overhead at all, and can be done without introducing new > features. Just for fun, I made a few additions to plain_fsm, to play around with this. The idea is that since you have a hook there anyway, you might parameterize that hook so that it can check certain limits upon receive. The example program fsm_example.erl had a state with an extended_receive and a timeout clause. I added an option to tell plain_fsm to react if the message queue grew past 3 messages: spawn_link() -> plain_fsm:spawn_link(?MODULE, fun() -> process_flag(trap_exit,true), queue_limit(), idle(mystate) end). queue_limit() -> plain_fsm:store_options( [{watch, [{queue, 3, fun(S) -> io:format("msg queue too long!~n"), flush(), S end}]} ]). Testing the code in the shell: 1> P = fsm_example:spawn_link(). <0.33.0> timeout in idle timeout in idle 2> [P ! hi || _ <- lists:seq(1,10)]. [hi,hi,hi,hi,hi,hi,hi,hi,hi,hi] timeout in idle msg queue too long! timeout in idle timeout in idle In the current version, you can insert checks for message queue length and heap size, and run_queue, as a quick and dirty way to detect CPU overload. I haven't checked it in in Jungerl - not convinced yet that it's a good idea. If anyone wants to play with it, I can send you the code. Anyway, you're absolutely right in that this kind of check can be made fairly easily without introducing new 'features'. BR, Ulf W From ok@REDACTED Wed Jun 4 00:57:52 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 4 Jun 2008 10:57:52 +1200 Subject: [erlang-questions] erlang-questions Digest, Vol 12, Issue 98 In-Reply-To: <8f24f4b10805300742r3cda023bi23f0d6fe3b398590@mail.gmail.com> References: <8f24f4b10805300742r3cda023bi23f0d6fe3b398590@mail.gmail.com> Message-ID: <74B9F493-2EB3-4A71-889B-2BFD5B746E58@cs.otago.ac.nz> I describe how selective multipattern receive could be done using abstract patterns. On 31 May 2008, at 2:42 am, John Haugeland wrote: > So what's the advantage of that over something like this? > > isa_quit(State) -> > case State of > {?MODULE,{signal,{window1,_}}} -> true; > {?MODULE,{signal,{quit1,_} }} -> true; > _ -> false > end. There was a clear distinction between states and messages. The abstract patterns were there to match messages, not states. > > loop(State) -> > receive RcvState -> if > isa_quit(RcvState) -> quit(); > isa_connect(RcvState) -> loop(conn(State)); > isa_disconnect(RcvState) -> loop(disc(State)); > isa_show_about(RcvState) -> loop(show_about(State)); > isa_hide_about(RcvState) -> loop(hide_about(State)); > isa_data(RcvState) -> loop(update(State, RcvState)); > X -> io:fwrite("got ~p~n", [X]), loop(State) > end end. Ah. You confused me very effectively by using the "State" name in the is_a_* functions, when you really meant "Message". (Why would you call something that isn't any kind of state "RcvState", and having done that, why would you call it "State" in other functions?) There are two major problems with your proposed rewrite: (1) It isn't legal. The conditions in 'if' expressions are *guards*, not expressions. User defined function calls are not allowed at all in guards. If course, neither is the version with abstract patterns legal. One way to think of abstract patterns is that they are user-defined functions, but defined in such a very limited way that they are painfully obviously safe to call in guards without sacrificing any of the properties that distinguish guards from general expressions. (And they do other things.) (2) It would not, in general, work at all, even if user-defined functions *were* allowed in guards. The reason is simple: your version absolutely depends on removing ANY old message from the mailbox and only later checking what you got. But the abstract pattern approach satisfies what I take to be Ulf Wiger's requirement, namely that it is a *selective* receieve: a message is ONLY removed from the mailbox if it matches some pattern. Put another way, the selective receive with abstract patterns does not need the "X" case at the end. There are also minor problems, such as calling update(State, {data,Data}) instead of update(State, Data), but they really don't matter. The bottom line is that *selective* receives where multiple pattern/guard matches are to be handled the same way warrants *some* kind of special treatment, whether it is abstract patterns, "decision tables", or some kind of multipattern extension, because the choice of what to receive has to be done *in* the receive. Abstract patterns say "make your choice using abstract patterns, moving the matching code out of the receive and naming it". Multipatterns say "make your choice using multiple patterns confusingly interleaved with the actions." The decision table approach says "the *choice* has to be made inside the receive but the actions *don't*, so move the actions out." From steve.arons@REDACTED Wed Jun 4 01:21:37 2008 From: steve.arons@REDACTED (Steve Arons) Date: Tue, 3 Jun 2008 19:21:37 -0400 Subject: [erlang-questions] Twoorl: an open source Twitter clone Message-ID: The stream of incoming tweets is bursty. Even if the stream were level, the backend load for store and retrieve would look nonlinear, because it depends on the number of followers for each poster. So the system needs a lot of expensive slack capacity to provide a reasonable quality of service, and this comes on top of a growing user base. Whether the answer is to hammer a database to retrieve a user's message queue, or to burn storage, I'm wondering if the structure of the social graph might suggest a way of distributing the storage and workload--stuff like the number, size and stability of connected components, and the feasibility of splitting larger components into communities, with the goal of maximizing the probability that a message delivered to a server will be retrieved by another user on that server. But I suppose that could get messy quickly as new cross-server clusters form on the graph. Anyway, I'd be curious to see an analysis of real data on the dynamics of Twitter's message stream and its social graph. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Jun 4 02:44:42 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 4 Jun 2008 12:44:42 +1200 Subject: [erlang-questions] ioctl support? In-Reply-To: <84fb38e30805311800t66e47fc9jab554a0abb6f7e40@mail.gmail.com> References: <84fb38e30805311552p33ef95ddxf62381b7cdde742e@mail.gmail.com> <1212275057.6409.138.camel@localhost> <84fb38e30805311800t66e47fc9jab554a0abb6f7e40@mail.gmail.com> Message-ID: <855A4AB2-EB41-421C-A002-DD86F74D4214@cs.otago.ac.nz> On 1 Jun 2008, at 1:00 pm, tsuraan wrote: > I'm trying to determine if my cd-rom drive tray is open. I think the > only way to do that in Linux is to use the CDROM_DRIVE_STATUS request > (an ioctl). CD-ROM trays don't open and close that fast, so presumably checking once a second or so would be ample. What I'd do would be to write a separate C program as a C node and have it send {cdrom,opened} and {cdrom,closed} messages. From bob@REDACTED Wed Jun 4 03:00:50 2008 From: bob@REDACTED (Bob Ippolito) Date: Tue, 3 Jun 2008 18:00:50 -0700 Subject: [erlang-questions] Debugging float math in geohashing module In-Reply-To: References: Message-ID: <6a36e7290806031800j6bc52209n69dc32b74c74c228@mail.gmail.com> I'm pretty sure your problem is the precision of io_lib:fwrite("~f", [N]). Don't use that, it only prints 6 digits of precision. On Mon, Jun 2, 2008 at 10:52 PM, Nick Gerakines wrote: > I've taken an interest in the concept of the geohash [1] and wrote a simple > module [2] to encode/decode latitude/longitude pairs into these geohashes. > The modules is loosely based on Geo::Hash [3] but with a number of obvious > changes. > > So far encoding works perfectly and the same test cases from the Geo::Hash > perl module pass in my module for encode operations. > > The decode functionality is different however. When I attempt to decode a > geohash into a latitude/longitude pair the end result is slightly* off. When > I say slightly off I mean that the float precision is incorrect from the > latitude/longitude that was initially given. I think it has something to do > with the way the floats are being divided and rounded in the process. I'm > really having a rough time working around this, any help is welcome. I've > tried (unsuccessfully) to work around it using big int tuples ({N, E}) but > that didn't go so well. > > [1] http://en.wikipedia.org/wiki/Geohash > [2] http://github.com/ngerakines/erlang_geohash/tree/master/geohash.erl > [3] http://search.cpan.org/~andya/Geo-Hash/lib/Geo/Hash.pm > > # Nick Gerakines > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From ok@REDACTED Wed Jun 4 03:03:01 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 4 Jun 2008 13:03:01 +1200 Subject: [erlang-questions] Performance of matches In-Reply-To: <97619b170806022112x73792e29y5a612cb966049305@mail.gmail.com> References: <483EDB2E.9070103@san.rr.com> <484126E2.5040000@adam.com.au> <4841DBFB.2000308@san.rr.com> <484462D6.8000001@san.rr.com> <97619b170806022112x73792e29y5a612cb966049305@mail.gmail.com> Message-ID: > On Mon, Jun 2, 2008 at 11:15 PM, Darren New wrote: > > Good to know the process dictionary is actually a hashtable, but > there's > only one per process and they don't nest, so that's pretty limited. Sure they nest: a key can map to another process and _that_ process has another process dictionary. It would be pretty simple to whip up a module implementing multi-key hash tables that way. From gustavo@REDACTED Wed Jun 4 05:44:36 2008 From: gustavo@REDACTED (Gustavo Niemeyer) Date: Wed, 04 Jun 2008 00:44:36 -0300 Subject: [erlang-questions] Debugging float math in geohashing module In-Reply-To: References: Message-ID: <48460FA4.1070908@niemeyer.net> Hi Nick, (...) > The decode functionality is different however. When I attempt to decode a > geohash into a latitude/longitude pair the end result is slightly* off. When (...) Can you present an example of what you'd expect to happen, and what's actually happening? If I understand what you mean, there might be nothing wrong with your code. Note that due to the nature of the Geohash algorithm itself, in most cases you won't be able to decode the value into a precise decimal floating point. What is done by the software behind geohash.org during encoding is to compute how many bits you need to at least match the decimal precision given as input, and during decoding the inverse is done: it only shows the decimal digits it can correctly represent given the number of bits available. As a hint, if you have 8 bits for the longitude code, and a maximum value of 360, the maximum error is 360 / 2^8, and with log(maximum_error, 10) you get the decimal exponent of the maximum error, which is pretty close to what you want. -- Gustavo Niemeyer http://niemeyer.net From yarivsadan@REDACTED Wed Jun 4 06:45:34 2008 From: yarivsadan@REDACTED (Yariv Sadan) Date: Tue, 3 Jun 2008 21:45:34 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> Message-ID: <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> I considered using a reliable queuing mechanism such as RabbitMQ or Amazon SQS but I don't think it would make the architecture inherently more scalable (more reliable maybe, but not more scalable). I think a Twitter like solution can be designed to scale using just Yaws, MySQL, and Mnesia or memcache (and maybe Ejabberd if you add an XMPP gateway). RabbitMQ or SQS would provide *reliable* asynchronous background processing, but if you don't need 100% reliability (Twoorl isn't a banking application after all), you can just spawn Erlang processes from Yaws to do background tasks after a user posts a message. Also, using persistent queues doesn't make the need for a database go away. When you pull a tweet from a queue you have to put it somewhere so it can be shown on rendered pages, and a database is the most reasonable place to put it. The main problem in scaling Twitter/Twoorl is how you architect your database backend -- partitioning, denormalization, replication, load balancing, caching, etc, will probably make or break your ability to scale. Yariv On Sun, Jun 1, 2008 at 1:41 AM, David Mitchell wrote: > This is a REALLY interesting discussion, but at this point it's > becoming obvious that I don't know enough about Twitter... > > Are you suggesting that Twoorl should be architected as follows: > - when they register, every user gets assigned their own RabbitMQ > incoming and outgoing queues > - user adds a message via Web/Yaws interface (I know, this could be > SMS or something else later...) > - message goes to that user's RabbitMQ incoming queue > - a backend reads messages from the user's incoming queue, looks up in > e.g. a Mnesia table to see who should be receiving messages from that > user and whether they're connected or not. If "yes" to both, RabbitMQ > then forwards the message to each of those users' outgoing queues > - either the receiving users poll their outgoing queue for the > forwarded message, or a COMET-type Yaws app springs to life and > forwards the message to their browser (again, ignoring SMS) > > This seems like a reasonable approach; I'm just curious if that's what > you're suggesting, or whether you've got something else in mind. > > Great thread, and thanks Yariv for getting this discussion going with Twoorl > > Regards > > Dave M. > > 2008/6/1 Steve : >> >> On May 31, 5:04 pm, "Yariv Sadan" wrote: >>> ...but it's the only way you can scale this kind of service when N is >>> big. >> >> Hmmm, Yariv, aren't you still thinking about this in the way that Dave >> Smith pointed to as the heart of the issue? i.e. >> Dave said: "My understanding is that the reason they have such poor >> uptime is due to the fact that they modeled the problem as a web-app >> instead of a messaging system." >> >> I'm aware that you are likely a good way away from hitting any >> scalability problems, but some kind of tiering would seem to be >> appropriate if twoorl is to be "twitter done right". Yaws at the front >> end, definitely - but rather /RabbitMQ/ at the back end. I believe >> that you'd then have the flexibility to distribute/cluster as >> necessary to scale to the required level (whatever that may be). >> >> For sure, Twoorl is a great demo of what can be done with Erlang in an >> incredibly short time. I'm a relative noob to Erlang, and have learned >> a great deal from your blog/code/examples. >> >> Steve >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From ok@REDACTED Wed Jun 4 07:09:01 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 4 Jun 2008 17:09:01 +1200 Subject: [erlang-questions] Allowed node names In-Reply-To: <6c2563b20806030751v51830b49p216daaf4b31e856@mail.gmail.com> References: <95be1d3b0806030207y32fad332g390ea226b73e9fb4@mail.gmail.com> <6c2563b20806030751v51830b49p216daaf4b31e856@mail.gmail.com> Message-ID: On 4 Jun 2008, at 2:51 am, Edwin Fine wrote: > Gosh, Vlad, how can you say it is not documented? I mean, it clearly > states in the comments to the Erlang C source code in erts/emulator/ > beam/dist.c that > > ** > ** n must be a valid node name: string of ([a-z][A-Z][0-9]_-)+ > ** > > It only took me 20 minutes to track this down. Don't you know, Real > Programmers Don't Need No Steenking Documentation, They Just Read > The Code ;-) > What makes this savage joke even better is that >> The comment is WRONG << It should be [-_a-zA-Z0-9]+ . 'a@REDACTED' *is* valid, but since it isn't a lower case letter followed by an upper case letter followed by a digit followed by a dash or underscore, the comment claims that it isn't. It gets even better: the next comment, "Scanned past the host name", is *also* wrong. The bit that has been scanned past is the *node* name; the host name follows the subsequent '@'. -- "I don't want to discuss evidence." -- Richard Dawkins, in an interview with Rupert Sheldrake. (Fortean times 232, p55.) From ok@REDACTED Wed Jun 4 07:56:21 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 4 Jun 2008 17:56:21 +1200 Subject: [erlang-questions] eep: multiple patterns In-Reply-To: <483FB9E0.80504@ericsson.com> References: <200805141620.19559.bekesa@sch.bme.hu> <4836A517.2040909@gmail.com> <200805271436.19305.bekesa@sch.bme.hu> <6BA6A9E9-EF89-4FE6-B683-05F3866E9E09@cs.otago.ac.nz> <483D0CB6.2080103@ericsson.com> <483E50BC.2060902@ericsson.com> <61214.63.82.98.51.1212092630.squirrel@mail.webfaction.com> <483F1E59.4090007@ericsson.com> <483FB9E0.80504@ericsson.com> Message-ID: On 30 May 2008, at 8:25 pm, Ulf Wiger (TN/EAB) wrote: > I thought I'd try to convey a feeling for just > how hairy the receive clauses can get. I think > this is the worst one. > > It would of course be a candidate for the sort of > refactoring you've shown on the other examples. > It is perhaps a matter of taste, whether or not > one prefers to extract as much code as possible out > of the receive statement (which in this case is 650 > lines of code + comments). Most of the snipped > action bodies are only a handful of lines of code. > A few are considerably longer (up to 30 LOC). At 650 lines, I suggest that 'a candidate for .. refactoring' is putting it mildly. Without knowing which actions are supposed to be the same, I can't comment on _what_ refactoring might be appropriate. Instead I'll comment on something else. I appreciate why things like {distr_msg, DistrMsgId, ?HCMSG_REMOVE_HC} -> exist. I appreciate how the use of suitable names here can make it easier to figure out the intention of the code. However, as soon as you have things like {a, B, ?C} -> ; {a, B, ?D} -> ; ... you discover that you cannot tell whether these patterns overlap without manually expanding the macros. In an earlier message in this thread I suggested that a tool that warns you of 'critical' pairs in receive, case, or function heads would be useful. For getting code of this level of complexity right, I must raise that to EXTREMELY useful. Something I should point out about abstract patterns is that one of the jobs they were designed to do was to describe message protocols. Of course you can also see them as a strict generalisation of the new type notation; it's a pity that abstract patterns weren't implemented first. One of the things that struck me when trying to read other people's Erlang was that I could tell without enormous trouble (because I hadn't seen the system Ulf Wiger was talking about!) what messages processes DID receive, but it was very much harder to tell what messages they SHOULD receive. There were two ways I thought of using abstract patterns for this. (1) As executable message filters. A process would be able to install an abstract pattern as its filter, and any message sent to it that _didn't_ match the filter would automatically be discarded at once. (2) As possibly-executable annotations. #protocol() receive Pattern1 when Guard1 -> Action1 ; ... end would allow a static checking tool to report Pattern/Guard cases that _didn't_ form a critical pair with #protocol(), that is, things you were matching that according to your claimed protocol wouldn't happen anyway. It would also allow such a tool to warn you if there were messages in the protocol that you *didn't* match in the body. As we've seen in explanations of selective receive, that doesn't mean there is a bug. My ideas were pretty vague; I was thinking in terms of flattening stuff and then using clause subsumption, but it is basically the same problem that Mercury faces when deciding whether an 'or' is really a 'switch'. As an executable check, before passing any message on to the receive body proper, it would give it to #protocol(), and you'd get an exception if the match failed. Again, the idea is an *optional* debugging tool activated when you explicitly say so, but it seemed as if it might be useful. Another thing that occurs to me with an example of this size and complexity is that perhaps some 'little language' could be used for specifying separate pieces and then weaving them together. I certainly don't begin to have the glimmerings of an initial approximation of understanding the outlines of the elements of the application domain, so I couldn't do it myself, but if I _did_, that's the way I'd be thinking. But then, I'm the sort of person of whom it was said "He'd rather be writing programs to help him write programs that help him write programs than write programs that help him write programs." (Anyone help me with the source of that?) From andreas.hillqvist@REDACTED Wed Jun 4 08:46:39 2008 From: andreas.hillqvist@REDACTED (Andreas Hillqvist) Date: Wed, 4 Jun 2008 08:46:39 +0200 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> Message-ID: <8268eea30806032346l463f42f4q798f47db08de84b3@mail.gmail.com> As I understand it, the discussion in this thread concerns the process "default" Message Queue. What if it would be possible to create Message Queues in addition to process "default" Message Queue? Where it would be possible to give options such as: * limit to the amount message to store * what happens when message queue is full * crasch of client/sender and/or server/receiver * notification by message * block sender (I guess this is not a good idea. Because it is the opposite to the current semantics of erlangs send.) etc... To suport multiple queues, it would probebly be an idea to extend the receive statement to accept variables/names to queues: Q1 = queue:new() % Unamed queue with default options, %similar to a default process queue Q2 = queue:new(name_of_queue_1, [{size, 3}]), receive Q1 of {msg1, Data} -> do_stuff(Data); {msg2, Data} -> do_some_other_stuff(Data) end, receive Q1; Q2 of {msg1, Data} -> do_stuff(Data); {msg2, Data} -> do_some_other_stuff(Data) end, Some questions: Would it solve some of the selective receive problematics? Would it be allowed a to pass a queue between threeds? Would it be possible to let more then one process receive messages? What is your feedback on user defined Message Queues? Kind regards Andreas Hillqvist 2008/5/31, Ulf Wiger : > I would really like to discourage people from avoiding > selective receive because it's "expensive". It can be > expensive on very large message queues, but this is > a pretty rare error condition, and fairly easily observable. > > (I know of projects that have banned use of selective > receive for this reason, but without having thought much > about what to use instead, and when.) > > You can use erlang:system_monitor/2 to quickly detect > if a process is growing memory in a strange way. > > An old legacy Ericsson system implemented selective receive > in a way that the message queue could hold at most 6 messages. > Any more than that was obviously an error. > > I think it might be useful to be able to specify such a limit as > a spawn option, perhaps together with maximum heap size. > Exceeding the limit could perhaps lead to the process being > killed (which might seem backwards in the case of the message > queue, but at least gives a visible indication), or that the sender > process would be suspended (which could potentially lead to the > whole system stopping.) > > BR, > Ulf W > > 2008/5/31 Christopher Atkins : > > Hello, I tried (poorly--I'm a complete novice) to implement a benchmark from > > your earlier statement. I didn't do the same thing (load up the message > > mailbox before consuming them), but what I did write led to a perplexing (to > > me) discovery. If I uncomment the line in [loop1/0] below, performance for > > that loop degrades by an order of magnitude. Why is that? > > > > -module(test_receive). > > -compile(export_all). > > > > start() -> > > statistics(runtime), > > statistics(wall_clock), > > PidLoop1 = spawn(?MODULE, loop1,[]), > > sender(PidLoop1, 10000000), > > {_, Loop1Time1} = statistics(runtime), > > {_, Loop1Time2} = statistics(wall_clock), > > io:format("Sent ~p messages in ~p /~p~n", [100000, Loop1Time1, > > Loop1Time2]), > > statistics(runtime), > > statistics(wall_clock), > > PidLoop2 = spawn(?MODULE, loop2,[]), > > sender(PidLoop2, 10000000), > > {_, Loop2Time1} = statistics(runtime), > > {_, Loop2Time2} = statistics(wall_clock), > > io:format("Sent ~p messages in ~p /~p~n", [100000, Loop2Time1, > > Loop2Time2]). > > > > sender(_, 0) -> void; > > sender(Pid, N) -> > > if > > N rem 2 =:= 2 -> > > Pid ! test2; > > true -> > > Pid ! test1 > > end, > > sender(Pid, N - 1). > > > > proc1(F) -> > > receive > > start -> spawn_link(F) > > end. > > > > loop1() -> > > receive > > %%test1 -> loop1(); > > test2 -> loop1() > > end. > > > > loop2() -> > > receive > > _ -> loop2() > > end. > > > > > > ------------------------------------------------------------------------------------------------------------------------------ > > Message: 2 > > Date: Fri, 30 May 2008 18:07:18 +0200 > > From: "Per Melin" > > Subject: Re: [erlang-questions] eep: multiple patterns > > To: "Sean Hinde" > > Cc: Erlang Questions > > Message-ID: > > > > Content-Type: text/plain; charset=ISO-8859-1 > > > > 2008/5/30 Per Melin : > >> If I send 100k 'foo' messages and then 100k 'bar' messages to a > >> process, and then do a catch-all receive until there are no messages > >> left, that takes 0.03 seconds. > >> > >> If I do a selective receive of only the 'bar' messages, it takes 90 > >> seconds. > > > > I found my old test code: > > > > -module(selective). > > > > -export([a/2, c/2]). > > > > a(Atom, N) -> > > spawn(fun() -> b(Atom, N) end). > > > > b(Atom, N) -> > > spam_me(foo, N), > > spam_me(bar, N), > > R = timer:tc(?MODULE, c, [Atom, N]), > > io:format("TC: ~p~n", [R]). > > > > c(Atom, N) -> > > receive > > Atom -> c(Atom, N - 1) > > after 0 -> > > N > > end. > > > > spam_me(Msg, Copies) -> > > lists:foreach(fun(_) -> self() ! Msg end, lists:duplicate(Copies, 0)). > > > > --- > > > > 2> selective:a(bar, 100000). > > <0.38.0> > > TC: {124130689,0} > > 3> selective:a(foo, 100000). > > <0.40.0> > > TC: {23176,0} > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From vladdu55@REDACTED Wed Jun 4 08:56:19 2008 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 4 Jun 2008 08:56:19 +0200 Subject: [erlang-questions] eep: multiple patterns In-Reply-To: References: <200805141620.19559.bekesa@sch.bme.hu> <6BA6A9E9-EF89-4FE6-B683-05F3866E9E09@cs.otago.ac.nz> <483D0CB6.2080103@ericsson.com> <483E50BC.2060902@ericsson.com> <61214.63.82.98.51.1212092630.squirrel@mail.webfaction.com> <483F1E59.4090007@ericsson.com> <483FB9E0.80504@ericsson.com> Message-ID: <95be1d3b0806032356k766ba695u7e482c71b181788f@mail.gmail.com> On Wed, Jun 4, 2008 at 7:56 AM, Richard A. O'Keefe wrote: > But then, > I'm the sort of person of whom it was said "He'd rather be > writing programs to help him write programs that help him > write programs than write programs that help him write programs." > (Anyone help me with the source of that?) > > I only got reminded of the slogan of Terence Parr (the maniac behind ANTLR, as he puts it himself): "Why program by hand in five days what you can spend five years of your life automating?" regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Wed Jun 4 09:06:41 2008 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 4 Jun 2008 09:06:41 +0200 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <8268eea30806032346l463f42f4q798f47db08de84b3@mail.gmail.com> References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <8268eea30806032346l463f42f4q798f47db08de84b3@mail.gmail.com> Message-ID: <95be1d3b0806040006q4ac93245x1c8405014398c5f7@mail.gmail.com> On Wed, Jun 4, 2008 at 8:46 AM, Andreas Hillqvist < andreas.hillqvist@REDACTED> wrote: > > What if it would be possible to create Message Queues in addition to > process "default" Message Queue? > > How would you send messages? Something like {Pid, Queue} ! Message ? Since the sender will have to know which queue to send to, this is no different than having a cluster of processes that cooperate (from the client's point of view). Possibly it would be somewhat easier to code the server, but it could get hairy too... Maybe if this would be combined with Richard O'Keefe's suggestion for filtered receive statements: each queue has a filter and messages are automagically added to the right queue (by the runtime), without the client having to care about this implementation detail... But then the server process must juggle with the different queues anyway and my first impression is that it won't get easier to understand what's happening. Personally, I would rather have some kind of framework that would help manage the cluster of processes mentioned above. I feel it would be much easier to get right and debug. regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmercer@REDACTED Tue Jun 3 22:25:00 2008 From: dmercer@REDACTED (David Mercer) Date: Tue, 3 Jun 2008 15:25:00 -0500 Subject: [erlang-questions] Message Receive Semantics (was eep:MultiplePatterns) In-Reply-To: References: Message-ID: <01d101c8c5b7$e6b77950$f21ea8c0@SSI.CORP> Jay Nelson wrote: > So here is the concrete challenge: > > 1) Send a message to a linked process: > a) {request, Msg} > 2) The other process should generate 20 or so messages randomly > and then quit > 3) Always handle messages in the following order: > a) 5 messages of {reply, high, Msg} > b) 3 messages of {reply, normal, Msg} > c) 1 message of {reply, low, Msg} > d) 'EXIT' message > > Of course, there may not be 5 remaining high messages or 3 remaining > normal messages so you need to deal with these end cases. In doing so, > make sure you don't just end up polling every 100 milliseconds. I've > simplified the problem by only having one process that can generate > replies, so that the selective receive doesn't become even more > complicated. > > I challenge any beginner or intermediate erlang programmer to write > and post the code for both processes and a description of your results. > All the advanced erlang people can help point out the issues that will > have to be addressed when you change the initial code approach. > > At least, that is my bet. Valentin may be right. Let's see some code! OK, let me begin with the output from my program so you can see if I got the answer functionally correct: 2> nelson:start("Message text"). Sending message 20 with high priority. Sending message 19 with high priority. Sending message 18 with normal priority. Sending message 17 with normal priority. Sending message 16 with high priority. Sending message 15 with high priority. Sending message 14 with high priority. Sending message 13 with high priority. Sending message 12 with low priority. Sending message 11 with normal priority. Sending message 10 with normal priority. Sending message 9 with low priority. Sending message 8 with high priority. Sending message 7 with low priority. Sending message 6 with normal priority. Sending message 5 with low priority. Sending message 4 with low priority. Sending message 3 with normal priority. Sending message 2 with normal priority. Sending message 1 with high priority. Exit (normal) signal received from sender; begin prioritized receive sequence. Received {"Message text",high,20}. Received {"Message text",high,19}. Received {"Message text",high,16}. Received {"Message text",high,15}. Received {"Message text",high,14}. Received {"Message text",normal,18}. Received {"Message text",normal,17}. Received {"Message text",normal,11}. Received {"Message text",low,12}. Received {"Message text",high,13}. Received {"Message text",high,8}. Received {"Message text",high,1}. Received {"Message text",normal,10}. Received {"Message text",normal,6}. Received {"Message text",normal,3}. Received {"Message text",low,9}. Received {"Message text",normal,2}. Received {"Message text",low,7}. Received {"Message text",low,5}. Received {"Message text",low,4}. complete It appears to me to do as requested. It receives 5 high priority messages before receiving 3 normal priority messages and then a single low priority message. When there are no more of any particular type, it skips it and moves to the next. Here's my code: %%% @author David Mercer %%% @doc Programming challenge by Jay Nelson, 6/1/08. %%% %%% Jay Nelson wrote: %%%

%%% > So here is the concrete challenge:

%%% > 

%%% > 1) Send a message to a linked process:

%%% >      a) {request, Msg}

%%% > 2) The other process should generate 20 or so messages randomly

%%% >      and then quit

%%% > 3) Always handle messages in the following order:

%%% >      a) 5 messages of {reply, high, Msg}

%%% >      b) 3 messages of {reply, normal, Msg}

%%% >      c) 1 message of {reply, low, Msg}

%%% >      d) 'EXIT' message

%%% > 

%%% > Of course, there may not be 5 remaining high messages or 3 remaining

%%% > normal messages so you need to deal with these end cases.  In doing
so,

%%% > make sure you don't just end up polling every 100 milliseconds.  I've

%%% > simplified the problem by only having one process that can generate

%%% > replies, so that the selective receive doesn't become even more

%%% > complicated.

%%% > 

%%% > I challenge any beginner or intermediate erlang programmer to write

%%% > and post the code for both processes and  a description of your
results.

%%% > All the advanced erlang people can help point out the issues that will

%%% > have to be addressed when you change the initial code approach.

%%% 
-module(nelson). %% API -export([start/1, start/3]). %% @spec start(Msg::any()) -> complete %% %% @doc Execute the Nelson challenge. %% %% Spawns a linked process, waits for the process to exit, then receives the %% messages in accordance with the priority rules laid out by Nelson. %% %% The Msg argument is simply passed through to the spawned process, which %% will return it in the messages it sends back. %% %% @equiv start(Msg, 20, {5, 3, 1}) start(Msg) -> start(Msg, 20, {5, 3, 1}). %% @spec start(Msg::any(), N::integer(), PriorityRules::PriorityRules) -> complete %% PriorityRules = {High::integer(), Med::integer(), Low::integer()} %% %% @doc Execute the Nelson challenge. %% %% Spawns a linked process, waits for the process to exit, then receives the %% messages in accordance with the priority rules laid out by Nelson. %% %% The Msg argument is simply passed through to the spawned process, which %% will return it in the messages it sends back. %% %% N is the number of messages for the spawned process to send. %% %% The PriorityRules specify the numbers of messages of each priority which %% are received in each cycle. Nelson specified that these numbers be 5, 3, %% and 1 respectively. start(Msg, N, {High, Med, Low}) -> % Create linked process for sending prioritized messages Self = self(), process_flag(trap_exit, true), Pid = spawn_link(fun() -> start_sending(Self, Msg, N) end), % Send the initial message Pid ! {request, Msg}, % Wait for the linked process to exit receive {'EXIT', Pid, Reason} -> % Generally we expect Reason = normal, but we don't care if it's something else io:format("Exit (~p) signal received from sender; begin prioritized receive sequence.\n", [Reason]), % Once we receive the exit message, begin receiving the messages loop({0, High}, {0, Med}, {0, Low}) end. start_sending(Pid, Msg, N) -> % Seed the random number generator {A1, A2, A3} = now(), random:seed(A1, A2, A3), % Send send(Pid, Msg, N). send(_Pid, _Msg, 0) -> complete; send(Pid, Msg, N) when N > 0 -> Priority = case random:uniform(3) of 1 -> high; 2 -> normal; 3 -> low end, io:format("Sending message ~p with ~p priority.\n", [N, Priority]), Pid ! {reply, Priority, {Msg, Priority, N}}, send(Pid, Msg, N - 1). %% loop/3 is the prioritized receive loop. It works by keeping track of %% counters for the three priorities, counting up each one in priority %% order until it reaches the max. If no message of that priority is %% received, then the max is set to 0. When all counters are zero, all messages have been received. loop({0, 0}, {0, 0}, {0, 0}) -> complete; loop({H, H}, {M, M}, {L, L}) -> loop({0, H}, {0, M}, {0, L}); loop({H1, H2}, {0, M}, {0, L}) when H2 > H1 -> receive {reply, high, Msg} -> io:format("Received ~p.\n", [Msg]), loop({H1 + 1, H2}, {0, M}, {0, L}) after 0 -> loop({0, 0}, {0, M}, {0, L}) end; loop({H, H}, {M1, M2}, {0, L}) when M2 > M1 -> receive {reply, normal, Msg} -> io:format("Received ~p.\n", [Msg]), loop({H, H}, {M1 + 1, M2}, {0, L}) after 0 -> loop({H, H}, {0, 0}, {0, L}) end; loop({H, H}, {M, M}, {L1, L2}) when L2 > L1 -> receive {reply, low, Msg} -> io:format("Received ~p.\n", [Msg]), loop({H, H}, {M, M}, {L1 + 1, L2}) after 0 -> loop({H, H}, {M, M}, {0, 0}) end. Please advise. Thank-you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From monch1962@REDACTED Wed Jun 4 10:12:07 2008 From: monch1962@REDACTED (David Mitchell) Date: Wed, 4 Jun 2008 18:12:07 +1000 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> Message-ID: Do you need a database at all for the individual messages? I'm aware that you need to be able to access an archive of a user's posts, but look at the Twitter use cases for *receiving* messages (and I REALLY hope I've got these right, otherwise I'll look like a klutz...): - user receives all messages that they've "subscribed" to (i.e. from users that they're following) - user receives all messages directed to them specifically (i.e. stuff directed to @me) - user receives all messages directed to them as part of a group (I assume this functionality is in Twitter somewhere...) - user looks at the history of messages sent by a specific user You could achieve ALL of these by forwarding messages to queues, and not storing the messages themselves in a central database. For the 4th use case above, you need to be able to retain messages in queues indefinitely, but there's ways to achieve that without relying on a single, central database. Yep, you're going to burn through disc space, but that's unavoidable unless you put in some mechanism for ageing out old messages. Essentially your queues *become* the database, and managing a vast number of queues (and the messages in those queues) is a different challenge than managing a vast database. If you assume that you can massively scale out the number of queues being managed (and IBM does a pretty good job of that with MQ running on a mainframe, so I assume it's at least feasible to do it with RabbitMQ or SQS), and you've got the capacity to store a large number of messages in those queues (a big assumption, but again IBM does it with MQ), the key 2 items of data you need to manage is (a) mapping of user IDs to the queues for those users (i.e. queue name, queue server name), and (b) mapping the many-to-many relationships between users. That's the bit that needs the rapid response, but you've reduced the database scalability problem to only that. Managing the scalability of the actual messages themselves then falls within the realm of the queue servers & software, and IBM's experience seems to be that you can go a very long way down the path of just throwing hardware at that problem before you hit the limits of that approach. It's HIGHLY likely I'm missing something here, since I'm still a novice as far as Twitter's functionality is concerned (and it's late, and I'm tired...). This thread continues to be a very interesting discussion of scaling a big bit of infrastructure, and I'm learning a lot from everyone participating. Please jump in and tell me where I'm wrong, and please don't think I'm questioning the decisions you've made. Twoorl is a really impressive bit of work, and a great demo of what can be achieved relatively easily with Erlyweb. For me at least, it's really opened my eyes to what could be achieved using pure Erlang infrastructure. Regards David Mitchell 2008/6/4 Yariv Sadan : > I considered using a reliable queuing mechanism such as RabbitMQ or > Amazon SQS but I don't think it would make the architecture inherently > more scalable (more reliable maybe, but not more scalable). I think a > Twitter like solution can be designed to scale using just Yaws, MySQL, > and Mnesia or memcache (and maybe Ejabberd if you add an XMPP > gateway). RabbitMQ or SQS would provide *reliable* asynchronous > background processing, but if you don't need 100% reliability (Twoorl > isn't a banking application after all), you can just spawn Erlang > processes from Yaws to do background tasks after a user posts a > message. Also, using persistent queues doesn't make the need for a > database go away. When you pull a tweet from a queue you have to put > it somewhere so it can be shown on rendered pages, and a database is > the most reasonable place to put it. The main problem in scaling > Twitter/Twoorl is how you architect your database backend -- > partitioning, denormalization, replication, load balancing, caching, > etc, will probably make or break your ability to scale. > > Yariv > > On Sun, Jun 1, 2008 at 1:41 AM, David Mitchell wrote: >> This is a REALLY interesting discussion, but at this point it's >> becoming obvious that I don't know enough about Twitter... >> >> Are you suggesting that Twoorl should be architected as follows: >> - when they register, every user gets assigned their own RabbitMQ >> incoming and outgoing queues >> - user adds a message via Web/Yaws interface (I know, this could be >> SMS or something else later...) >> - message goes to that user's RabbitMQ incoming queue >> - a backend reads messages from the user's incoming queue, looks up in >> e.g. a Mnesia table to see who should be receiving messages from that >> user and whether they're connected or not. If "yes" to both, RabbitMQ >> then forwards the message to each of those users' outgoing queues >> - either the receiving users poll their outgoing queue for the >> forwarded message, or a COMET-type Yaws app springs to life and >> forwards the message to their browser (again, ignoring SMS) >> >> This seems like a reasonable approach; I'm just curious if that's what >> you're suggesting, or whether you've got something else in mind. >> >> Great thread, and thanks Yariv for getting this discussion going with Twoorl >> >> Regards >> >> Dave M. >> >> 2008/6/1 Steve : >>> >>> On May 31, 5:04 pm, "Yariv Sadan" wrote: >>>> ...but it's the only way you can scale this kind of service when N is >>>> big. >>> >>> Hmmm, Yariv, aren't you still thinking about this in the way that Dave >>> Smith pointed to as the heart of the issue? i.e. >>> Dave said: "My understanding is that the reason they have such poor >>> uptime is due to the fact that they modeled the problem as a web-app >>> instead of a messaging system." >>> >>> I'm aware that you are likely a good way away from hitting any >>> scalability problems, but some kind of tiering would seem to be >>> appropriate if twoorl is to be "twitter done right". Yaws at the front >>> end, definitely - but rather /RabbitMQ/ at the back end. I believe >>> that you'd then have the flexibility to distribute/cluster as >>> necessary to scale to the required level (whatever that may be). >>> >>> For sure, Twoorl is a great demo of what can be done with Erlang in an >>> incredibly short time. I'm a relative noob to Erlang, and have learned >>> a great deal from your blog/code/examples. >>> >>> Steve >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > From ulf@REDACTED Wed Jun 4 10:16:29 2008 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 4 Jun 2008 10:16:29 +0200 Subject: [erlang-questions] eep: multiple patterns In-Reply-To: References: <200805141620.19559.bekesa@sch.bme.hu> <6BA6A9E9-EF89-4FE6-B683-05F3866E9E09@cs.otago.ac.nz> <483D0CB6.2080103@ericsson.com> <483E50BC.2060902@ericsson.com> <61214.63.82.98.51.1212092630.squirrel@mail.webfaction.com> <483F1E59.4090007@ericsson.com> <483FB9E0.80504@ericsson.com> Message-ID: <8209f740806040116p578ab5b1laccb6e2509b06f3@mail.gmail.com> Thanks, Richard. Very good observations. 2008/6/4 Richard A. O'Keefe : > > On 30 May 2008, at 8:25 pm, Ulf Wiger (TN/EAB) wrote: > Instead I'll comment on something else. > I appreciate why things like > > {distr_msg, DistrMsgId, ?HCMSG_REMOVE_HC} -> > > exist. I appreciate how the use of suitable names here can > make it easier to figure out the intention of the code. > However, as soon as you have things like > {a, B, ?C} -> > ; {a, B, ?D} -> > ; ... > you discover that you cannot tell whether these patterns > overlap without manually expanding the macros. Indeed. It would perhaps be good to use a type naming convention, similar to that used in the erts sources (e.g. AM_TRUE, to signify the atom 'true'.) The best thing would be to do without the macros, and use abstract patterns instead (which, in a sense, is what the macros are in this case - albeit pretty bad ones.) A historical reason for this use of macros is that it gives you a form of simple spell check. Whereas a pattern using the atom 'flase' (common typo) might well go undetected, a misspelt macro reference gives a compile-time error. With dialyzer, many of the reasons for doing this go away, but dialyzer will not detect typos in receive patterns. > In an earlier message in this thread I suggested that a tool > that warns you of 'critical' pairs in receive, case, or > function heads would be useful. For getting code of this > level of complexity right, I must raise that to EXTREMELY > useful. Yes! > One of > the things that struck me when trying to read other people's > Erlang was that I could tell without enormous trouble (because > I hadn't seen the system Ulf Wiger was talking about!) what > messages processes DID receive, but it was very much harder > to tell what messages they SHOULD receive. Yes. This is actually a very strong argument against structuring large systems using message-passing interfaces, even though that might seem quite the natural thing to do in Erlang. Using function interfaces, the compiler, xref, dialyzer et al can give you lots of help, but with message passing no tool support can help you figure out (1) whether the message you send is properly formatted, or (2) how to jump to the place in the code where the message is handled. The bigger the system, the more terrible this problem becomes. > > There were two ways I thought of using abstract patterns > for this. > (1) As executable message filters. A process would be able > to install an abstract pattern as its filter, and any message > sent to it that _didn't_ match the filter would automatically > be discarded at once. > (2) As possibly-executable annotations. > > #protocol() receive > Pattern1 when Guard1 -> Action1 > ; ... > end This would be very useful, I think. > Another thing that occurs to me with an example of this size > and complexity is that perhaps some 'little language' could > be used for specifying separate pieces and then weaving them > together. Yes, perhaps, but I'd like to see how far we could come using abstract patterns first. (: BR, Ulf W From bob@REDACTED Wed Jun 4 10:18:15 2008 From: bob@REDACTED (Bob Ippolito) Date: Wed, 4 Jun 2008 01:18:15 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> Message-ID: <6a36e7290806040118v14e60b44j3b7a99dd6821f7d9@mail.gmail.com> Well once you start denormalizing and partitioning/sharding a database you really start losing the advantages of having one. It'd probably be easy enough to have an append-only file(s) on disk strategy per user to represent their queue (something like disk_log that is easy to read backwards), partitioning that would be very easy. You could then have a second file or database that lists the incoming and outgoing connections so the process that represents that user knows which queues to publish and subscribe to. On Tue, Jun 3, 2008 at 9:45 PM, Yariv Sadan wrote: > I considered using a reliable queuing mechanism such as RabbitMQ or > Amazon SQS but I don't think it would make the architecture inherently > more scalable (more reliable maybe, but not more scalable). I think a > Twitter like solution can be designed to scale using just Yaws, MySQL, > and Mnesia or memcache (and maybe Ejabberd if you add an XMPP > gateway). RabbitMQ or SQS would provide *reliable* asynchronous > background processing, but if you don't need 100% reliability (Twoorl > isn't a banking application after all), you can just spawn Erlang > processes from Yaws to do background tasks after a user posts a > message. Also, using persistent queues doesn't make the need for a > database go away. When you pull a tweet from a queue you have to put > it somewhere so it can be shown on rendered pages, and a database is > the most reasonable place to put it. The main problem in scaling > Twitter/Twoorl is how you architect your database backend -- > partitioning, denormalization, replication, load balancing, caching, > etc, will probably make or break your ability to scale. > > Yariv > > On Sun, Jun 1, 2008 at 1:41 AM, David Mitchell wrote: >> This is a REALLY interesting discussion, but at this point it's >> becoming obvious that I don't know enough about Twitter... >> >> Are you suggesting that Twoorl should be architected as follows: >> - when they register, every user gets assigned their own RabbitMQ >> incoming and outgoing queues >> - user adds a message via Web/Yaws interface (I know, this could be >> SMS or something else later...) >> - message goes to that user's RabbitMQ incoming queue >> - a backend reads messages from the user's incoming queue, looks up in >> e.g. a Mnesia table to see who should be receiving messages from that >> user and whether they're connected or not. If "yes" to both, RabbitMQ >> then forwards the message to each of those users' outgoing queues >> - either the receiving users poll their outgoing queue for the >> forwarded message, or a COMET-type Yaws app springs to life and >> forwards the message to their browser (again, ignoring SMS) >> >> This seems like a reasonable approach; I'm just curious if that's what >> you're suggesting, or whether you've got something else in mind. >> >> Great thread, and thanks Yariv for getting this discussion going with Twoorl >> >> Regards >> >> Dave M. >> >> 2008/6/1 Steve : >>> >>> On May 31, 5:04 pm, "Yariv Sadan" wrote: >>>> ...but it's the only way you can scale this kind of service when N is >>>> big. >>> >>> Hmmm, Yariv, aren't you still thinking about this in the way that Dave >>> Smith pointed to as the heart of the issue? i.e. >>> Dave said: "My understanding is that the reason they have such poor >>> uptime is due to the fact that they modeled the problem as a web-app >>> instead of a messaging system." >>> >>> I'm aware that you are likely a good way away from hitting any >>> scalability problems, but some kind of tiering would seem to be >>> appropriate if twoorl is to be "twitter done right". Yaws at the front >>> end, definitely - but rather /RabbitMQ/ at the back end. I believe >>> that you'd then have the flexibility to distribute/cluster as >>> necessary to scale to the required level (whatever that may be). >>> >>> For sure, Twoorl is a great demo of what can be done with Erlang in an >>> incredibly short time. I'm a relative noob to Erlang, and have learned >>> a great deal from your blog/code/examples. >>> >>> Steve >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From tony@REDACTED Tue Jun 3 20:39:14 2008 From: tony@REDACTED (Tony Rogvall) Date: Tue, 3 Jun 2008 20:39:14 +0200 Subject: [erlang-questions] Allowed node names In-Reply-To: <6c2563b20806030834g3ead2993y2b1b32ad4cdbffc6@mail.gmail.com> References: <95be1d3b0806030207y32fad332g390ea226b73e9fb4@mail.gmail.com> <6c2563b20806030751v51830b49p216daaf4b31e856@mail.gmail.com> <6c2563b20806030834g3ead2993y2b1b32ad4cdbffc6@mail.gmail.com> Message-ID: <938BA35A-CAD1-4757-B67E-46012E37A3C6@rogvall.se> The code surely needs to be fixed ;-) /Tony On 3 jun 2008, at 17.34, Edwin Fine wrote: > LOL! That's why I wrote "They Just Read The Code". It just so > happens that in this case, the code and the comments are ACTUALLY > congruent!!! Amazing. > Ed > > On Tue, Jun 3, 2008 at 11:02 AM, Anders Nygren > wrote: > 2008/6/3 Edwin Fine : > > Gosh, Vlad, how can you say it is not documented? I mean, it > clearly states > > in the comments to the Erlang C source code in erts/emulator/beam/ > dist.c > > that > > > > ** > > ** n must be a valid node name: string of ([a-z][A-Z][0-9]_-)+ > > ** > > > > It only took me 20 minutes to track this down. Don't you know, Real > > Programmers Don't Need No Steenking Documentation, They Just Read > The Code > > Oh, come on, every Really Real Programmer(tm) knows that You can't > trust the comments. > Only a noob would believe the lies that are in the comments. > > /Anders :) > > > ;-) > > > > (Disclaimer: In case anyone inadvertently takes this rant > literally and gets > > ticked off, it was written with heavy irony and tongue firmly in > cheek). > > > > > > 2008/6/3 Vlad Dumitrescu : > >> > >> Hi, > >> > >> I couldn't find any documentation regarding which node names are > allowed, > >> so I had to trace it down in the emulator code. > >> > >> Node names must conform to the following regexp: [a-zA-Z0-9_-]+ > >> > >> Maybe it should be added to the docs. > >> regards, > >> Vlad > >> > >> -- > >> .......__o > >> .......\<, > >> ....( )/ ( )... > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://www.erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Wed Jun 4 11:53:06 2008 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 4 Jun 2008 11:53:06 +0200 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <1212531375.7281.5.camel@localhost> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <9b08084c0806010710y1eb4e077k2888a7d14d37f347@mail.gmail.com> <4845C05A.2050406@dominicwilliams.net> <1212531375.7281.5.camel@localhost> Message-ID: <9b08084c0806040253o23816aa8gef1ada3a1e97f3c6@mail.gmail.com> On Wed, Jun 4, 2008 at 12:16 AM, Paul Fisher wrote: > On Wed, 2008-06-04 at 00:06 +0200, Dominic Williams wrote: >> Hi Joe, >> >> > It would be a zillion times easier if the *client* did >> > all this messing around. If the client knew about N >> > server address it could automatically redirect to a >> > different server if the primary server failed. if the >> > client knew about N machines and performed the md5(Key) >> > calculation to find the correct machine then the problem >> > would be very easy to solve with no messing around in the >> > server. >> >> Indeed, and I've used this strategy in the past on systems >> in which we were responsible for both client and server. >> >> It has one problem, though, which is if the network is not >> completely reliable and can become partitioned - different >> clients can end up using different servers, resulting in an >> inconsistent state. The way around this - leader election - >> works nicely in small systems but surely it doesn't scale to >> millions of clients? Right - but now we need to analyse the consequences of of a partitioning. I can imagine two extreme end-cases cases 1) best effort repair - basically ignore consistency - two different users could get different subsets of the information - this will only happen when the system is failing and will only effect users on the failing machines. 2) strong consistency (all uses get the same consistent view) Expensive and difficult (then there are many inbetween cases) For a free service then 1) would be easy to implement and "good enough" For a fully paid service then 2) 1) should be easy to do without a fully synchronised back-end - use a simple gossip and the notes will "eventually" become consistent. As long as the users are aware of this it shouldn't be a problem. /Joe > > This is why systems that use hashing to distribute load like Amazon's > dynamo also employ a "gossip protocol" to maintain group consistency and > perform failure detection. Even though it is not stated more > specifically in the paper, it is a reasonable guess, base on the > background of their CTO (Werner Vogels), that this employs some form of > virtual synchrony. > > The node receiving the call from a client (which may be behind in group > membership and partition knowledge) will set it straight. > > > -- > paul > > From andreas.hillqvist@REDACTED Wed Jun 4 14:51:09 2008 From: andreas.hillqvist@REDACTED (Andreas Hillqvist) Date: Wed, 4 Jun 2008 14:51:09 +0200 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <95be1d3b0806040006q4ac93245x1c8405014398c5f7@mail.gmail.com> References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <8268eea30806032346l463f42f4q798f47db08de84b3@mail.gmail.com> <95be1d3b0806040006q4ac93245x1c8405014398c5f7@mail.gmail.com> Message-ID: <8268eea30806040551r46127891hd2a9db0d8882c7e8@mail.gmail.com> Thank you for your answer. For sending I was thinking that it would be possible to use the Queue Identifyer (somthing similar to a PID): Queue ! Message And that Queue are only linked to process. One possebility with Multiple Queues is that it would be possible to Prioritize messages more efficiently: * If there is a message in Queue read messsage, else repeat for next Queue. * If there is no message in any Queue, wait for a message from any Queue. It might be nice to have one Queue for system messages and such. I do not know if the Multiple Queues is the right path. It would be possible to build some of the functionality with helper process for Queue. And that could be part of the framework you mentioned and let the future decide if the Multiple Queues semantics will catch on. It might then be omre efficently be implemented in the Erlang Runtime. Kind Regards Andreas Hillqvist 2008/6/4, Vlad Dumitrescu : > On Wed, Jun 4, 2008 at 8:46 AM, Andreas Hillqvist > wrote: > > > > What if it would be possible to create Message Queues in addition to > > process "default" Message Queue? > > > > > How would you send messages? Something like > {Pid, Queue} ! Message > ? > > Since the sender will have to know which queue to send to, this is no > different than having a cluster of processes that cooperate (from the > client's point of view). Possibly it would be somewhat easier to code the > server, but it could get hairy too... > > Maybe if this would be combined with Richard O'Keefe's suggestion for > filtered receive statements: each queue has a filter and messages are > automagically added to the right queue (by the runtime), without the client > having to care about this implementation detail... But then the server > process must juggle with the different queues anyway and my first impression > is that it won't get easier to understand what's happening. > > Personally, I would rather have some kind of framework that would help > manage the cluster of processes mentioned above. I feel it would be much > easier to get right and debug. > > regards, > Vlad > > From rtrlists@REDACTED Wed Jun 4 14:51:52 2008 From: rtrlists@REDACTED (Robert Raschke) Date: Wed, 4 Jun 2008 13:51:52 +0100 Subject: [erlang-questions] eep: multiple patterns In-Reply-To: <95be1d3b0806032356k766ba695u7e482c71b181788f@mail.gmail.com> References: <200805141620.19559.bekesa@sch.bme.hu> <483D0CB6.2080103@ericsson.com> <483E50BC.2060902@ericsson.com> <61214.63.82.98.51.1212092630.squirrel@mail.webfaction.com> <483F1E59.4090007@ericsson.com> <483FB9E0.80504@ericsson.com> <95be1d3b0806032356k766ba695u7e482c71b181788f@mail.gmail.com> Message-ID: <6a3ae47e0806040551v5d396f9ch32b11fb16149f59a@mail.gmail.com> On 6/4/08, Vlad Dumitrescu wrote: > On Wed, Jun 4, 2008 at 7:56 AM, Richard A. O'Keefe > wrote: > > > But then, > > I'm the sort of person of whom it was said "He'd rather be > > writing programs to help him write programs that help him > > write programs than write programs that help him write programs." > > (Anyone help me with the source of that?) > > > > I only got reminded of the slogan of Terence Parr (the maniac behind ANTLR, > as he puts it himself): > > "Why program by hand in five days what you can spend > five years of your life automating?" > I once noted down "I'd rather write programs that write programs than write programs." credited to some R. Sites. I've no memory of who that might be though. Robby -- "Five days in the lab beats a day in the library anytime." From steven.charles.davis@REDACTED Wed Jun 4 14:57:25 2008 From: steven.charles.davis@REDACTED (Steve Davis) Date: Wed, 4 Jun 2008 05:57:25 -0700 (PDT) Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> Message-ID: <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> Hi all, This is indeed an interesting discussion, albeit totally orthogonal to what I'm *supposed* to be focused on... :) WRT to the use of MQ --- David Mitchell seems to have covered the issue of persistent queues and thus points out - correctly to my mind - that there's no need for RDBMS storage of tweet content, but then... On Jun 3, 11:45 pm, "Yariv Sadan" wrote: > I considered using a reliable queuing mechanism such as RabbitMQ or > Amazon SQS but I don't think it would make the architecture inherently > more scalable (more reliable maybe, but not more scalable). Yariv, did you take a look at RabbitMQ clustering? It's claimed to be near-linear (though I don't have practical experience of this, it feels like it should be a valid claim)... http://www.rabbitmq.com/clustering.html BTW, in case anyone is wondering -- i have no commercial interest or connection with either lshift or cohesive. Unlike Joe, I'm not concerned by the Web server (Yaws) scaling -- if you were to get serious you'd probably shove a hardware load balancer setup in front of your web servers and that would allow you to scale httpd instances horizontally pretty much without limit (an example for those interested in the HWLB would be Cisco's Content Services Switch - http://www.cisco.com/en/US/products/hw/contnetw/ps792/ ) -- I'm assuming here that the web server user session becomes decoupled from the persistent tweet data by moving the latter out into the (persistent queue) MQ cluster... Also - I did do a bit of reading around after seeing Patrick Logan's suggestion for using XMPP/Ejabberd as an alternative to MQ. It does appear that it also could be a back-end candidate - and indeed Mickael Remond has written on that issue directly http://www.process-one.net/en/blogs/article/introducing_the_xmpp_application_server/ --- using XMPP is a less "traditional" approach than using an MQ, but Mickael himself does present evidence that the idea has legs. So yes, ejabberd does look like an alternative candidate to RabbitMQ that's worth examining closely for anyone actually building a "Twitter done right". regs /s From vladdu55@REDACTED Wed Jun 4 15:27:02 2008 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 4 Jun 2008 15:27:02 +0200 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <8268eea30806040551r46127891hd2a9db0d8882c7e8@mail.gmail.com> References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <8268eea30806032346l463f42f4q798f47db08de84b3@mail.gmail.com> <95be1d3b0806040006q4ac93245x1c8405014398c5f7@mail.gmail.com> <8268eea30806040551r46127891hd2a9db0d8882c7e8@mail.gmail.com> Message-ID: <95be1d3b0806040627x6a0105c0x3a9147d264c5924@mail.gmail.com> On Wed, Jun 4, 2008 at 2:51 PM, Andreas Hillqvist < andreas.hillqvist@REDACTED> wrote: > > One possebility with Multiple Queues is that it would be possible to > Prioritize messages more efficiently: > * If there is a message in Queue read messsage, else repeat for next > Queue. > * If there is no message in any Queue, wait for a message from any Queue. > > It might be nice to have one Queue for system messages and such. > > > I do not know if the Multiple Queues is the right path. > It would be possible to build some of the functionality with helper > process for Queue. > I think that conceptually, multiple queues would be just like wrapping all messages with {QueueId, Message} and use selective receive to discriminate, only faster. Especially if one queue is low priority and high volume. I think that it is enough with only one extra queue, for system messages. With it, it's very easy to implement any number of queues with the help of assisting processes. The system queue would be used to send receive requests to the helper process without these requests getting in a possibly large queue of normal messages. regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From koenigjm@REDACTED Wed Jun 4 15:24:24 2008 From: koenigjm@REDACTED (John Koenig) Date: Wed, 04 Jun 2008 09:24:24 -0400 Subject: [erlang-questions] Troubles Bridging SNMP and Mnesia Message-ID: <48469788.9070104@acalledshot.net> I am working on an application that utilizes snmpa to make a series of mnesia tables accessible to various non-erlang based snmp managers. I have setup the mnesia tables for snmp access, stored data in the databases, generated corresponding mibs, compiled the mibs, and loaded the mibs into the snmpa running in my application space. No errors are reported during the initialization of the mibs, mnesia objects, or the call to application:start(snmp). The rub occurs when I try to use snmpget to fetch data out of the mnesia tables using the OIDs specified in the mibs. I can snmpwalk the agent just fine and it returns the following ( I have removed some sensitive bits): snmpwalk -c public -v2c 127.0.0.1:4000 SNMPv2-MIB::sysDescr.0 = STRING: SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.31130 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (405973) 1:07:39.73 SNMPv2-MIB::sysContact.0 = STRING: SNMPv2-MIB::sysName.0 = STRING: SNMPv2-MIB::sysLocation.0 = STRING: SNMPv2-MIB::sysServices.0 = INTEGER: 72 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (0) 0:00:00.00 SNMPv2-MIB::snmpInPkts.0 = Counter32: 91 SNMPv2-MIB::snmpInBadVersions.0 = Counter32: 0 SNMPv2-MIB::snmpInBadCommunityNames.0 = Counter32: 0 SNMPv2-MIB::snmpInBadCommunityUses.0 = Counter32: 0 SNMPv2-MIB::snmpInASNParseErrs.0 = Counter32: 0 SNMPv2-MIB::snmpEnableAuthenTraps.0 = INTEGER: enabled(1) SNMPv2-MIB::snmpSilentDrops.0 = Counter32: 0 SNMPv2-MIB::snmpProxyDrops.0 = Counter32: 0 But when I try to call snmpget specifying the OID I outlined in the mib (1.3.6.1.4.1.31130.7.1, the Object whose name matches the mnesia table) and get back: snmpget -v 2c -c public localhost:4000 1.3.6.1.4.1.31130 SNMPv2-SMI::enterprises.31130 = No Such Object available on this agent at this OID I configured the agent similarly to trapexit's SNMP Quick Start Guide , found here: http://www.trapexit.org/SNMP_Quick_Start# What am I missing? Does this seem to be a configuration issue? Or am I simply misusing snmpget? I haven't included any config files or code snippets as I was unsure if it would be needed. If you would like to know more don't hesitate to ask. John Koenig From erlang-questions_efine@REDACTED Wed Jun 4 15:52:58 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Wed, 4 Jun 2008 09:52:58 -0400 Subject: [erlang-questions] Allowed node names In-Reply-To: References: <95be1d3b0806030207y32fad332g390ea226b73e9fb4@mail.gmail.com> <6c2563b20806030751v51830b49p216daaf4b31e856@mail.gmail.com> Message-ID: <6c2563b20806040652x6aec20e9ndfef1e230a8ded11@mail.gmail.com> Heck, you are right. I didn't really look at the RE in the comment closely. ([A-Z][a-z][0-9]_-)+ will match Bd9_-, but not abcde. That's wild. On Wed, Jun 4, 2008 at 1:09 AM, Richard A. O'Keefe wrote: > > On 4 Jun 2008, at 2:51 am, Edwin Fine wrote: > > Gosh, Vlad, how can you say it is not documented? I mean, it clearly >> states in the comments to the Erlang C source code in >> erts/emulator/beam/dist.c that >> >> ** >> ** n must be a valid node name: string of ([a-z][A-Z][0-9]_-)+ >> ** >> >> It only took me 20 minutes to track this down. Don't you know, Real >> Programmers Don't Need No Steenking Documentation, They Just Read The Code >> ;-) >> >> What makes this savage joke even better is that > >> The comment is WRONG << > > It should be [-_a-zA-Z0-9]+ . > 'a@REDACTED' *is* valid, but since it isn't > a lower case letter followed by an upper case letter > followed by a digit followed by a dash or underscore, > the comment claims that it isn't. > > It gets even better: the next comment, "Scanned past the host name", > is *also* wrong. The bit that has been scanned past is the *node* > name; the host name follows the subsequent '@'. > > -- > "I don't want to discuss evidence." -- Richard Dawkins, in an > interview with Rupert Sheldrake. (Fortean times 232, p55.) > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steven.charles.davis@REDACTED Wed Jun 4 16:06:27 2008 From: steven.charles.davis@REDACTED (Steve Davis) Date: Wed, 4 Jun 2008 07:06:27 -0700 (PDT) Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> Message-ID: <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> I just reread the entire thread and have three short "addenda"... 1) I realized that I simply reposted the link to process-one that Dale Harvey introduced and set me off on looking at ejabberd - apologies for that. 2) Joe's very first post in the thread suggests using mnesia as a front-end cache - for a first step in dealing with scaling - i.e. evolving your architecture to keep step with demand and so "go vertical first" - this suggestion seems to me to be exactly right. 3) What has struck me from this discussion is that there are a large number of options *already available* as good architectural components for building full-scale systems. Beyond hardware and the inevitable(?) RDBMS, everything can be done with Erlang/OTP + OTP applications. Interesting... From dmorton@REDACTED Wed Jun 4 16:16:09 2008 From: dmorton@REDACTED (Damien Morton) Date: Thu, 05 Jun 2008 00:16:09 +1000 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> Message-ID: <4846A3A9.9090406@bitfurnace.com> I think Bob hit the nail on the head with this. No database is needed. Everything can pretty much be achieved with files containing fixed-length records. On 6/5/2008 12:06 AM, Steve Davis wrote: > I just reread the entire thread and have three short "addenda"... > 1) I realized that I simply reposted the link to process-one that Dale > Harvey introduced and set me off on looking at ejabberd - apologies > for that. > 2) Joe's very first post in the thread suggests using mnesia as a > front-end cache - for a first step in dealing with scaling - i.e. > evolving your architecture to keep step with demand and so "go > vertical first" - this suggestion seems to me to be exactly right. > 3) What has struck me from this discussion is that there are a large > number of options *already available* as good architectural components > for building full-scale systems. Beyond hardware and the inevitable(?) > RDBMS, everything can be done with Erlang/OTP + OTP applications. > Interesting... > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > > From anders.nygren@REDACTED Wed Jun 4 16:24:32 2008 From: anders.nygren@REDACTED (Anders Nygren) Date: Wed, 4 Jun 2008 09:24:32 -0500 Subject: [erlang-questions] Allowed node names In-Reply-To: <6c2563b20806040652x6aec20e9ndfef1e230a8ded11@mail.gmail.com> References: <95be1d3b0806030207y32fad332g390ea226b73e9fb4@mail.gmail.com> <6c2563b20806030751v51830b49p216daaf4b31e856@mail.gmail.com> <6c2563b20806040652x6aec20e9ndfef1e230a8ded11@mail.gmail.com> Message-ID: 2008/6/4 Edwin Fine : > Heck, you are right. I didn't really look at the RE in the comment closely. > ([A-Z][a-z][0-9]_-)+ will match Bd9_-, but not abcde. That's wild. What did I tell You? Never, ever, trust the comments. /Anders > > On Wed, Jun 4, 2008 at 1:09 AM, Richard A. O'Keefe > wrote: >> >> On 4 Jun 2008, at 2:51 am, Edwin Fine wrote: >> >>> Gosh, Vlad, how can you say it is not documented? I mean, it clearly >>> states in the comments to the Erlang C source code in >>> erts/emulator/beam/dist.c that >>> >>> ** >>> ** n must be a valid node name: string of ([a-z][A-Z][0-9]_-)+ >>> ** >>> >>> It only took me 20 minutes to track this down. Don't you know, Real >>> Programmers Don't Need No Steenking Documentation, They Just Read The Code >>> ;-) >>> >> What makes this savage joke even better is that >> >> The comment is WRONG << >> >> It should be [-_a-zA-Z0-9]+ . >> 'a@REDACTED' *is* valid, but since it isn't >> a lower case letter followed by an upper case letter >> followed by a digit followed by a dash or underscore, >> the comment claims that it isn't. >> >> It gets even better: the next comment, "Scanned past the host name", >> is *also* wrong. The bit that has been scanned past is the *node* >> name; the host name follows the subsequent '@'. >> >> -- >> "I don't want to discuss evidence." -- Richard Dawkins, in an >> interview with Rupert Sheldrake. (Fortean times 232, p55.) >> >> >> >> >> >> >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From per.melin@REDACTED Wed Jun 4 16:34:02 2008 From: per.melin@REDACTED (Per Melin) Date: Wed, 4 Jun 2008 16:34:02 +0200 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <4846A3A9.9090406@bitfurnace.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> Message-ID: 2008/6/4 Damien Morton : > I think Bob hit the nail on the head with this. > > No database is needed. Everything can pretty much be achieved with files > containing fixed-length records. And indeed, Twitter wrote on their blog Saturday: "Our new architecture will move our reliance to a simple, elegant filesystem-based approach, rather than a collection of database [sic]." From steven.charles.davis@REDACTED Wed Jun 4 16:56:14 2008 From: steven.charles.davis@REDACTED (Steve Davis) Date: Wed, 4 Jun 2008 07:56:14 -0700 (PDT) Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <4846A3A9.9090406@bitfurnace.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> Message-ID: On Jun 4, 9:16 am, Damien Morton wrote: > I think Bob hit the nail on the head with this. > > No database is needed. Everything can pretty much be achieved with files > containing fixed-length records. What you and Bob have said is, of course, true. I suspect that as the system becomes large scale then the software required to access and manage those files would eventually end up looking a lot like one of... an RDBMS, a Message Queue, an IM server, etc. From steven.charles.davis@REDACTED Wed Jun 4 17:02:47 2008 From: steven.charles.davis@REDACTED (Steve Davis) Date: Wed, 4 Jun 2008 08:02:47 -0700 (PDT) Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <4846A3A9.9090406@bitfurnace.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> Message-ID: On Jun 4, 9:16 am, Damien Morton wrote: > No database is needed. Everything can pretty much be achieved with files > containing fixed-length records. ...but you probably took my comment "and the inevitable(?) RDBMS" as meaning that the architecture should be database-centric - actually that's not what it meant -- as I described previously the RDBMS does have a place for user accounts (not for the tweet storage). The "inevitability" part was referring to the fact that pretty much any (not all!) large-scale architecture will "need" an RDBMS "somewhere". From vlm@REDACTED Wed Jun 4 17:12:28 2008 From: vlm@REDACTED (Lev Walkin) Date: Wed, 04 Jun 2008 08:12:28 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> Message-ID: <4846B0DC.4010004@lionet.info> Steve Davis wrote: > > On Jun 4, 9:16 am, Damien Morton wrote: >> I think Bob hit the nail on the head with this. >> >> No database is needed. Everything can pretty much be achieved with files >> containing fixed-length records. > > What you and Bob have said is, of course, true. I suspect that as the > system becomes large scale then the software required to access and > manage those files would eventually end up looking a lot like one > of... an RDBMS, a Message Queue, an IM server, etc. ... yet, with relaxed ACID properties, making it much more scalable. From bob@REDACTED Wed Jun 4 17:18:20 2008 From: bob@REDACTED (Bob Ippolito) Date: Wed, 4 Jun 2008 08:18:20 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> Message-ID: <6a36e7290806040818s16050118yce8463ec3a15ac7e@mail.gmail.com> On Wed, Jun 4, 2008 at 7:34 AM, Per Melin wrote: > 2008/6/4 Damien Morton : >> I think Bob hit the nail on the head with this. >> >> No database is needed. Everything can pretty much be achieved with files >> containing fixed-length records. > > And indeed, Twitter wrote on their blog Saturday: > "Our new architecture will move our reliance to a simple, elegant > filesystem-based approach, rather than a collection of database > [sic]." I totally forgot that twitter messages are limited to 160 characters. Yeah, that makes it even easier... with those constraints I doubt I'd have even used a database in a prototype, let alone production. -bob From bekesa@REDACTED Wed Jun 4 17:54:44 2008 From: bekesa@REDACTED (Andras Georgy Bekes) Date: Wed, 04 Jun 2008 17:54:44 +0200 Subject: [erlang-questions] eep: multiple patterns In-Reply-To: <8209f740806040116p578ab5b1laccb6e2509b06f3@mail.gmail.com> References: <200805141620.19559.bekesa@sch.bme.hu> <8209f740806040116p578ab5b1laccb6e2509b06f3@mail.gmail.com> Message-ID: <200806041754.45325.bekesa@sch.bme.hu> > With dialyzer, many of the reasons for doing this go away, but > dialyzer will not detect typos in receive patterns. Dialyzer could then detect singleton-atoms, i.e. atoms that show up only once, in only one module. Georgy From steven.charles.davis@REDACTED Wed Jun 4 18:08:48 2008 From: steven.charles.davis@REDACTED (Steve Davis) Date: Wed, 4 Jun 2008 09:08:48 -0700 (PDT) Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <4846B0DC.4010004@lionet.info> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> <4846B0DC.4010004@lionet.info> Message-ID: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> On Jun 4, 10:12 am, Lev Walkin wrote: > ... yet, with relaxed ACID properties, making it much more scalable. The level to which I disagree with your conclusion here is quite profound! I see a large-scale future of lost messages, data corruption, file replication and synchronization issues, and *downtime* for the FS approach. The raw FS solution IMHO will only "work" up to a point... in a sense the FS suggestion parallels (but is not the same as) MySQL's early and conscious sacrifice of referential integrity for performance (InnoDB does not resolve this btw, for those that know the issues with MySQL). This decision at MySQL has resulted in numerous high-profile scaling issues for many services that committed to MySQL for their persistence needs. Twitter isn't a good example of this kind of failure - it shouldn't have been using the RDBMS at all in the way that it did. All in all, the fundamental application scope of Twitter simply *screams* Message Queue at me. I'm not sure why the "experts" that Twitter have scavenged from IBM and Google haven't come to that conclusion also. Since Twitter have appeared to commit to a FS approach, I guess we'll have to see if future history proves me incorrect :) /s From als@REDACTED Wed Jun 4 16:48:41 2008 From: als@REDACTED (Anthony Shipman) Date: Thu, 5 Jun 2008 00:48:41 +1000 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <95be1d3b0806040627x6a0105c0x3a9147d264c5924@mail.gmail.com> References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <8268eea30806040551r46127891hd2a9db0d8882c7e8@mail.gmail.com> <95be1d3b0806040627x6a0105c0x3a9147d264c5924@mail.gmail.com> Message-ID: <200806050048.41673.als@iinet.net.au> On Wed, 4 Jun 2008 11:27:02 pm Vlad Dumitrescu wrote: > On Wed, Jun 4, 2008 at 2:51 PM, Andreas Hillqvist < > > I think that conceptually, multiple queues would be just like wrapping all > messages with {QueueId, Message} and use selective receive to discriminate, > only faster. Especially if one queue is low priority and high volume. > > I think that it is enough with only one extra queue, for system messages. > With it, it's very easy to implement any number of queues with the help of > assisting processes. The system queue would be used to send receive > requests to the helper process without these requests getting in a possibly > large queue of normal messages. First-class queues are used in Concurrent ML to good effect. If a process wants to make an RPC it creates a queue for the response and passes a handle to it in the call. The remote procedure sends its response through the queue. The caller blocks on pulling the response from the queue. The equivalent in Erlang is to shove all messages into the one process queue and use a reference to pull out the response. This can lead to the slow queue searching that everyone is talking about. Changing the paradigm now though would make the language rather messy. -- Anthony Shipman Mamas don't let your babies als@REDACTED grow up to be outsourced. From geoff@REDACTED Wed Jun 4 18:13:15 2008 From: geoff@REDACTED (Geoff Lane) Date: Wed, 04 Jun 2008 11:13:15 -0500 Subject: [erlang-questions] OT: Reply-To? In-Reply-To: <3225392D-B203-45C1-9BB2-9AB5130F3987@duomark.com> References: <004a01c8c47e$9a355910$6501a8c0@moneymaker2> <3225392D-B203-45C1-9BB2-9AB5130F3987@duomark.com> Message-ID: <4846BF1B.6070801@zorched.net> Jay Nelson wrote: > Valentin accidentally replied directly to me: Maybe the list admins can set the reply-to address to go to the list? I just did this too - and it seems to happen quite a bit... -- Geoff Lane From dmorton@REDACTED Wed Jun 4 17:16:48 2008 From: dmorton@REDACTED (Damien Morton) Date: Thu, 05 Jun 2008 01:16:48 +1000 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> Message-ID: <4846B1E0.1010105@bitfurnace.com> Well, the relational database is suitable for the quickly evolving part of their business, and there will always be a quickly evolving part of any business, so yes - inevitability is the perfect description. Now that their business has stabilised and they know what a larger part of it is supposed to do and what the performance goals they need to reach are, they can begin to crate a more specialised datastructure optimised for their (now known) use case. Question is - can they create such a specialised datastructure such that it _can_ evolve, for example, in the case where they wanted to handle not just 140 byte tweets, but also photos and video (twotos, and twideos)? On 6/5/2008 1:02 AM, Steve Davis wrote: > On Jun 4, 9:16 am, Damien Morton wrote: > >> No database is needed. Everything can pretty much be achieved with files >> containing fixed-length records. >> > > ...but you probably took my comment "and the inevitable(?) RDBMS" as > meaning that the architecture should be database-centric - actually > that's not what it meant -- as I described previously the RDBMS does > have a place for user accounts (not for the tweet storage). The > "inevitability" part was referring to the fact that pretty much any > (not all!) large-scale architecture will "need" an RDBMS "somewhere" From dnew@REDACTED Wed Jun 4 16:45:53 2008 From: dnew@REDACTED (Darren New) Date: Wed, 04 Jun 2008 07:45:53 -0700 Subject: [erlang-questions] eep: multiple patterns In-Reply-To: <8209f740806040116p578ab5b1laccb6e2509b06f3@mail.gmail.com> References: <200805141620.19559.bekesa@sch.bme.hu> <6BA6A9E9-EF89-4FE6-B683-05F3866E9E09@cs.otago.ac.nz> <483D0CB6.2080103@ericsson.com> <483E50BC.2060902@ericsson.com> <61214.63.82.98.51.1212092630.squirrel@mail.webfaction.com> <483F1E59.4090007@ericsson.com> <483FB9E0.80504@ericsson.com> <8209f740806040116p578ab5b1laccb6e2509b06f3@mail.gmail.com> Message-ID: <4846AAA1.4020108@san.rr.com> Ulf Wiger wrote: > This is actually a very strong argument against structuring large > systems using message-passing interfaces, even though that > might seem quite the natural thing to do in Erlang. It's not a fundamental problem. It's a lack of tools. There are a lot of tools for (for example) LOTOS, which is (for purposes of this discussion) very similar to Erlang except much more formal. These tools do things like find deadlocks, tell you whether the new version of your protocol will interact with the older version of your protocol, and so on. Of course, Erlang adds more real-world capabilities (like the idea half the system might crash out from under you) while LOTOS is exceedingly difficult to even compile. And the tools (that is, the theory behind the tools) have bits that make them very slow in cases with large fan-out. But it's practical problems, not theoretical problems, that make figuring out what you "should" receive difficult. -- Darren New / San Diego, CA, USA (PST) "That's pretty. Where's that?" "It's the Age of Channelwood." "We should go there on vacation some time." From steven.charles.davis@REDACTED Wed Jun 4 18:31:45 2008 From: steven.charles.davis@REDACTED (Steve Davis) Date: Wed, 4 Jun 2008 09:31:45 -0700 (PDT) Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <4846B1E0.1010105@bitfurnace.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> <4846B1E0.1010105@bitfurnace.com> Message-ID: <42856f73-4570-43bf-beba-6ad83e74e566@a70g2000hsh.googlegroups.com> On Jun 4, 10:16 am, Damien Morton wrote: > Question is - can they create such a specialised datastructure such that > it _can_ evolve, for example, in the case where they wanted to handle > not just 140 byte tweets, but also photos and video (twotos, and twideos)? Actually, I immediately can see how to cope with this already if were to use the MQ-backend approach. You have a number of regionalized upload servers (i.e. asset servers) where the media gets stored and served - probably exposing RESTful locatars, and the mediatweet that goes through the MQ becomes just a URL reference. Maybe we could even use Joe's Shoutcast server (the Crosswalk Book on p. 259) to stream out the media...? (OK that was just a trick to keep this thread relevant to Erlang :) /s From dmorton@REDACTED Wed Jun 4 18:33:47 2008 From: dmorton@REDACTED (Damien Morton) Date: Thu, 05 Jun 2008 02:33:47 +1000 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> <4846B0DC.4010004@lionet.info> <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> Message-ID: <4846C3EB.5030901@bitfurnace.com> Allow me to disagree with you here. The purpose of an RDMS is to enable easy changes to schemas and making arbitrary queries. It is a thing designed to facilitate rapid changes in direction. If you have a look at the scientific community, for example, if performance is required for a known application, specialised databases tend to flourish. On 6/5/2008 2:08 AM, Steve Davis wrote: > On Jun 4, 10:12 am, Lev Walkin wrote: > >> ... yet, with relaxed ACID properties, making it much more scalable. >> > > The level to which I disagree with your conclusion here is quite > profound! I see a large-scale future of lost messages, data > corruption, file replication and synchronization issues, and > *downtime* for the FS approach. > > The raw FS solution IMHO will only "work" up to a point... in a sense > the FS suggestion parallels (but is not the same as) MySQL's early and > conscious sacrifice of referential integrity for performance (InnoDB > does not resolve this btw, for those that know the issues with MySQL). > This decision at MySQL has resulted in numerous high-profile scaling > issues for many services that committed to MySQL for their persistence > needs. Twitter isn't a good example of this kind of failure - it > shouldn't have been using the RDBMS at all in the way that it did. > > All in all, the fundamental application scope of Twitter simply > *screams* Message Queue at me. I'm not sure why the "experts" that > Twitter have scavenged from IBM and Google haven't come to that > conclusion also. Since Twitter have appeared to commit to a FS > approach, I guess we'll have to see if future history proves me > incorrect :) > > /s > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > > From bob@REDACTED Wed Jun 4 18:36:47 2008 From: bob@REDACTED (Bob Ippolito) Date: Wed, 4 Jun 2008 09:36:47 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <4846B1E0.1010105@bitfurnace.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> <4846B1E0.1010105@bitfurnace.com> Message-ID: <6a36e7290806040936m285ca11o3d5d0839fa1a3ea@mail.gmail.com> Two words: tiny URLs :) That's what people are doing anyway. Something like oEmbed but a little smarter (it needs autodetection, via headers or a microformat) can handle that part: http://oembed.com/ On Wed, Jun 4, 2008 at 8:16 AM, Damien Morton wrote: > Well, the relational database is suitable for the quickly evolving part > of their business, and there will always be a quickly evolving part of > any business, so yes - inevitability is the perfect description. > > Now that their business has stabilised and they know what a larger part > of it is supposed to do and what the performance goals they need to > reach are, they can begin to crate a more specialised datastructure > optimised for their (now known) use case. > > Question is - can they create such a specialised datastructure such that > it _can_ evolve, for example, in the case where they wanted to handle > not just 140 byte tweets, but also photos and video (twotos, and twideos)? > > On 6/5/2008 1:02 AM, Steve Davis wrote: >> On Jun 4, 9:16 am, Damien Morton wrote: >> >>> No database is needed. Everything can pretty much be achieved with files >>> containing fixed-length records. >>> >> >> ...but you probably took my comment "and the inevitable(?) RDBMS" as >> meaning that the architecture should be database-centric - actually >> that's not what it meant -- as I described previously the RDBMS does >> have a place for user accounts (not for the tweet storage). The >> "inevitability" part was referring to the fact that pretty much any >> (not all!) large-scale architecture will "need" an RDBMS "somewhere" > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From vlm@REDACTED Wed Jun 4 18:45:28 2008 From: vlm@REDACTED (Lev Walkin) Date: Wed, 04 Jun 2008 09:45:28 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> <4846B0DC.4010004@lionet.info> <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> Message-ID: <4846C6A8.7080108@lionet.info> Steve Davis wrote: > On Jun 4, 10:12 am, Lev Walkin wrote: >> ... yet, with relaxed ACID properties, making it much more scalable. > > The level to which I disagree with your conclusion here is quite > profound! I see a large-scale future of lost messages, data > corruption, file replication and synchronization issues, and > *downtime* for the FS approach. > > The raw FS solution Who talks about _raw_ FS solution? You are? I am certainly not. > IMHO will only "work" up to a point... in a sense > the FS suggestion parallels (but is not the same as) MySQL's early and > conscious sacrifice of referential integrity for performance (InnoDB > does not resolve this btw, for those that know the issues with MySQL). > This decision at MySQL has resulted in numerous high-profile scaling > issues for many services that committed to MySQL for their persistence > needs. Twitter isn't a good example of this kind of failure - it > shouldn't have been using the RDBMS at all in the way that it did. > > All in all, the fundamental application scope of Twitter simply > *screams* Message Queue at me. I'm not sure why the "experts" that > Twitter have scavenged from IBM and Google haven't come to that > conclusion also. Since Twitter have appeared to commit to a FS > approach, I guess we'll have to see if future history proves me > incorrect :) Steve, I run a quite high traffic, distributed, file-system based (as opposed to RDBMS based) service, js-kit.com http://www.techaddress.com/2008/05/29/js-kit-scores-deal-with-worldnow-adds-19-million-potential-users/ The service internally provides itself a CID guarantee, with "eventual consistency" instead of A, quite like Amazon SimpleDB: http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html We also use message queues internally to provide reliability and sychronization. One of the most important thing which allowed us to gain scalability is our use of file system storage behind all this CID machinery on top of it. When, as Per Melin says, Twitter wrote on their blog Saturday: "Our new architecture will move our reliance to a simple, elegant filesystem-based approach, rather than a collection of database [sic]." I believe the emphasis was placed on replacement of RDBMS with something _on top of a raw FS_, not just _with_ a raw FS. -- Lev Walkin vlm@REDACTED From per.melin@REDACTED Wed Jun 4 18:46:48 2008 From: per.melin@REDACTED (Per Melin) Date: Wed, 4 Jun 2008 18:46:48 +0200 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> <4846B0DC.4010004@lionet.info> <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> Message-ID: 2008/6/4 Steve Davis : > All in all, the fundamental application scope of Twitter simply > *screams* Message Queue at me. Doesn't a message queue imply that each message is, at some point, delivered and then removed from the queue? In that case: Delivered where? The two main interfaces to Twitter, their website and their HTTP API, both just fetch and show the N last messages received. They don't even have a read/unread flag on the messages. It seems to me that it's more of a message log than a message queue. From fritchie@REDACTED Wed Jun 4 18:51:36 2008 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Wed, 04 Jun 2008 11:51:36 -0500 Subject: [erlang-questions] how: prefix matching on external term format In-Reply-To: Message of "Mon, 02 Jun 2008 10:17:27 PDT." Message-ID: <50549.1212598296@snookles.snookles.com> Paul Mineiro wrote: pm> so i've envisioned the following choices: Hrm, I wonder if it's a viable option to avoid the official Erlang external format? If I recall correctly(*), any tuple stored in ETS is not serialized using the external term format. Instead, the tuple is copied into a separate heap. That heap is big enough to store only the tuple. But the data inside that heap uses the same tagged pointer scheme as any Erlang term inside a process's heap, so all the term comparison macros & functions work, regardless of type (process heap or ETS mini-heap). (*) If I'm wrong, someone please yell loudly at me and the list. I wouldn't want the falsehood to live unrefuted forever. If my memory is correct, then storing an ETS mini-heap onto disk will suffer from pointer offset problems: the VM's tagged pointers for use in RAM won't point to the necessary byte offset within a file on disk. The simplest (?) kludge would be to write an ETS mini-heap with all pointers rewritten to be relative to the start of the mini-heap, and then re-set those pointers to RAM-valid values after reading from disk but before any comparisons. (Simplest = avoid reimplementing term comparison and match spec functions.) -Scott From icfp.publicity@REDACTED Wed Jun 4 18:52:52 2008 From: icfp.publicity@REDACTED (Matthew Fluet (ICFP Publicity Chair)) Date: Wed, 4 Jun 2008 11:52:52 -0500 Subject: [erlang-questions] ICFP2008 Call for Poster proposals Message-ID: <53ff55480806040952y7f8c8019x3b3c9ea6468a2caf@mail.gmail.com> ICFP 2008 poster session September 21, 2008 Call for presentation proposals ICFP 2008 will feature a poster session for researchers and practitioners, including students. The session will provide friendly feedback for work that is in gestation or ongoing, as well as opportunities to meet each other and exchange ideas. We welcome poster submissions on all ICFP topics, especially presentations of - applications of and to functional programming; - recent work presented at more distant venues; and - ongoing work, whether or not submitted to ICFP. There will be no formal proceedings, but presenters will be invited to submit working notes, demo code, and other materials to supplement their abstract and poster. These materials will be released informally on a Web page dedicated to the poster session. An accepted submission is not intended to replace conference or journal publication. Persons interested in presenting a poster are invited to submit a one-page abstract in SIGPLAN conference style http://www.acm.org/sigs/sigplan/authorInformation.htm to the Web site https://www.softconf.com/s08/icfp08-posters/submit.html by June 30, 2008. The program committee will review the submissions for relevance and interest, and notify the authors by July 14, 2008. Accepted posters must be presented by the authors in person on Sunday, September 21, 2008. Important dates: Submission: Monday, June 30, 2008 Notification: Monday, July 14, 2008 Presentation: Sunday, September 21, 2008 Program committee: Benjamin Pierce (University of Pennsylvania) Colin Runciman (University of York) Chung-chieh Shan (Rutgers University) From fritchie@REDACTED Wed Jun 4 18:57:36 2008 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Wed, 04 Jun 2008 11:57:36 -0500 Subject: [erlang-questions] Relocating Erlang installation on Windows? In-Reply-To: Message of "Tue, 03 Jun 2008 09:12:23 +0200." <351ce2d70806030012i3c846706jc30aea6b173da705@mail.gmail.com> Message-ID: <50952.1212598656@snookles.snookles.com> Johan Holmberg wrote: jh> Doesn't your script still produce an installation that is not jh> position independent, just relocated to another place? (since one jh> absolute path is replaced with a new one in the scripts). Yup, you're correct. Packaging that's path/position independent requires more packaging mucking than I was willing to endure, and, as you've pointed out, the packaging is different for Windows (which I don't use regularly). I don't know what the Wings3D folks do for packaging the Erlang runtime, but it might be worth taking a peek; most of its installations use the Windows platform, IIRC. -Scott From thomasl_erlang@REDACTED Wed Jun 4 19:04:09 2008 From: thomasl_erlang@REDACTED (Thomas Lindgren) Date: Wed, 4 Jun 2008 10:04:09 -0700 (PDT) Subject: [erlang-questions] eep: multiple patterns In-Reply-To: <8209f740806040116p578ab5b1laccb6e2509b06f3@mail.gmail.com> Message-ID: <706270.31513.qm@web38807.mail.mud.yahoo.com> --- Ulf Wiger wrote: > > One of > > the things that struck me when trying to read > other people's > > Erlang was that I could tell without enormous > trouble (because > > I hadn't seen the system Ulf Wiger was talking > about!) what > > messages processes DID receive, but it was very > much harder > > to tell what messages they SHOULD receive. > > Yes. > > This is actually a very strong argument against > structuring large > systems using message-passing interfaces, even > though that > might seem quite the natural thing to do in Erlang. > Using function > interfaces, the compiler, xref, dialyzer et al can > give you lots of > help, but with message passing no tool support can > help you > figure out (1) whether the message you send is > properly formatted, > or (2) how to jump to the place in the code where > the message is > handled. > > The bigger the system, the more terrible this > problem becomes. Sven-Olof Nystrom at Uppsala some years ago presented an Erlang analyzer that could derive 'process types'. As I understood it, those types were more or less automata describing the process mailbox, so you could, for instance, check whether a sequence of messages was valid. (As an example, such types can handle processes with multiple receive statements with some precision.) Anyway, I haven't heard anything about that for a while, but there's still a page and a download: http://user.it.uu.se/~svenolof/SA/ Best, Thomas From paul-trapexit@REDACTED Wed Jun 4 19:10:26 2008 From: paul-trapexit@REDACTED (Paul Mineiro) Date: Wed, 4 Jun 2008 10:10:26 -0700 (PDT) Subject: [erlang-questions] how: prefix matching on external term format In-Reply-To: <50549.1212598296@snookles.snookles.com> References: <50549.1212598296@snookles.snookles.com> Message-ID: Oh hey since there's interest, I got this working by forking erl_compare_ext and adding special tags for smallest and largest erlang term. My original fear of forking was unfounded since 1) the canonical erl_compare_ext has several bugs and 2) the canonical erl_compare_ext does not support ERL_NEW_FLOAT_EXT. That last one was pretty funny since my main fear of forking was missing out on new type tags. Well, the directory *does* say "legacy" ... I'm in the end game now, within a week I should have something on google code that sports the same interface as dets but supports ordered_set and ordered_bag (and 8EB [9.22e18 bytes] byte file size limit). -- p On Wed, 4 Jun 2008, Scott Lystig Fritchie wrote: > Paul Mineiro wrote: > > pm> so i've envisioned the following choices: > > Hrm, I wonder if it's a viable option to avoid the official Erlang > external format? If I recall correctly(*), any tuple stored in ETS is > not serialized using the external term format. Instead, the tuple is > copied into a separate heap. That heap is big enough to store only the > tuple. But the data inside that heap uses the same tagged pointer > scheme as any Erlang term inside a process's heap, so all the term > comparison macros & functions work, regardless of type (process heap or > ETS mini-heap). > > (*) If I'm wrong, someone please yell loudly at me and the list. I > wouldn't want the falsehood to live unrefuted forever. > > If my memory is correct, then storing an ETS mini-heap onto disk will > suffer from pointer offset problems: the VM's tagged pointers for use in > RAM won't point to the necessary byte offset within a file on disk. The > simplest (?) kludge would be to write an ETS mini-heap with all pointers > rewritten to be relative to the start of the mini-heap, and then re-set > those pointers to RAM-valid values after reading from disk but before > any comparisons. (Simplest = avoid reimplementing term comparison and > match spec functions.) > > -Scott > In an artificial world, only extremists live naturally. -- Paul Graham From pfisher@REDACTED Wed Jun 4 19:12:04 2008 From: pfisher@REDACTED (Paul Fisher) Date: Wed, 04 Jun 2008 12:12:04 -0500 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> <4846B0DC.4010004@lionet.info> <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> Message-ID: <1212599524.7281.24.camel@localhost> On Wed, 2008-06-04 at 09:08 -0700, Steve Davis wrote: > On Jun 4, 10:12 am, Lev Walkin wrote: > > ... yet, with relaxed ACID properties, making it much more scalable. > > The level to which I disagree with your conclusion here is quite > profound! I see a large-scale future of lost messages, data > corruption, file replication and synchronization issues, and > *downtime* for the FS approach. > I promise that the RDBMS wall exists... we deal with more "records" that you would ever imagine putting into a relational database. For example, on one small portion of our infrastructure we still use MySQL databases with 8-core 32GB machines, and we have to limit each instance to no more than a few days worth of data in order to keep it going with reasonable performance. Imagine having to commission database instances every other day of 100s of GBs. Any of the Google papers or recent articles from Stonebreaker discuss this in directly in terms of the limits of RDBMS systems. Finally, i am also here to attest that the filesystem approach scales well past were the RDBMS falls over. You just need to architect the system to balance the trade-offs, which naturally exist at extreme levels of scale. -- paul From kevin@REDACTED Wed Jun 4 19:22:42 2008 From: kevin@REDACTED (Kevin Scaldeferri) Date: Wed, 4 Jun 2008 10:22:42 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> <4846B0DC.4010004@lionet.info> <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> Message-ID: <47334298-E090-4B08-B796-082D9F964328@scaldeferri.com> On Jun 4, 2008, at 9:46 AM, Per Melin wrote: > 2008/6/4 Steve Davis : >> All in all, the fundamental application scope of Twitter simply >> *screams* Message Queue at me. > > Doesn't a message queue imply that each message is, at some point, > delivered and then removed from the queue? > > In that case: Delivered where? > > The two main interfaces to Twitter, their website and their HTTP API, > both just fetch and show the N last messages received. Most people seem to get tweets via IM or SMS, I thought. -k From kevin@REDACTED Wed Jun 4 19:23:50 2008 From: kevin@REDACTED (Kevin Scaldeferri) Date: Wed, 4 Jun 2008 10:23:50 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <4846C3EB.5030901@bitfurnace.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> <4846B0DC.4010004@lionet.info> <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> <4846C3EB.5030901@bitfurnace.com> Message-ID: <2030E1A3-16C0-4DDB-984F-2268DA48B878@scaldeferri.com> On Jun 4, 2008, at 9:33 AM, Damien Morton wrote: > The purpose of an RDMS is to enable easy changes to schemas and making > arbitrary queries. It is a thing designed to facilitate rapid > changes in > direction. Could you tell that to my DBAs? ;-) Seems like it usually takes months to make a schema change around here. -k From steven.charles.davis@REDACTED Wed Jun 4 19:29:14 2008 From: steven.charles.davis@REDACTED (Steve Davis) Date: Wed, 4 Jun 2008 10:29:14 -0700 (PDT) Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <4846C6A8.7080108@lionet.info> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <9b08084c0805300718i636d1d8ci4b923ffe335ec8f9@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> <4846B0DC.4010004@lionet.info> <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> <4846C6A8.7080108@lionet.info> Message-ID: On Jun 4, 11:45 am, Lev Walkin wrote: > Who talks about _raw_ FS solution? You are? I am certainly not. Lev: Actually, I didn't see anyone mention anything other than a raw FS solution, hence the "rider", so thanks for expanding on your initial comment! > Steve, I run a quite high traffic, distributed, file-system > based (as opposed to RDBMS based) service, js-kit.com > > http://www.techaddress.com/2008/05/29/js-kit-scores-deal-with-worldno... > > The service internally provides itself a CID guarantee, > with "eventual consistency" instead of A, quite like Amazon SimpleDB: > > http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html > > We also use message queues internally to provide reliability and > sychronization. > > One of the most important thing which allowed us to gain scalability > is our use of file system storage behind all this CID machinery on top > of it. At first sight, I don't see the immediate applicability of these architectures to the "twitter problem", but there's a lot of detail there so I'll take the time to look more closely and actually fully read the amazon paper to see what's there. At the very least, it should be an interesting/informative read. > When, as Per Melin says, Twitter wrote on their blog Saturday: > "Our new architecture will move our reliance to a simple, elegant > filesystem-based approach, rather than a collection of database > [sic]." > > I believe the emphasis was placed on replacement of RDBMS > with something _on top of a raw FS_, not just _with_ a raw FS. Whilst elegant, neither your solution nor Amazon's seem "simple". One can only hope that Twitter have thought it through to the extent you have for your architecture, and that the "something on top" that they have in mind does actually address the issues that I raised. For now and for me, the best candidate for that "something on top of the FS" remains a Message Queue :) To Per Melin: I'm thinking subscription queues (pubsub) and messages having a time-to-live (TTL). Perhaps one subscriber might be a logger for permanent record. /s From fritchie@REDACTED Wed Jun 4 19:54:12 2008 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Wed, 04 Jun 2008 12:54:12 -0500 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: Message of "Wed, 04 Jun 2008 09:08:48 PDT." <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> Message-ID: <54568.1212602052@snookles.snookles.com> Steve Davis wrote: sd> All in all, the fundamental application scope of Twitter simply sd> *screams* Message Queue at me. I'm not sure why the "experts" that sd> Twitter have scavenged from IBM and Google haven't come to that sd> conclusion also. Since Twitter have appeared to commit to a FS sd> approach, I guess we'll have to see if future history proves me sd> incorrect :) In the serious, serious business of arm-chair architecting, I'd say that a file system approach is almost certainly more scalable than their current implemention. As for "message queueing", there may be a misunderstanding over how MQ systems typically work: they have producers *and* consumers, and (more importantly) consumers actually "consume". Consuming a queue item usually means also deleting it from the queue. A single Twitter user X can have thousands of consumers all trying to consume the same messages, but in a typical MQ system, all but the first consumer would find X's queue empty. For one example, see the RabbitMQ FAQ, "Q. How do I archive to RDBMS?". -Scott From steven.charles.davis@REDACTED Wed Jun 4 20:10:17 2008 From: steven.charles.davis@REDACTED (Steve Davis) Date: Wed, 4 Jun 2008 11:10:17 -0700 (PDT) Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <54568.1212602052@snookles.snookles.com> References: <54568.1212602052@snookles.snookles.com> Message-ID: Scott Lystig Fritchie wrote: > In the serious, serious business of arm-chair architecting... http://en.wikipedia.org/wiki/Publish/subscribe BTW I find veiled ad hominem attacks somewhat distasteful. From yarivsadan@REDACTED Wed Jun 4 20:16:08 2008 From: yarivsadan@REDACTED (Yariv Sadan) Date: Wed, 4 Jun 2008 11:16:08 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <4846B1E0.1010105@bitfurnace.com> References: <17244f480805282212g32eaa0e1o78493087bd12fd0@mail.gmail.com> <17244f480805311504h8845f34h50bf3bf6bd3bc00c@mail.gmail.com> <65f74def-b907-4bc9-8b37-31da346cefee@d45g2000hsc.googlegroups.com> <17244f480806032145j41686f94n567f5e2820417c72@mail.gmail.com> <8f06f6ac-0685-4306-93c5-968b5b39e7d2@z72g2000hsb.googlegroups.com> <3f2c3db2-69ae-4eec-b296-b6186e99cf51@a1g2000hsb.googlegroups.com> <4846A3A9.9090406@bitfurnace.com> <4846B1E0.1010105@bitfurnace.com> Message-ID: <17244f480806041116i3f96a171jddd466ff150439fc@mail.gmail.com> You nailed it. Just last night I added a couple of columns to the 'msg' table to be able to store the user's gravatar settings in a denormalized fashion. If I had committed to a fixed file-based format for all user archives, making such a change would have been a pretty big pain. I see file based storage as premature optimization. You sacrifice functionality for speed. That's probably why Google built BigTable on top of GFS. Yariv On Wed, Jun 4, 2008 at 8:16 AM, Damien Morton wrote: > Well, the relational database is suitable for the quickly evolving part > of their business, and there will always be a quickly evolving part of > any business, so yes - inevitability is the perfect description. > > Now that their business has stabilised and they know what a larger part > of it is supposed to do and what the performance goals they need to > reach are, they can begin to crate a more specialised datastructure > optimised for their (now known) use case. > > Question is - can they create such a specialised datastructure such that > it _can_ evolve, for example, in the case where they wanted to handle > not just 140 byte tweets, but also photos and video (twotos, and twideos)? > > On 6/5/2008 1:02 AM, Steve Davis wrote: >> On Jun 4, 9:16 am, Damien Morton wrote: >> >>> No database is needed. Everything can pretty much be achieved with files >>> containing fixed-length records. >>> >> >> ...but you probably took my comment "and the inevitable(?) RDBMS" as >> meaning that the architecture should be database-centric - actually >> that's not what it meant -- as I described previously the RDBMS does >> have a place for user accounts (not for the tweet storage). The >> "inevitability" part was referring to the fact that pretty much any >> (not all!) large-scale architecture will "need" an RDBMS "somewhere" > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From mbj@REDACTED Wed Jun 4 20:47:22 2008 From: mbj@REDACTED (Martin Bjorklund) Date: Wed, 04 Jun 2008 20:47:22 +0200 (CEST) Subject: [erlang-questions] Troubles Bridging SNMP and Mnesia In-Reply-To: <48469788.9070104@acalledshot.net> References: <48469788.9070104@acalledshot.net> Message-ID: <20080604.204722.149632208.mbj@tail-f.com> Hi, John Koenig wrote: > But when I try to call snmpget specifying the OID I outlined in the mib > (1.3.6.1.4.1.31130.7.1, the Object whose name matches the mnesia table) > and get back: > > snmpget -v 2c -c public localhost:4000 1.3.6.1.4.1.31130 The OID on the line above ask for the enterprise OID 31130 (Basho Technologies). But you said you wanted to ask for enterprises.31130.7.1. So either try to ask directly for enterprises.31130.7.1, or even better, do snmpwalk -v 2c -c public localhost:4000 1.3.6.1.4.1.31130 to print all objects in you entererprise MIB. /martin From koenigjm@REDACTED Wed Jun 4 21:14:04 2008 From: koenigjm@REDACTED (John Koenig) Date: Wed, 04 Jun 2008 15:14:04 -0400 Subject: [erlang-questions] Troubles Bridging SNMP and Mnesia In-Reply-To: <20080604.204722.149632208.mbj@tail-f.com> References: <48469788.9070104@acalledshot.net> <20080604.204722.149632208.mbj@tail-f.com> Message-ID: <4846E97C.9080202@acalledshot.net> Martin and All, I have fiddled with my configuration a bit and was finally able to get snmpwalk to browse the mnesia table under 1.3.6.1.4.1.31130 and 1.3.6.1.4.1.31130.7.1 (defined in my POLLDATA.mib). However, an error is thrown when snmpwalk reaches the end of the database. The first error is the one listed in the erlang snmp-agent and the second is output by snmpwalk. ---- Erlang error from publisher_snmp =ERROR REPORT==== 4-Jun-2008::12:58:26 === ** User error: Got {'reporter_server@REDACTED ',reporter} from {snmp_generic,table_func,[{pollData,mnesia}]}. Using wrongType ----Error from snmpwalk > snmpwalk -v 2c -c public localhost:4000 1.3.6.1.4.1.31130 POLLDATA::rowId.1 = INTEGER: 1 POLLDATA::rowId.2 = INTEGER: 2 . . . ?POLLDATA::rowId.614 = INTEGER: 614 POLLDATA::rowId.615 = INTEGER: 615 POLLDATA::rowId.616 = INTEGER: 616 Error in packet. Reason: (genError) A general failure occured Failed object: POLLDATA::rowId.616 The data that appears in the table (all 616 rows of it) looks exactly the same and are of the form: {tableName, rowID (int and key for snmp), {node, process}, stackSize, heapSize, messageqlen, date(), time()} Also, to comple POLLDATA.mib I included a POLLDATA.funcs which contains the following: { pollData, { snmp_generic, table_func, [{pollData, mnesia}]}}. Thoughts? John Martin Bjorklund wrote: > Hi, > > John Koenig wrote: > >> But when I try to call snmpget specifying the OID I outlined in the mib >> (1.3.6.1.4.1.31130.7.1, the Object whose name matches the mnesia table) >> and get back: >> >> snmpget -v 2c -c public localhost:4000 1.3.6.1.4.1.31130 >> > > The OID on the line above ask for the enterprise OID 31130 (Basho > Technologies). But you said you wanted to ask for > enterprises.31130.7.1. > > So either try to ask directly for enterprises.31130.7.1, or even > better, do > > snmpwalk -v 2c -c public localhost:4000 1.3.6.1.4.1.31130 > > to print all objects in you entererprise MIB. > > > /martin > > From mbj@REDACTED Wed Jun 4 21:30:18 2008 From: mbj@REDACTED (Martin Bjorklund) Date: Wed, 04 Jun 2008 21:30:18 +0200 (CEST) Subject: [erlang-questions] Troubles Bridging SNMP and Mnesia In-Reply-To: <4846E97C.9080202@acalledshot.net> References: <48469788.9070104@acalledshot.net> <20080604.204722.149632208.mbj@tail-f.com> <4846E97C.9080202@acalledshot.net> Message-ID: <20080604.213018.89334247.mbj@tail-f.com> John Koenig wrote: > Thoughts? Check out the debug support in snmpa(3). snmpa:verbosity(all, trace). % this usage of 'all' is not % documented in the man page. will give you lots and lots of information. /martin From mbj@REDACTED Wed Jun 4 21:38:22 2008 From: mbj@REDACTED (Martin Bjorklund) Date: Wed, 04 Jun 2008 21:38:22 +0200 (CEST) Subject: [erlang-questions] Troubles Bridging SNMP and Mnesia In-Reply-To: <4846E97C.9080202@acalledshot.net> References: <48469788.9070104@acalledshot.net> <20080604.204722.149632208.mbj@tail-f.com> <4846E97C.9080202@acalledshot.net> Message-ID: <20080604.213822.84731595.mbj@tail-f.com> Hi, Hmm, I didn't look too closely at the error message you got... John Koenig wrote: > ---- Erlang error from publisher_snmp > =ERROR REPORT==== 4-Jun-2008::12:58:26 === > ** User error: Got {'reporter_server@REDACTED ',reporter} from {snmp_generic,table_func,[{pollData,mnesia}]}. Using wrongType This looks weird. You have a really strange atom 'reporter_server@REDACTED '. But maybe the part was added by a clever email program? > The data that appears in the table (all 616 rows of it) looks > exactly the same and are of the form: > > {tableName, rowID (int and key for snmp), {node, process}, > stackSize, heapSize, messageqlen, date(), time()} Aha! You cannot use the 2-tuple {node, process} when you store MIB data. The erlang data type must match the type defined in the MIB. If the second column in your MIB is a string, you must store a string in the Mnesia table. Hope this helps. /martin From gleber.p@REDACTED Wed Jun 4 21:42:49 2008 From: gleber.p@REDACTED (Gleb Peregud) Date: Wed, 4 Jun 2008 21:42:49 +0200 Subject: [erlang-questions] Reia - Python/Ruby mix targeting Beam/HiPE Message-ID: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com> I've stumbled upon this link few moments ago: http://reia-lang.org/ And thought it will be interesing for fellow erlangers. > Reia (pronounced RAY-uh) is a Python/Ruby-like mixed-paradigm language > targeting the Erlang virtual machine (BEAM) and high-performance > native compiler (HiPE). Reia aims to expose all the features and functionality > of Erlang in a language more familiar to programmers of scripting > languages, while improving string handling, regular expressions, > linking with external libraries, and other tasks which are generally > considered outside the scope of Erlang. -- Gleb Peregud http://gleber.pl/ Every minute is to be grasped. Time waits for nobody. -- Inscription on a Zen Gong From ulf.wiger@REDACTED Wed Jun 4 21:48:36 2008 From: ulf.wiger@REDACTED (Ulf Wiger (TN/EAB)) Date: Wed, 04 Jun 2008 21:48:36 +0200 Subject: [erlang-questions] eep: multiple patterns In-Reply-To: <4846AAA1.4020108@san.rr.com> References: <200805141620.19559.bekesa@sch.bme.hu> <6BA6A9E9-EF89-4FE6-B683-05F3866E9E09@cs.otago.ac.nz> <483D0CB6.2080103@ericsson.com> <483E50BC.2060902@ericsson.com> <61214.63.82.98.51.1212092630.squirrel@mail.webfaction.com> <483F1E59.4090007@ericsson.com> <483FB9E0.80504@ericsson.com> <8209f740806040116p578ab5b1laccb6e2509b06f3@mail.gmail.com> <4846AAA1.4020108@san.rr.com> Message-ID: <4846F194.6080202@ericsson.com> Darren New wrote: > Ulf Wiger wrote: > >> This is actually a very strong argument against structuring large >> systems using message-passing interfaces, even though that >> might seem quite the natural thing to do in Erlang. >> > > It's not a fundamental problem. It's a lack of tools. There are a lot of > tools for (for example) LOTOS, which is (for purposes of this > discussion) very similar to Erlang except much more formal. These tools > do things like find deadlocks, tell you whether the new version of your > protocol will interact with the older version of your protocol, and so on. > I didn't mean to imply that it was a fundamental problem, only that it's not a good strategy in large systems due to the lack of tool support. It's possible that some changes/additions to the language might be warranted as well. Time will surely tell... BR, Ulf W From rvirding@REDACTED Wed Jun 4 22:20:28 2008 From: rvirding@REDACTED (Robert Virding) Date: Wed, 4 Jun 2008 22:20:28 +0200 Subject: [erlang-questions] Performance of matches In-Reply-To: References: <483EDB2E.9070103@san.rr.com> <484126E2.5040000@adam.com.au> <4841DBFB.2000308@san.rr.com> <484462D6.8000001@san.rr.com> <97619b170806022112x73792e29y5a612cb966049305@mail.gmail.com> Message-ID: <3dbc6d1c0806041320g6d4aed8axf6c119ab71d60178@mail.gmail.com> 2008/6/4 Richard A. O'Keefe : > > On Mon, Jun 2, 2008 at 11:15 PM, Darren New wrote: > > > > > Good to know the process dictionary is actually a hashtable, but > > there's > > only one per process and they don't nest, so that's pretty limited. > > Sure they nest: a key can map to another process and _that_ process has > another process dictionary. It would be pretty simple to whip up a > module implementing multi-key hash tables that way. There is a much easier way to nest them: the value can be a dict, which is a hash table. :-) Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From circularfunc@REDACTED Wed Jun 4 22:02:48 2008 From: circularfunc@REDACTED (Circular Function) Date: Wed, 4 Jun 2008 20:02:48 +0000 (GMT) Subject: [erlang-questions] running programs from win com prompt? Message-ID: <676945.84592.qm@web28306.mail.ukl.yahoo.com> hi i know how to do basic calculations and write anonymous functions. however i dont understand from the tutorial how to write and save modules and run the from the erlang shell or command prompt. if i save a file as example.erl and try to run it from the commandprompt in windows it says is not recognized as a command etc. also, where should i save a file/module to be able to load it in the shell? __________________________________________________________ Ta semester! - s?k efter resor hos Yahoo! Shopping. J?mf?r pris p? flygbiljetter och hotellrum h?r: http://shopping.yahoo.se/c-169901-resor-biljetter.html?partnerId=96914052 -------------- next part -------------- An HTML attachment was scrubbed... URL: From monch1962@REDACTED Thu Jun 5 00:03:35 2008 From: monch1962@REDACTED (David Mitchell) Date: Thu, 5 Jun 2008 08:03:35 +1000 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <54568.1212602052@snookles.snookles.com> References: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> <54568.1212602052@snookles.snookles.com> Message-ID: 2008/6/5 Scott Lystig Fritchie : > Steve Davis wrote: > As for "message queueing", there may be a misunderstanding over how MQ > systems typically work: they have producers *and* consumers, and (more > importantly) consumers actually "consume". Consuming a queue item > usually means also deleting it from the queue. A single Twitter user X > can have thousands of consumers all trying to consume the same messages, > but in a typical MQ system, all but the first consumer would find X's > queue empty. > > For one example, see the RabbitMQ FAQ, "Q. How do I archive to RDBMS?". In case anyone's losing track, I was the one who suggested keeping tweets in queues essentially forever, and having users retrieve them from queues without deleting the message from the que. I understand how MQ works in normal environments; what I'm suggesting is that Twitter (and any clones) aren't "normal" once they start to scale up to many millions of users. The reasons I suggested storing messages in queues indefinitely are: - experience says that queueing systems can scale very large, and that it appears to be an "easier" problem to solve than scaling a database very large. I'll accept it if anyone complains about "gross generalisation"... - the APIs for storing messages to queues and then retrieving them are designed to be very fast, and (again referencing IBM's MQ) we know they can scale to queues holding very large numbers of messages Storing messages in flat files seems to have a couple of limitations to me: - if you're going to store 1 message per flat file, you need a database (or database-like thing) to track those zillions of flat files. I figure that's going to put you back where you started in terms of scalability - assuming you're always appending messages to the end of flat files, you'd have to assume that most requests will be for the most recent message i.e. the last message in the file. Do you really want to be seeking through to the last record of flat files all the time? That doesn't seem to be a scalable approach - alternately, if you always add the most recent message to the *start* of a flat file, you'll constantly be rewriting the entire file (at least, that's the case in any file system I can think of; there might be an exception). I suppose you could write your own file system to optimise that... Please speak up if you've got any thoughts - I'm treating this like a bunch of intellectuals throwing ideas around, rather than an argument about right and wrong, and it seems that everyone else is too at this stage. Very happy to be convinced I'm wrong, in other words Regards David Mitchell From vlm@REDACTED Thu Jun 5 00:26:17 2008 From: vlm@REDACTED (Lev Walkin) Date: Wed, 04 Jun 2008 15:26:17 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> <54568.1212602052@snookles.snookles.com> Message-ID: <48471689.40701@lionet.info> David Mitchell wrote: > 2008/6/5 Scott Lystig Fritchie : >> Steve Davis wrote: > >> As for "message queueing", there may be a misunderstanding over how MQ >> systems typically work: they have producers *and* consumers, and (more >> importantly) consumers actually "consume". Consuming a queue item >> usually means also deleting it from the queue. A single Twitter user X >> can have thousands of consumers all trying to consume the same messages, >> but in a typical MQ system, all but the first consumer would find X's >> queue empty. >> >> For one example, see the RabbitMQ FAQ, "Q. How do I archive to RDBMS?". > > In case anyone's losing track, I was the one who suggested keeping > tweets in queues essentially forever, and having users retrieve them > from queues without deleting the message from the que. > > I understand how MQ works in normal environments; what I'm suggesting > is that Twitter (and any clones) aren't "normal" once they start to > scale up to many millions of users. > > The reasons I suggested storing messages in queues indefinitely are: > - experience says that queueing systems can scale very large, and that > it appears to be an "easier" problem to solve than scaling a database > very large. I'll accept it if anyone complains about "gross > generalisation"... > - the APIs for storing messages to queues and then retrieving them are > designed to be very fast, and (again referencing IBM's MQ) we know > they can scale to queues holding very large numbers of messages > > Storing messages in flat files seems to have a couple of limitations to me: > - if you're going to store 1 message per flat file, you need a > database (or database-like thing) to track those zillions of flat > files. I figure that's going to put you back where you started in > terms of scalability better coalesce the messages for a particular user's consumption in a single file. better for FS inodes utilization, seek times (latency), disk and memory fragmentation. > - assuming you're always appending messages to the end of flat files, > you'd have to assume that most requests will be for the most recent > message i.e. the last message in the file. Do you really want to be > seeking through to the last record of flat files all the time? That > doesn't seem to be a scalable approach btw, seek to the end is O(1), if not O(0) (jokingly), if the file entries are self-delimiting and (as an optimization) double-tagging (message size at the beginning and at the end of the message). > - alternately, if you always add the most recent message to the > *start* of a flat file, you'll constantly be rewriting the entire file > (at least, that's the case in any file system I can think of; there > might be an exception). I suppose you could write your own file > system to optimise that... this is not necessary. > Please speak up if you've got any thoughts - I'm treating this like a > bunch of intellectuals throwing ideas around, rather than an argument > about right and wrong, and it seems that everyone else is too at this > stage. Very happy to be convinced I'm wrong, in other words > > Regards > > David Mitchell > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From ok@REDACTED Thu Jun 5 01:51:30 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 5 Jun 2008 11:51:30 +1200 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <8268eea30806032346l463f42f4q798f47db08de84b3@mail.gmail.com> References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <8268eea30806032346l463f42f4q798f47db08de84b3@mail.gmail.com> Message-ID: On 4 Jun 2008, at 6:46 pm, Andreas Hillqvist wrote: > As I understand it, the discussion in this thread concerns the process > "default" Message Queue. > > What if it would be possible to create Message Queues in addition to > process "default" Message Queue? > Where it would be possible to give options such as: > * limit to the amount message to store > * what happens when message queue is full > * crasch of client/sender and/or server/receiver > * notification by message > * block sender (I guess this is not a good idea. Because it is > the opposite to the current semantics of erlangs send.) > etc... I'm beginning to sound like Little Johnny One-Note, but we can already get the effect of multiple message queues by having helper processes and giving out their PIDs instead. There's a joke about Haskell programmers: give them a problem and they start by saying "I'll just define a new combinator algebra for that". The joke about Erlang programmers is "I'll just spawn another process for that". This means that we don't find ourselves having to do things like {Pid,urgent} ! Msg or however alternative queues are identified. Special machinery for some of these options could well make them more efficient, but LJON again, we don't desperately need them. Take limiting the size of the mail box. What we do is set up two processes +---------+ +--------------+ clients --> | counter | --> | real process | +---------+ +--------------+ \______________/ where the counter process simply forwards all messages to the real process until it reaches a limit. Periodically, the real process sends a message to the counter process saying that it is willing to receive N more messages. counter(Self, Limit) when Limit > 0 -> receive {Self,Delta} -> counter(Self, Limit+Delta) ; Msg -> Self ! Msg, counter(Self, Limit-1) end; counter(Self, 0) -> receive {Self,Delta} -> counter(Self, Delta) end. We can vary the second clause to discard excess messages, to crash (and via linking, take down Self as well), to send a special message to Self, or what you please. We can use the front end process to filter out bad messages. If we're willing for the front end process to understand the protocol, and the protocol includes the sender's pid, we can track which process sends the most messages and log or kill the culprit on overflow. The sky is the limit. After experimentation, some of these approaches may turn out to be specially valuable, and deserve special support. The Erlang process system as it exists now is *extremely* powerful, despite (because of?) its deceptive simplicity. > > What is your feedback on user defined Message Queues? That they are articifially limited processes, and we might as well use the Real Thing. From nick@REDACTED Thu Jun 5 02:00:03 2008 From: nick@REDACTED (Nick Gerakines) Date: Wed, 4 Jun 2008 17:00:03 -0700 Subject: [erlang-questions] Debugging float math in geohashing module In-Reply-To: <48460FA4.1070908@niemeyer.net> References: <48460FA4.1070908@niemeyer.net> Message-ID: Gustavo, Thanks! Your explanation and a little more reading gave me everything I need. # Nick Gerakines On Tue, Jun 3, 2008 at 8:44 PM, Gustavo Niemeyer wrote: > Hi Nick, > > (...) >> >> The decode functionality is different however. When I attempt to decode a >> geohash into a latitude/longitude pair the end result is slightly* off. >> When > > (...) > > Can you present an example of what you'd expect to happen, and what's > actually happening? > > If I understand what you mean, there might be nothing wrong with your code. > Note that due to the nature of the Geohash algorithm itself, in most cases > you won't be able to decode the value into a precise decimal floating > point. > > What is done by the software behind geohash.org during encoding is to > compute how many bits you need to at least match the decimal precision > given as input, and during decoding the inverse is done: it only shows > the decimal digits it can correctly represent given the number of bits > available. > > As a hint, if you have 8 bits for the longitude code, and a maximum value > of 360, the maximum error is 360 / 2^8, and with log(maximum_error, 10) > you get the decimal exponent of the maximum error, which is pretty close > to what you want. > > -- > Gustavo Niemeyer > http://niemeyer.net > From erlang-questions_efine@REDACTED Thu Jun 5 04:13:19 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Wed, 4 Jun 2008 22:13:19 -0400 Subject: [erlang-questions] Allowed node names In-Reply-To: References: <95be1d3b0806030207y32fad332g390ea226b73e9fb4@mail.gmail.com> <6c2563b20806030751v51830b49p216daaf4b31e856@mail.gmail.com> <6c2563b20806040652x6aec20e9ndfef1e230a8ded11@mail.gmail.com> Message-ID: <6c2563b20806041913o11f4edderbf0995a80c1a32d8@mail.gmail.com> :( You were right. I should have known better On Wed, Jun 4, 2008 at 10:24 AM, Anders Nygren wrote: > 2008/6/4 Edwin Fine : > > Heck, you are right. I didn't really look at the RE in the comment > closely. > > ([A-Z][a-z][0-9]_-)+ will match Bd9_-, but not abcde. That's wild. > > What did I tell You? > Never, ever, trust the comments. > > /Anders > > > > > On Wed, Jun 4, 2008 at 1:09 AM, Richard A. O'Keefe > > wrote: > >> > >> On 4 Jun 2008, at 2:51 am, Edwin Fine wrote: > >> > >>> Gosh, Vlad, how can you say it is not documented? I mean, it clearly > >>> states in the comments to the Erlang C source code in > >>> erts/emulator/beam/dist.c that > >>> > >>> ** > >>> ** n must be a valid node name: string of ([a-z][A-Z][0-9]_-)+ > >>> ** > >>> > >>> It only took me 20 minutes to track this down. Don't you know, Real > >>> Programmers Don't Need No Steenking Documentation, They Just Read The > Code > >>> ;-) > >>> > >> What makes this savage joke even better is that > >> >> The comment is WRONG << > >> > >> It should be [-_a-zA-Z0-9]+ . > >> 'a@REDACTED' *is* valid, but since it isn't > >> a lower case letter followed by an upper case letter > >> followed by a digit followed by a dash or underscore, > >> the comment claims that it isn't. > >> > >> It gets even better: the next comment, "Scanned past the host name", > >> is *also* wrong. The bit that has been scanned past is the *node* > >> name; the host name follows the subsequent '@'. > >> > >> -- > >> "I don't want to discuss evidence." -- Richard Dawkins, in an > >> interview with Rupert Sheldrake. (Fortean times 232, p55.) > >> > >> > >> > >> > >> > >> > >> > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From saleyn@REDACTED Thu Jun 5 04:46:38 2008 From: saleyn@REDACTED (Serge Aleynikov) Date: Wed, 04 Jun 2008 22:46:38 -0400 Subject: [erlang-questions] mnesia:table_info(Table, size) returns 0 for remote tables Message-ID: <4847538E.5040405@gmail.com> I ran into this non-intuitive behavior of the mnesia:table_info(Table, size) function. Unlike other calls of the mnesia:table_info/2 family this one returns 0 if a table is not local. The work around is to do: Node = mnesia:table_info(Table, where_to_read), rpc:call(Node, mnesia, table_info, [Table, size]). Though, it's not clear to me why the caller of mnesia:table_info/2 has to go though this trouble. Can this be fixed? Serge From saleyn@REDACTED Thu Jun 5 05:05:21 2008 From: saleyn@REDACTED (Serge Aleynikov) Date: Wed, 04 Jun 2008 23:05:21 -0400 Subject: [erlang-questions] mnesia:dirty_select/3 feature request Message-ID: <484757F1.10901@gmail.com> Currently there are mnesia:dirty_select/2 and mnesia:select/{3,4} functions, but no mnesia:dirty_select/3 with the following signature: mnesia:dirty_select(Table, MatchSpec, Limit) -> {Records, Continuation} I believe it would be very useful to have this function for cases when a user needs to get only a few matching records out of a very large table. Or perhaps am I missing some other ways to do mnesia selects using fast read matching operations with a Limit clause that don't involve transactions? Serge From saleyn@REDACTED Thu Jun 5 03:56:21 2008 From: saleyn@REDACTED (Serge Aleynikov) Date: Wed, 04 Jun 2008 21:56:21 -0400 Subject: [erlang-questions] Troubles Bridging SNMP and Mnesia In-Reply-To: <4846E97C.9080202@acalledshot.net> References: <48469788.9070104@acalledshot.net> <20080604.204722.149632208.mbj@tail-f.com> <4846E97C.9080202@acalledshot.net> Message-ID: <484747C5.3080900@gmail.com> The value types in mnesia tables with snmp support must correspond to the SNMP mib spec. As such no tuples are allowed (except for the compound key column) when you use default instrumentation functions. So date(), time(), {node(), process} types must be replaced with integer() and string() types. Serge John Koenig wrote: > Martin and All, > > I have fiddled with my configuration a bit and was finally able to get > snmpwalk to browse the mnesia table under 1.3.6.1.4.1.31130 and > 1.3.6.1.4.1.31130.7.1 (defined in my POLLDATA.mib). However, an error is > thrown when snmpwalk reaches the end of the database. The first error is > the one listed in the erlang snmp-agent and the second is output by > snmpwalk. > > ---- Erlang error from publisher_snmp > =ERROR REPORT==== 4-Jun-2008::12:58:26 === > ** User error: Got {'reporter_server@REDACTED > ',reporter} from > {snmp_generic,table_func,[{pollData,mnesia}]}. Using wrongType > > > ----Error from snmpwalk > > snmpwalk -v 2c -c public localhost:4000 1.3.6.1.4.1.31130 > > POLLDATA::rowId.1 = INTEGER: 1 > POLLDATA::rowId.2 = INTEGER: 2 > . > . > . > ?POLLDATA::rowId.614 = INTEGER: 614 > POLLDATA::rowId.615 = INTEGER: 615 > POLLDATA::rowId.616 = INTEGER: 616 > Error in packet. > Reason: (genError) A general failure occured > Failed object: POLLDATA::rowId.616 > > > The data that appears in the table (all 616 rows of it) looks exactly > the same and are of the form: > > {tableName, rowID (int and key for snmp), {node, process}, stackSize, > heapSize, messageqlen, date(), time()} > > > Also, to comple POLLDATA.mib I included a POLLDATA.funcs which contains > the following: > > { pollData, { snmp_generic, table_func, [{pollData, mnesia}]}}. > > > Thoughts? > > John > > > Martin Bjorklund wrote: >> Hi, >> >> John Koenig wrote: >> >>> But when I try to call snmpget specifying the OID I outlined in the mib >>> (1.3.6.1.4.1.31130.7.1, the Object whose name matches the mnesia table) >>> and get back: >>> >>> snmpget -v 2c -c public localhost:4000 1.3.6.1.4.1.31130 >>> >> The OID on the line above ask for the enterprise OID 31130 (Basho >> Technologies). But you said you wanted to ask for >> enterprises.31130.7.1. >> >> So either try to ask directly for enterprises.31130.7.1, or even >> better, do >> >> snmpwalk -v 2c -c public localhost:4000 1.3.6.1.4.1.31130 >> >> to print all objects in you entererprise MIB. >> >> >> /martin >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From bengt.kleberg@REDACTED Thu Jun 5 07:12:05 2008 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Thu, 05 Jun 2008 07:12:05 +0200 Subject: [erlang-questions] running programs from win com prompt? In-Reply-To: <676945.84592.qm@web28306.mail.ukl.yahoo.com> References: <676945.84592.qm@web28306.mail.ukl.yahoo.com> Message-ID: <1212642725.2758.16.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> Greetings, To run a function in a compiled module you do erl -noshell -run module function -init stop Modules are (usually) written and saved from your favourite editor. I have an erlang shell running in one pane of acme (http://plan9.bell-labs.com/wiki/plan9/Using_acme/index.html) and copy working code from the shell to the file under development (in another pane). Is the file you mention as example.erl a module or a lot of nice command lines for the shell? A module should be compiled with erlc (http://www.erlang.org/doc/man/erlc.html) before loading it. Or use the shell command c(module_name) which will load it automatically. Shell commands I copy from the file into the shell. You could also look at file:consult/1 or file:eval/1 (http://www.erlang.org/doc/man/file.html). The shell will load compiled modules from the working directory or from the load path. Files are located relative to the working directory. bengt On Wed, 2008-06-04 at 20:02 +0000, Circular Function wrote: > hi > > i know how to do basic calculations and write anonymous functions. > however i dont understand from the tutorial how to write and save > modules and run the from the erlang shell or command prompt. > > if i save a file as example.erl and try to run it from the > commandprompt in windows it says > is not recognized as a command etc. > > also, where should i save a file/module to be able to load it in the > shell? > > > > > > > > ______________________________________________________________________ > L?na pengar utan s?kerhet. > S?k och j?mf?r hos Yahoo! Shopping. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From paul-trapexit@REDACTED Thu Jun 5 08:19:13 2008 From: paul-trapexit@REDACTED (Paul Mineiro) Date: Wed, 4 Jun 2008 23:19:13 -0700 (PDT) Subject: [erlang-questions] mnesia:dirty_select/3 feature request In-Reply-To: <484757F1.10901@gmail.com> References: <484757F1.10901@gmail.com> Message-ID: Serge, Via mnesia:activity/4, you can pretty much do any sequence of mnesia operations as dirty as you want. However the enclosing context can increase the transaction semantics associated with an operation, e.g., if you make a call to mnesia:async_dirty/1 within a call to mnesia:sync_transaction/1, the operation will not be dirty. In contrast, the dirty_* functions are *always* dirty, and do not inherit transaction semantics from the enclosing context. So while an mnesia:dirty_select/3 would be different in this regard than mnesia:async_dirty (fun () -> mnesia:select (...) end), maybe the latter is good enough for you? Cheers, -- p On Wed, 4 Jun 2008, Serge Aleynikov wrote: > Currently there are mnesia:dirty_select/2 and mnesia:select/{3,4} > functions, but no mnesia:dirty_select/3 with the following signature: > > mnesia:dirty_select(Table, MatchSpec, Limit) -> > {Records, Continuation} > > I believe it would be very useful to have this function for cases when a > user needs to get only a few matching records out of a very large table. > > Or perhaps am I missing some other ways to do mnesia selects using fast > read matching operations with a Limit clause that don't involve > transactions? > > Serge > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > In an artificial world, only extremists live naturally. -- Paul Graham From raimo+erlang-questions@REDACTED Thu Jun 5 09:49:40 2008 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 5 Jun 2008 09:49:40 +0200 Subject: [erlang-questions] OT: Reply-To? In-Reply-To: <4846BF1B.6070801@zorched.net> References: <004a01c8c47e$9a355910$6501a8c0@moneymaker2> <3225392D-B203-45C1-9BB2-9AB5130F3987@duomark.com> <4846BF1B.6070801@zorched.net> Message-ID: <20080605074939.GA22038@erix.ericsson.se> Search the archives. I prefer to leave it to the client. On Wed, Jun 04, 2008 at 11:13:15AM -0500, Geoff Lane wrote: > Jay Nelson wrote: > > Valentin accidentally replied directly to me: > > Maybe the list admins can set the reply-to address to go to the list? I > just did this too - and it seems to happen quite a bit... http://www.erlang.org/pipermail/erlang-questions/2006-January/018493.html http://www.erlang.org/pipermail/erlang-questions/2006-September/022814.html http://www.erlang.org/pipermail/erlang-questions/2007-June/027198.html > > -- > Geoff Lane > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From ulf.wiger@REDACTED Thu Jun 5 09:57:12 2008 From: ulf.wiger@REDACTED (Ulf Wiger (TN/EAB)) Date: Thu, 05 Jun 2008 09:57:12 +0200 Subject: [erlang-questions] mnesia:table_info(Table, size) returns 0 for remote tables In-Reply-To: <4847538E.5040405@gmail.com> References: <4847538E.5040405@gmail.com> Message-ID: <48479C58.3070101@ericsson.com> Serge Aleynikov wrote: > I ran into this non-intuitive behavior of the mnesia:table_info(Table, > size) function. Unlike other calls of the mnesia:table_info/2 family > this one returns 0 if a table is not local. > > The work around is to do: > Node = mnesia:table_info(Table, where_to_read), > rpc:call(Node, mnesia, table_info, [Table, size]). > > Though, it's not clear to me why the caller of mnesia:table_info/2 has > to go though this trouble. Can this be fixed? > > Serge > You can use the mnesia_frag callback: Eshell V5.5.4 (abort with ^G) (m1@REDACTED)1> mnesia:start(). ok (m1@REDACTED)2> mnesia:create_table(t2,[{ram_copies,[m2@REDACTED]}]). {atomic,ok} (m1@REDACTED)3> mnesia:table_info(t2,size). 0 (m1@REDACTED)4> mnesia:activity(transaction,fun() -> mnesia:table_info(t2,size) end,mnesia_frag). 1 (m1@REDACTED)5> BR, Ulf W From vychodil.hynek@REDACTED Thu Jun 5 11:44:54 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Thu, 5 Jun 2008 11:44:54 +0200 Subject: [erlang-questions] Reia - Python/Ruby mix targeting Beam/HiPE In-Reply-To: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com> References: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com> Message-ID: <4d08db370806050244rc1fb3cdve8dee9ba32efd49f@mail.gmail.com> I feel it bad. Breaking single assignment is not good idea. In mine opinion it would be messy. References (aliasing) will work inside one process but doesn't outside by message passing. Or is there plan to enable aliasing between processes? Weird. There is not any word about it. On Wed, Jun 4, 2008 at 9:42 PM, Gleb Peregud wrote: > I've stumbled upon this link few moments ago: > > http://reia-lang.org/ > > And thought it will be interesing for fellow erlangers. > > > Reia (pronounced RAY-uh) is a Python/Ruby-like mixed-paradigm language > > targeting the Erlang virtual machine (BEAM) and high-performance > > native compiler (HiPE). Reia aims to expose all the features and > functionality > > of Erlang in a language more familiar to programmers of scripting > > languages, while improving string handling, regular expressions, > > linking with external libraries, and other tasks which are generally > > considered outside the scope of Erlang. > > -- > Gleb Peregud > http://gleber.pl/ > > Every minute is to be grasped. > Time waits for nobody. > -- Inscription on a Zen Gong > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Thu Jun 5 11:52:18 2008 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 5 Jun 2008 11:52:18 +0200 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> <54568.1212602052@snookles.snookles.com> Message-ID: <9b08084c0806050252wcda24a8q17cf55cdd8b0dc8@mail.gmail.com> On Thu, Jun 5, 2008 at 12:03 AM, David Mitchell wrote: > 2008/6/5 Scott Lystig Fritchie : >> Steve Davis wrote: > >> As for "message queueing", there may be a misunderstanding over how MQ >> systems typically work: they have producers *and* consumers, and (more >> importantly) consumers actually "consume". Consuming a queue item >> usually means also deleting it from the queue. A single Twitter user X >> can have thousands of consumers all trying to consume the same messages, >> but in a typical MQ system, all but the first consumer would find X's >> queue empty. >> >> For one example, see the RabbitMQ FAQ, "Q. How do I archive to RDBMS?". > > In case anyone's losing track, I was the one who suggested keeping > tweets in queues essentially forever, and having users retrieve them > from queues without deleting the message from the que. > > I understand how MQ works in normal environments; what I'm suggesting > is that Twitter (and any clones) aren't "normal" once they start to > scale up to many millions of users. > > The reasons I suggested storing messages in queues indefinitely are: > - experience says that queueing systems can scale very large, and that > it appears to be an "easier" problem to solve than scaling a database > very large. I'll accept it if anyone complains about "gross > generalisation"... > - the APIs for storing messages to queues and then retrieving them are > designed to be very fast, and (again referencing IBM's MQ) we know > they can scale to queues holding very large numbers of messages > Yes yes yes - I have for a long time thought that non-destructive persistent queues are the perfect data structure for many applications. I can't see why REST has GET, PUT, POST and DELETE - It should have GET and APPEND (only). Appending things to a input queue *and never deleting them* seems to me a perfect way to deal with things. If you every want to delete things it can only be for pragmatic reasons and should be done years later in a garbage collection sweep. If you never delete things you can always go back later an fix errors! The question of how to store the queue is unrelated to the abstraction - probably disk replicated with a ram replica of the last N entries in the tail of the queue. If the queue entries are fixed length (or multiples of a fixed length) then life becomes a lot easier Many things can be build using this abstraction. Add fault-tolerance and location transparency and you have a wonderfully useful mechanism. (ie it would be very nice to have GUID's that identify persistent queues - how to do this is orthogonal to the storage mechanisms involved). To start with a queues identified by {Ip,Port,name} would be useful :-) Cheers /Joe Armstrong > Storing messages in flat files seems to have a couple of limitations to me: > - if you're going to store 1 message per flat file, you need a > database (or database-like thing) to track those zillions of flat > files. I figure that's going to put you back where you started in > terms of scalability > - assuming you're always appending messages to the end of flat files, > you'd have to assume that most requests will be for the most recent > message i.e. the last message in the file. Do you really want to be > seeking through to the last record of flat files all the time? That > doesn't seem to be a scalable approach > - alternately, if you always add the most recent message to the > *start* of a flat file, you'll constantly be rewriting the entire file > (at least, that's the case in any file system I can think of; there > might be an exception). I suppose you could write your own file > system to optimise that... > > Please speak up if you've got any thoughts - I'm treating this like a > bunch of intellectuals throwing ideas around, rather than an argument > about right and wrong, and it seems that everyone else is too at this > stage. Very happy to be convinced I'm wrong, in other words > > Regards > > David Mitchell > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From dmitriid@REDACTED Thu Jun 5 12:25:15 2008 From: dmitriid@REDACTED (Dmitrii Dimandt) Date: Thu, 5 Jun 2008 13:25:15 +0300 Subject: [erlang-questions] Reia - Python/Ruby mix targeting Beam/HiPE In-Reply-To: <4d08db370806050244rc1fb3cdve8dee9ba32efd49f@mail.gmail.com> References: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com> <4d08db370806050244rc1fb3cdve8dee9ba32efd49f@mail.gmail.com> Message-ID: On Jun 5, 2008, at 12:44 PM, Hynek Vychodil wrote: > I feel it bad. Breaking single assignment is not good idea. In mine > opinion it would be messy. References (aliasing) will work inside > one process but doesn't outside by message passing. Or is there plan > to enable aliasing between processes? Weird. There is not any word > about it. > If my memory serves me right, Nemerle has a similar approach where mutable variables must be explicitly declared as such From gleber.p@REDACTED Thu Jun 5 13:12:54 2008 From: gleber.p@REDACTED (Gleb Peregud) Date: Thu, 5 Jun 2008 13:12:54 +0200 Subject: [erlang-questions] Reia - Python/Ruby mix targeting Beam/HiPE In-Reply-To: <4d08db370806050244rc1fb3cdve8dee9ba32efd49f@mail.gmail.com> References: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com> <4d08db370806050244rc1fb3cdve8dee9ba32efd49f@mail.gmail.com> Message-ID: <14f0e3620806050412k4bc8a145nf4d4e161a6fd189@mail.gmail.com> 2008/6/5 Hynek Vychodil : > I feel it bad. Breaking single assignment is not good idea. In mine opinion > it would be messy. References (aliasing) will work inside one process but > doesn't outside by message passing. Or is there plan to enable aliasing > between processes? Weird. There is not any word about it. > > On Wed, Jun 4, 2008 at 9:42 PM, Gleb Peregud wrote: >> >> I've stumbled upon this link few moments ago: >> >> http://reia-lang.org/ >> >> And thought it will be interesing for fellow erlangers. >> >> > Reia (pronounced RAY-uh) is a Python/Ruby-like mixed-paradigm language >> > targeting the Erlang virtual machine (BEAM) and high-performance >> > native compiler (HiPE). Reia aims to expose all the features and >> > functionality >> > of Erlang in a language more familiar to programmers of scripting >> > languages, while improving string handling, regular expressions, >> > linking with external libraries, and other tasks which are generally >> > considered outside the scope of Erlang. >> >> -- >> Gleb Peregud >> http://gleber.pl/ >> >> Every minute is to be grasped. >> Time waits for nobody. >> -- Inscription on a Zen Gong According to http://wiki.reia-lang.org/wiki/Roadmap : The First Milestone: Compile to Erlang AST And I assume that Erlang AST does not permit mutable variables, hence it will probably not include mutable variables (unless hidden by Reia -> Erlang AST translator) I'm not sure, but doesn't compiling to Core Erlang is better way of implementing new language for Beam/HiPE VM? From saleyn@REDACTED Thu Jun 5 13:22:57 2008 From: saleyn@REDACTED (Serge Aleynikov) Date: Thu, 05 Jun 2008 07:22:57 -0400 Subject: [erlang-questions] mnesia:dirty_select/3 feature request In-Reply-To: References: <484757F1.10901@gmail.com> Message-ID: <4847CC91.9060300@gmail.com> Paul, Thanks for the tip. In my search through mnesia documentation for "dirty_*" functions I indeed missed the "*_dirty" pattern! Though having mnesia:dirty_select/3 would be quite logical as it would be symmetric to it's transaction-based mnesia:select/3 counterpart. Serge Paul Mineiro wrote: > Serge, > > Via mnesia:activity/4, you can pretty much do any sequence of mnesia > operations as dirty as you want. However the enclosing context can > increase the transaction semantics associated with an operation, e.g., if > you make a call to mnesia:async_dirty/1 within a call to > mnesia:sync_transaction/1, the operation will not be dirty. > > In contrast, the dirty_* functions are *always* dirty, and do not inherit > transaction semantics from the enclosing context. So while an > mnesia:dirty_select/3 would be different in this regard than > mnesia:async_dirty (fun () -> mnesia:select (...) end), maybe the latter > is good enough for you? > > Cheers, > > -- p > > On Wed, 4 Jun 2008, Serge Aleynikov wrote: > >> Currently there are mnesia:dirty_select/2 and mnesia:select/{3,4} >> functions, but no mnesia:dirty_select/3 with the following signature: >> >> mnesia:dirty_select(Table, MatchSpec, Limit) -> >> {Records, Continuation} >> >> I believe it would be very useful to have this function for cases when a >> user needs to get only a few matching records out of a very large table. >> >> Or perhaps am I missing some other ways to do mnesia selects using fast >> read matching operations with a Limit clause that don't involve >> transactions? >> >> Serge >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > In an artificial world, only extremists live naturally. > > -- Paul Graham > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From saleyn@REDACTED Thu Jun 5 13:35:58 2008 From: saleyn@REDACTED (Serge Aleynikov) Date: Thu, 05 Jun 2008 07:35:58 -0400 Subject: [erlang-questions] mnesia:table_info(Table, size) returns 0 for remote tables In-Reply-To: <48479C58.3070101@ericsson.com> References: <4847538E.5040405@gmail.com> <48479C58.3070101@ericsson.com> Message-ID: <4847CF9E.5000404@gmail.com> Is this any better than doing: Node = mnesia:table_info(Table, where_to_read), rpc:call(Node, mnesia, table_info, [Table, size]). I think that mnesia:activity(transaction,fun() -> mnesia:table_info(t2,size) end,mnesia_frag). boils down to the rpc call anyway. Though it looks like a simple call to mnesia:table_info(Table, size) that would abstract implementation details of where the table is would be cleaner. Serge Ulf Wiger (TN/EAB) wrote: > Serge Aleynikov wrote: >> I ran into this non-intuitive behavior of the mnesia:table_info(Table, >> size) function. Unlike other calls of the mnesia:table_info/2 family >> this one returns 0 if a table is not local. >> >> The work around is to do: >> Node = mnesia:table_info(Table, where_to_read), >> rpc:call(Node, mnesia, table_info, [Table, size]). >> >> Though, it's not clear to me why the caller of mnesia:table_info/2 has >> to go though this trouble. Can this be fixed? >> >> Serge >> > > You can use the mnesia_frag callback: > > Eshell V5.5.4 (abort with ^G) > (m1@REDACTED)1> mnesia:start(). > ok > (m1@REDACTED)2> mnesia:create_table(t2,[{ram_copies,[m2@REDACTED]}]). > {atomic,ok} > (m1@REDACTED)3> mnesia:table_info(t2,size). > 0 > (m1@REDACTED)4> mnesia:activity(transaction,fun() -> > mnesia:table_info(t2,size) end,mnesia_frag). > 1 > (m1@REDACTED)5> > > > BR, > Ulf W > From vychodil.hynek@REDACTED Thu Jun 5 14:14:58 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Thu, 5 Jun 2008 14:14:58 +0200 Subject: [erlang-questions] Reia - Python/Ruby mix targeting Beam/HiPE In-Reply-To: <14f0e3620806050412k4bc8a145nf4d4e161a6fd189@mail.gmail.com> References: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com> <4d08db370806050244rc1fb3cdve8dee9ba32efd49f@mail.gmail.com> <14f0e3620806050412k4bc8a145nf4d4e161a6fd189@mail.gmail.com> Message-ID: <4d08db370806050514l2348959fm18b361a7d7f23744@mail.gmail.com> On Thu, Jun 5, 2008 at 1:12 PM, Gleb Peregud wrote: > 2008/6/5 Hynek Vychodil : > > I feel it bad. Breaking single assignment is not good idea. In mine > opinion > > it would be messy. References (aliasing) will work inside one process but > > doesn't outside by message passing. Or is there plan to enable aliasing > > between processes? Weird. There is not any word about it. > > > > On Wed, Jun 4, 2008 at 9:42 PM, Gleb Peregud wrote: > >> > >> I've stumbled upon this link few moments ago: > >> > >> http://reia-lang.org/ > >> > >> And thought it will be interesing for fellow erlangers. > >> > >> > Reia (pronounced RAY-uh) is a Python/Ruby-like mixed-paradigm language > >> > targeting the Erlang virtual machine (BEAM) and high-performance > >> > native compiler (HiPE). Reia aims to expose all the features and > >> > functionality > >> > of Erlang in a language more familiar to programmers of scripting > >> > languages, while improving string handling, regular expressions, > >> > linking with external libraries, and other tasks which are generally > >> > considered outside the scope of Erlang. > >> > >> -- > >> Gleb Peregud > >> http://gleber.pl/ > >> > >> Every minute is to be grasped. > >> Time waits for nobody. > >> -- Inscription on a Zen Gong > > According to http://wiki.reia-lang.org/wiki/Roadmap : > The First Milestone: Compile to Erlang AST > And I assume that Erlang AST does not permit mutable variables, hence > it will probably not include mutable variables (unless hidden by Reia > -> Erlang AST translator) > Yes, mutable variables can be easily introduced by some AST translation, but big problems come from mutable structures. It is much more harder implement this by erlang and I think things comes much more simple if there will not be mutable variables nor mutable structures. Mutable variables without mutable structures will be messy in mine opinion. Mutable variables without aliasing will be as much confusing as immutable functional world for Reia's target developers. IMHO learn immutable concept is easier than understand why something can be mutable and something not. Without mutable structures solution Reia looks for me not well designed from start idea. > I'm not sure, but doesn't compiling to Core Erlang is better way of > implementing new language for Beam/HiPE VM? > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Thu Jun 5 14:23:40 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 05 Jun 2008 14:23:40 +0200 Subject: [erlang-questions] Reia - Python/Ruby mix targeting Beam/HiPE In-Reply-To: <14f0e3620806050412k4bc8a145nf4d4e161a6fd189@mail.gmail.com> References: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com> <4d08db370806050244rc1fb3cdve8dee9ba32efd49f@mail.gmail.com> <14f0e3620806050412k4bc8a145nf4d4e161a6fd189@mail.gmail.com> Message-ID: "Gleb Peregud" writes: > According to http://wiki.reia-lang.org/wiki/Roadmap : > The First Milestone: Compile to Erlang AST > And I assume that Erlang AST does not permit mutable variables, hence > it will probably not include mutable variables (unless hidden by Reia > -> Erlang AST translator) It should be trivial to implement mutable variables, simply by inventing a new, unused variable name each time a mutable variable is assigned. (Further references to the mutable variable should be changed to refer to the new name, of course.) /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From mikpe@REDACTED Thu Jun 5 15:19:06 2008 From: mikpe@REDACTED (Mikael Pettersson) Date: Thu, 5 Jun 2008 15:19:06 +0200 Subject: [erlang-questions] Reia - Python/Ruby mix targeting Beam/HiPE In-Reply-To: References: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com> <4d08db370806050244rc1fb3cdve8dee9ba32efd49f@mail.gmail.com> <14f0e3620806050412k4bc8a145nf4d4e161a6fd189@mail.gmail.com> Message-ID: <18503.59338.589318.969300@harpo.it.uu.se> Bjorn Gustavsson writes: > "Gleb Peregud" writes: > > > According to http://wiki.reia-lang.org/wiki/Roadmap : > > The First Milestone: Compile to Erlang AST > > And I assume that Erlang AST does not permit mutable variables, hence > > it will probably not include mutable variables (unless hidden by Reia > > -> Erlang AST translator) > > It should be trivial to implement mutable variables, simply by inventing > a new, unused variable name each time a mutable variable is assigned. > (Further references to the mutable variable should be changed to refer > to the new name, of course.) Define "mutable variable". I know of (at least) two different kinds: 1. Each variable binding starts a new scope, nested within but otherwise independent of the previous/surrounding scope. This is what SML's LET and Scheme's LET* implement. It's still purely functional and can be implemented via a simple renaming operation (alpha conversion). 2. A variable is bound to an updateable memory cell, and assignment updates that cell, like in C/Pascal/etc. If these cells can escape (e.g. via closures), then simple renaming doesn't suffice, and you're really forced to have a mutable store. Stores can be emulated in the Erlang VM, but it won't be cheap. From vladdu55@REDACTED Thu Jun 5 15:24:29 2008 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Thu, 5 Jun 2008 15:24:29 +0200 Subject: [erlang-questions] Reia - Python/Ruby mix targeting Beam/HiPE In-Reply-To: References: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com> <4d08db370806050244rc1fb3cdve8dee9ba32efd49f@mail.gmail.com> <14f0e3620806050412k4bc8a145nf4d4e161a6fd189@mail.gmail.com> Message-ID: <95be1d3b0806050624y2ff391e8t6c26d03ea9326b2c@mail.gmail.com> On Thu, Jun 5, 2008 at 2:23 PM, Bjorn Gustavsson wrote: > "Gleb Peregud" writes: > > > According to http://wiki.reia-lang.org/wiki/Roadmap : > > The First Milestone: Compile to Erlang AST > > And I assume that Erlang AST does not permit mutable variables, hence > > it will probably not include mutable variables (unless hidden by Reia > > -> Erlang AST translator) > > It should be trivial to implement mutable variables, simply by inventing > a new, unused variable name each time a mutable variable is assigned. > (Further references to the mutable variable should be changed to refer > to the new name, of course.) > > The big problem with this matter is that Erlang doesn't have variable assignment, it has pattern matching. So if we encounter a reference to an old variable in a pattern, should the old variable be used to check if it matches, or should a new fresh variable be created and matched against the right side? This could be solved for example by using special characters to denote fresh matching, but IMO the resulting mess is worse than the issue that it was created to solve. regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Thu Jun 5 15:33:25 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 05 Jun 2008 15:33:25 +0200 Subject: [erlang-questions] Reia - Python/Ruby mix targeting Beam/HiPE In-Reply-To: <18503.59338.589318.969300@harpo.it.uu.se> References: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com> <4d08db370806050244rc1fb3cdve8dee9ba32efd49f@mail.gmail.com> <14f0e3620806050412k4bc8a145nf4d4e161a6fd189@mail.gmail.com> <18503.59338.589318.969300@harpo.it.uu.se> Message-ID: Mikael Pettersson writes: > Bjorn Gustavsson writes: > > It should be trivial to implement mutable variables, simply by inventing > > a new, unused variable name each time a mutable variable is assigned. > > (Further references to the mutable variable should be changed to refer > > to the new name, of course.) > > Define "mutable variable". I know of (at least) two different kinds: > > 1. Each variable binding starts a new scope, nested within but > otherwise independent of the previous/surrounding scope. > This is what SML's LET and Scheme's LET* implement. > It's still purely functional and can be implemented via > a simple renaming operation (alpha conversion). That was the kind of mutable variable I was thinking of, and it seems to be the kind that REIA supports. /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From vsi@REDACTED Thu Jun 5 11:53:06 2008 From: vsi@REDACTED (Ville Silventoinen) Date: Thu, 5 Jun 2008 10:53:06 +0100 (BST) Subject: [erlang-questions] setuid Message-ID: Hi, I'm new to the world of Erlang and I'm just trying some things out (very impressive language!). I got interested in Erlang after seeing Joe Armstrong's presentation in this year's ACCU conference in Oxford and reading his excellent book. I'm writing a prototype of a network filesystem using the fuserl package. So far so good, I have a little network filesystem that uses fuserl at the client-side and communicates with a gateway written with gen_server, which does the file system operations. One of the things I'd like to do is to map the client-side Unix UID and GID numbers (#fuse_ctx.uid and #fuse_ctx.gid) to different UID and GID numbers at the gateway. I wrote a little linked in C module that allows me to call setuid, seteuid, etc. from Erlang, but I'm not sure how this will affect other processes doing file operations in the Erlang runtime environment. Would setuid change the UID for all Erlang processes or for just the process that called it? I'll find out the answer soon by experimenting, but I thought I should ask if anyone has better ideas! Thanks, Ville From gleber.p@REDACTED Thu Jun 5 16:33:35 2008 From: gleber.p@REDACTED (Gleb Peregud) Date: Thu, 5 Jun 2008 16:33:35 +0200 Subject: [erlang-questions] setuid In-Reply-To: References: Message-ID: <14f0e3620806050733s2d7fe727tde8c6c13902ceb66@mail.gmail.com> Processes in Erlang does not map directly to kernel threads. Erlang processes are like green threads in Java. Operating system is not aware of them. They are managed by Erlang VM entierly - they are just, simplifying, internal data structures inside Erlang VM. Erlang processes are all run inside Erlang VM, which is run on one (or more if it is SMP system) kernel thread. Linked-in drivers are dynamically linked in into Erlang VM [1], hence setuid, seteuid, setguid, etc. will affect whole Erlang VM (probably only one of the Erlang VM processes (kernel threads) in case of SMP system, but i'm not sure). Hence these functions called in linked-in drivers will affect every Erlang processes inside VM. I hope this text is not messed too much :) [1] http://www.erlang.org/doc/tutorial/c_portdriver.html On Thu, Jun 5, 2008 at 11:53 AM, Ville Silventoinen wrote: > Hi, > > I'm new to the world of Erlang and I'm just trying some things out (very > impressive language!). I got interested in Erlang after seeing Joe > Armstrong's presentation in this year's ACCU conference in Oxford and > reading his excellent book. > > I'm writing a prototype of a network filesystem using the fuserl package. > So far so good, I have a little network filesystem that uses fuserl at the > client-side and communicates with a gateway written with gen_server, which > does the file system operations. > > One of the things I'd like to do is to map the client-side Unix UID and > GID numbers (#fuse_ctx.uid and #fuse_ctx.gid) to different UID and GID > numbers at the gateway. I wrote a little linked in C module that allows me > to call setuid, seteuid, etc. from Erlang, but I'm not sure how this will > affect other processes doing file operations in the Erlang runtime > environment. Would setuid change the UID for all Erlang processes or for > just the process that called it? > > I'll find out the answer soon by experimenting, but I thought I should ask > if anyone has better ideas! > > Thanks, > Ville > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- Gleb Peregud http://gleber.pl/ Every minute is to be grasped. Time waits for nobody. -- Inscription on a Zen Gong From mde@REDACTED Thu Jun 5 18:01:01 2008 From: mde@REDACTED (Micah Elliott) Date: Thu, 05 Jun 2008 09:01:01 -0700 Subject: [erlang-questions] Command line option parser (getopt?) Message-ID: <20080605160101.GA10700@micahelliott.com> Hi. I'm brand new to Erlang, and really enjoying it so far. I'm trying to get a feel for the language by writing a toy CLI tool. I haven't yet been able to locate a getopt-like module. Does one exist? I expect it must considering, e.g., perl has dozens in CPAN, and Python has a least a couple nice ones (optparse, argparse). Thanks for any pointers. -- Micah Elliott | mde@REDACTED | http://MicahElliott.blogspot.com From patrickdlogan@REDACTED Thu Jun 5 19:23:40 2008 From: patrickdlogan@REDACTED (Patrick Logan) Date: Thu, 5 Jun 2008 10:23:40 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone Message-ID: From: Steve Davis >> No database is needed. Everything can pretty much be achieved with >> files containing fixed-length records. > > What you and Bob have said is, of course, true. I suspect that as > the system becomes large scale then the software required to access > and manage those files would eventually end up looking a lot like > one of... an RDBMS, a Message Queue, an IM server, etc. Hmm. Sounds kind of like a "record" has most of the properties of an Atom format "Entry" and a series of these are like a "Feed". Whereever these are stored transiently or archived semi-permanently, these are small, time-ordered, series of information. Not surprising. From: Steve Davis > The level to which I disagree with your conclusion here is quite > profound! I see a large-scale future of lost messages, data > corruption, file replication and synchronization issues, and > *downtime* for the FS approach. > > All in all, the fundamental application scope of Twitter simply > *screams* Message Queue at me. I'm not sure why the "experts" that > Twitter have scavenged from IBM and Google haven't come to that > conclusion also. Since Twitter have appeared to commit to a FS > approach, I guess we'll have to see if future history proves me > incorrect :) This decision could probably be broken down into some general "needs" (e.g. Twoorl needs time-ordering, categorization by author, persistence of an entry for some length of time, etc.). Then various solution alternatives could be mapped onto some or all of those needs. I think it's too much too soon to say either "use a FS approach" or "use a queue approach". Plus you'd certainly have to get to the level of which kind of file system or queue for which needs? e.g. I wrote before about XMPP as a suitable queue for several of the needs, and for file systems, ZFS may have some nice attributes for some of these needs. From: Damien Morton > Well, the relational database is suitable for the quickly evolving > part of their business, and there will always be a quickly evolving > part of any business, so yes - inevitability is the perfect > description. Ironic since I hardly ever see the phrases "relational database" and "quickly evolving" used together except in an inverse relationship. 8^D > Question is - can they create such a specialised datastructure such > that it _can_ evolve, for example, in the case where they wanted to > handle not just 140 byte tweets, but also photos and video (twotos, > and twideos)? I would probably consider something like an Amazon SimpleDB service as a potential solution alternative. -Patrick From dmorton@REDACTED Thu Jun 5 19:33:46 2008 From: dmorton@REDACTED (Damien Morton) Date: Fri, 06 Jun 2008 03:33:46 +1000 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: Message-ID: <4848237A.7090103@bitfurnace.com> On 6/6/2008 3:23 AM, Patrick Logan wrote: > From: Damien Morton > >> Well, the relational database is suitable for the quickly evolving >> part of their business, and there will always be a quickly evolving >> part of any business, so yes - inevitability is the perfect >> description. >> > > Ironic since I hardly ever see the phrases "relational database" and > "quickly evolving" used together except in an inverse > relationship. 8^D > That's true - relational databases have ossified. The original Date book on SQL databases, IIRC, seemed to cast them in the light of being a tool for executives to quickly and arbitrarily query the company's information repository. I cant quite remember what the orginal IBM SQL database was called, but Query-By-Example seemed to be an essential element. Still, can you name a database that better at evolving than a relational database? From chsu79@REDACTED Thu Jun 5 21:18:13 2008 From: chsu79@REDACTED (Christian S) Date: Thu, 5 Jun 2008 21:18:13 +0200 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <4848237A.7090103@bitfurnace.com> References: <4848237A.7090103@bitfurnace.com> Message-ID: > That's true - relational databases have ossified. The original Date book > on SQL databases, IIRC, seemed to cast them in the light of being a tool > for executives to quickly and arbitrarily query the company's > information repository. I cant quite remember what the orginal IBM SQL > database was called, but Query-By-Example seemed to be an essential element. My experience with rdbms told me that it is a really bad idea to write queries that depend on table names and layouts. That is what makes it so hard to evolve a db. All the applications using it depend on the current schema. So stored procedures are a really good idea. I assume there is a similar lesson to mnesia, put all your transaction funs for a given mnesia-using application in a single module. From tpot@REDACTED Thu Jun 5 23:18:26 2008 From: tpot@REDACTED (Tim Potter) Date: Fri, 06 Jun 2008 07:18:26 +1000 Subject: [erlang-questions] Parsing a DTD file with xmerl Message-ID: <1212700706.9015.2.camel@tigerella> [Resending from correct account] Hi everyone. Just a quick question about xmerl - is it possible to parse a DTD file and receive back something useful? For example, running xmerl_scan:file("CIM_DTD_V22.dtd") gives back the following error: 1698- fatal: {error,{wfc_PEs_In_Internal_Subset}} ** exited: {fatal,{{error,{wfc_PEs_In_Internal_Subset}}, {file,"CIM_DTD_V22.dtd"}, {line,102}, {col,10}}} ** Tracing down through the error a bit it appears that xmerl is complaining about a parameter-entity reference appearing within a markup declaration which I don't think is the case here. This DTD file is parsed by other XML parsers, for example Python. Tim. From ok@REDACTED Fri Jun 6 00:55:46 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 6 Jun 2008 10:55:46 +1200 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <9b08084c0806050252wcda24a8q17cf55cdd8b0dc8@mail.gmail.com> References: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> <54568.1212602052@snookles.snookles.com> <9b08084c0806050252wcda24a8q17cf55cdd8b0dc8@mail.gmail.com> Message-ID: <2CB2D37B-386B-4078-A83E-D391716A31B2@cs.otago.ac.nz> On 5 Jun 2008, at 9:52 pm, Joe Armstrong wrote: > Appending things to a input queue *and never deleting them* seems to > me a perfect way to deal with things. > If you every want to delete things it can only be for pragmatic > reasons and should be done years later > in a garbage collection sweep. If you never delete things you can > always go back later an fix errors! I note that this is one of the reasons why SAP is successful. Businesses use SAP because they *can* use SAP; their national tax departments are happy with SAP. Their national tax departments are happy with SAP because they can *audit* business data held in SAP. And they can do *that* because (ta-dah!) SAP never deletes anything! If there is an incorrect data entry, for example, you can mark it as incorrect and replace it with a corrected record, but you cannot remove the old record from the system. I note that you can now buy a Western Digital WD10EACS 1 Terabyte 7200 RPM disc drive for USD 253 (which is currently NZD 330), price from pixelUSA.com. Deleting stuff makes a lot less sense than it used to. From bkelly.ie@REDACTED Fri Jun 6 04:52:14 2008 From: bkelly.ie@REDACTED (Barry Kelly) Date: Fri, 06 Jun 2008 03:52:14 +0100 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <9b08084c0806050252wcda24a8q17cf55cdd8b0dc8@mail.gmail.com> References: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> <54568.1212602052@snookles.snookles.com> <9b08084c0806050252wcda24a8q17cf55cdd8b0dc8@mail.gmail.com> Message-ID: <6i9h44lc45nu67im1huupbofetkbkova6g@4ax.com> Joe Armstrong wrote: > Yes yes yes - I have for a long time thought that non-destructive > persistent queues are the perfect data structure for > many applications. I can't see why REST has GET, PUT, POST and DELETE > - It should have GET and APPEND > (only). > Appending things to a input queue *and never deleting them* seems to > me a perfect way to deal with things. There are privacy and security ramifications to such a design. A service provider implemented using such a model may hold passwords and customer data much longer than it needs to. Furthermore, the semantic operations of PUT and DELETE still need implementing - customers still want to logically upsert and delete resources, so a second-level API convention or standard is still required. Why not simply implement PUT, POST and DELETE as enqueued operations on the server side? -- Barry -- http://barrkel.blogspot.com/ From Raymond.Xiong@REDACTED Fri Jun 6 05:53:58 2008 From: Raymond.Xiong@REDACTED (Raymond Xiong) Date: Fri, 06 Jun 2008 11:53:58 +0800 Subject: [erlang-questions] "erlexec: HOME must be set" error - a build question Message-ID: <20080606035357.GA12789@Sun.Com> I am integrating Erlang into a build environment, where people ususally unset all env varialbes first and then build, for example: $ env - build.ksh The purpose is to put all needed env variables into build.ksh script so that people with different settings can get the same build result. That unfortunately caused "erlexec: HOME must be set" error when I built Erlang, because it assumed HOME env variable should be set. My question is how the HOME directory is used? Does Erlang saves any config files under it? Is it OK to set it to, say, /tmp directory in this case(If so, then I can it define it in build.ksh script)? Thanks, Raymond From Raymond.Xiong@REDACTED Fri Jun 6 08:01:29 2008 From: Raymond.Xiong@REDACTED (Raymond Xiong) Date: Fri, 06 Jun 2008 14:01:29 +0800 Subject: [erlang-questions] "erlexec: HOME must be set" error - a build question In-Reply-To: <449D56A8-FA63-46D2-A165-3CD2A9E9A1AF@cs.otago.ac.nz> References: <20080606035357.GA12789@Sun.Com> <449D56A8-FA63-46D2-A165-3CD2A9E9A1AF@cs.otago.ac.nz> Message-ID: <20080606060128.GA12902@Sun.Com> (I guess Richard forgot to copy to the mailig list, so I reply to all) On 06/06/08, Richard A. O'Keefe wrote: > It's always possible to write a little C program that > (1) Calls getuid() to find who is running it. > (2) Calls getpwuid() to find the /etc/passwd record for that user. > (3) Calls puts() to write the pw_dir field of that record. > > #include > #include > #include > > int main(void) { > struct passwd *pw = getpwuid(getuid()); > puts(pw == 0 ? "/tmp" : pw->pw_dir); > return 0; > } > > Your build.ksh script can then do HOME=`homedir` and you are done. > > Almost. There is one flaw in this, which means that HOME cannot > in general be precisely reconstructed. UNIX allows more than one > logname to map to the same uid, and these lognames might have > different home directories. (I've known this done.) In that > case you will recover >A< home directory that might not be >THE< > $HOME you started with. Thanks for your suggestion, but I am afraid that seems a bit complex for a build script(though I didn't realize it can be solved in that way). I am still curious how HOME directory is used by Erlang. > My understanding is that "env -" should be "env -i" these days. Didn't notice this before. Thanks for the tip. Raymond > The C code above raises the question, if the Erlang build > needs this, why doesn't it do that itself? > > From Raymond.Xiong@REDACTED Fri Jun 6 09:30:50 2008 From: Raymond.Xiong@REDACTED (Raymond Xiong) Date: Fri, 06 Jun 2008 15:30:50 +0800 Subject: [erlang-questions] "erlexec: HOME must be set" error - a build question In-Reply-To: <20080606035357.GA12789@Sun.Com> References: <20080606035357.GA12789@Sun.Com> Message-ID: <20080606073050.GA13134@Sun.Com> On 06/06/08, Raymond Xiong wrote: > I am integrating Erlang into a build environment, where people > ususally unset all env varialbes first and then build, for example: > > $ env - build.ksh > > The purpose is to put all needed env variables into build.ksh script > so that people with different settings can get the same build result. > > That unfortunately caused "erlexec: HOME must be set" error when > I built Erlang, because it assumed HOME env variable should be set. In case I didn't make it clear, that wasn't a build script error. Instead it was erlexec(the VM launcher, if I get it right) error. I wonder what stuff under HOME directory are accessed by Erlang runtime, and can I ignore it? Thanks, Raymond > My question is how the HOME directory is used? Does Erlang saves > any config files under it? Is it OK to set it to, say, /tmp directory > in this case(If so, then I can it define it in build.ksh script)? > > Thanks, > Raymond > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From mazen@REDACTED Fri Jun 6 11:59:28 2008 From: mazen@REDACTED (Mazen Harake) Date: Fri, 06 Jun 2008 10:59:28 +0100 Subject: [erlang-questions] Command line option parser (getopt?) In-Reply-To: <20080605160101.GA10700@micahelliott.com> References: <20080605160101.GA10700@micahelliott.com> Message-ID: <48490A80.4020001@erlang-consulting.com> Hiya (And welcome? :) Try init:get_arguments/0 or init:get_argument/1 http://www.erlang.org/doc/man/init.html Also, incase you will want to read input you will eventually need: io:read/1 http://www.erlang.org/doc/man/io.html Good luck, Mazen Micah Elliott wrote: > Hi. I'm brand new to Erlang, and really enjoying it so far. I'm > trying to get a feel for the language by writing a toy CLI tool. > I haven't yet been able to locate a getopt-like module. Does one > exist? I expect it must considering, e.g., perl has dozens in > CPAN, and Python has a least a couple nice ones (optparse, > argparse). Thanks for any pointers. > > -- Mazen Harake Erlang Software Developer and Consultant, Erlang Training & Consulting, Ltd Mobile Phone: +44 (0)795 13 26 317 Office Phone: +44 (0)207 45 61 020 Office Address: 401 London Fruit & Wool Exchange Brushfield St, London, E1 6EL United Kingdom From chlorophil@REDACTED Fri Jun 6 14:13:14 2008 From: chlorophil@REDACTED (Philip Robinson) Date: Fri, 6 Jun 2008 22:13:14 +1000 Subject: [erlang-questions] Reia - Python/Ruby mix targeting Beam/HiPE In-Reply-To: References: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com> <4d08db370806050244rc1fb3cdve8dee9ba32efd49f@mail.gmail.com> <14f0e3620806050412k4bc8a145nf4d4e161a6fd189@mail.gmail.com> <95be1d3b0806050624y2ff391e8t6c26d03ea9326b2c@mail.gmail.com> Message-ID: 2008/6/5 Vlad Dumitrescu : > The big problem with this matter is that Erlang doesn't have variable > assignment, it has pattern matching. So if we encounter a reference to an > old variable in a pattern, should the old variable be used to check if it > matches, or should a new fresh variable be created and matched against the > right side? This could be solved for example by using special characters to > denote fresh matching, but IMO the resulting mess is worse than the issue > that it was created to solve. That's exactly what Reia pattern-matching does, with an asterisk in front of the variable name to denote pattern matching instead of binding a new value. An example from the Reia pattern-matching documentation (http://wiki.reia-lang.org/wiki/Pattern_matching): >> foo = 42 42 >> (*foo, [bar, baz]) = (1, [2, 3]) NoMatch: right hand side value 1 I'm not yet sure how to handle multiple occurrences of a value, though. Perhaps this is how it would be done: >> (blub, *blub) = (42, 42) I have only browsed the Reia site and haven't actually downloaded and tried it - to be honest I'm more interested in LFE right now. :-) Cheers, Philip From erlang@REDACTED Fri Jun 6 14:48:16 2008 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 6 Jun 2008 14:48:16 +0200 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <6i9h44lc45nu67im1huupbofetkbkova6g@4ax.com> References: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> <54568.1212602052@snookles.snookles.com> <9b08084c0806050252wcda24a8q17cf55cdd8b0dc8@mail.gmail.com> <6i9h44lc45nu67im1huupbofetkbkova6g@4ax.com> Message-ID: <9b08084c0806060548x701d07b2w7380e04a2ad50946@mail.gmail.com> On Fri, Jun 6, 2008 at 4:52 AM, Barry Kelly wrote: > Joe Armstrong wrote: > >> Yes yes yes - I have for a long time thought that non-destructive >> persistent queues are the perfect data structure for >> many applications. I can't see why REST has GET, PUT, POST and DELETE >> - It should have GET and APPEND >> (only). > >> Appending things to a input queue *and never deleting them* seems to >> me a perfect way to deal with things. > > There are privacy and security ramifications to such a design. A service > provider implemented using such a model may hold passwords and customer > data much longer than it needs to. > Security has nothing to do with this argument - if a password is sent over the network it has been sent - nothing can alter that. A man in the middle might store the message forever so it would make no difference if the server stores the data for a millisecond or a trillion years. The reason for storing things in an appended log is to be able to replay the log later if things go wrong and recover from errors - it has nothing to do with security. Security and privacy has to do with the level of encryption that is applied to the items in the log. /Joe > Furthermore, the semantic operations of PUT and DELETE still need > implementing - customers still want to logically upsert and delete > resources, so a second-level API convention or standard is still > required. > > Why not simply implement PUT, POST and DELETE as enqueued operations on > the server side? > > -- Barry > > -- > http://barrkel.blogspot.com/ > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From francesco@REDACTED Fri Jun 6 15:25:20 2008 From: francesco@REDACTED (Francesco Cesarini) Date: Fri, 06 Jun 2008 14:25:20 +0100 Subject: [erlang-questions] Erlang eXchange Updated Speaker List June 26th/27th Message-ID: <48493AC0.5090100@erlang-consulting.com> Hi All, A note to say that with less than two weeks to go, we now have over 30 speakers lined up, with up to three parallel tracks and plenty of tutorials? Not to mention the three days of University kicking off all of the activities. You can see the latest program on the exchange site: http://www.erlang-exchange.com/conference and http://www.erlang-exchange.com/speakers We are constantly updating the program as the final abstracts are coming in, so keep an eye on the site in the next few weeks. Contact me if you have any questions. Francesco -- http://www.erlang-consulting.com From dmorton@REDACTED Fri Jun 6 15:37:18 2008 From: dmorton@REDACTED (Damien Morton) Date: Fri, 06 Jun 2008 23:37:18 +1000 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <9b08084c0806060548x701d07b2w7380e04a2ad50946@mail.gmail.com> References: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> <54568.1212602052@snookles.snookles.com> <9b08084c0806050252wcda24a8q17cf55cdd8b0dc8@mail.gmail.com> <6i9h44lc45nu67im1huupbofetkbkova6g@4ax.com> <9b08084c0806060548x701d07b2w7380e04a2ad50946@mail.gmail.com> Message-ID: <48493D8E.9000003@bitfurnace.com> On 6/6/2008 10:48 PM, Joe Armstrong wrote: > > Security has nothing to do with this argument - if a password is sent > over the network it has been > sent - nothing can alter that. > > A man in the middle might store the message forever so it would make > no difference if the > server stores the data for a millisecond or a trillion years. > > The reason for storing things in an appended log is to be able to > replay the log later if things go wrong and > recover from errors - it has nothing to do with security. Security and > privacy has to do with the level of > encryption that is applied to the items in the log. > > /Joe > Yes and no - security against legal attacks depends on the information being deleted (irrecoverable) after a certain point. From alexander.lamb@REDACTED Fri Jun 6 16:43:34 2008 From: alexander.lamb@REDACTED (Alexander Lamb) Date: Fri, 6 Jun 2008 16:43:34 +0200 Subject: [erlang-questions] Use of uuidgen with a mnesia application Message-ID: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> Hello, I need to generate ids to be used as mnesia keys which will be unique not only withing a cluster of Erlang nodes but also globally (e.g. may be read from a Java program). I initially thought of writing my own uuidgen function, but it seems like reading the mac address of network interfaces can be rather cumbersom depending on systems. So I decided to use uuidgen from the system and convert it to binary (for efficiency). Is this the correct way of doing things: verify_guuid(Value) -> A = lists:nth(9,Value), B = lists:nth(14,Value), C = lists:nth(19,Value), case [A,B,C] of "---" -> true; _ -> false end. make_guuid() -> First = os:cmd("uuidgen -t"), case verify_guuid(First) of true -> Result = First; false -> Second = os:cmd("uuidgen"), case verify_guuid(Second) of true -> Result = Second; false -> Result = "error", throw({uuidgen_error,Second}) end end, erlang:list_to_binary(Result). Now, the reason I have this cascade of case is because on Linux (our Ubuntu servers) it recognizes the -t option to generate the uuid and on MacOSX (our dev machines) it does not. Call me crazy, but I never feel confortable about using a totally random uuid to identity medical records that might be copied from system to system, exported, etc... even if the probability of collision is lower than that of a large asteroid hitting earth! What is your strategy for this (maybe this is a question to ask the couchdb developers, since they use guuids to identify the documents)? Thanks, Alex -- Alexander Lamb Founding Associate RODANOTECH S?rl 4 ch. de la Tour de Champel 1206 Geneva Switzerland Tel: 022 347 77 37 Fax: 022 347 77 38 http://www.rodanotech.ch From pfisher@REDACTED Fri Jun 6 16:58:52 2008 From: pfisher@REDACTED (Paul Fisher) Date: Fri, 06 Jun 2008 09:58:52 -0500 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> Message-ID: <1212764332.7281.105.camel@localhost> On Fri, 2008-06-06 at 16:43 +0200, Alexander Lamb wrote: > make_guuid() -> > First = os:cmd("uuidgen -t"), > case verify_guuid(First) of > true -> Result = First; > false -> Second = os:cmd("uuidgen"), > case verify_guuid(Second) of > true -> Result = Second; > false -> Result = "error", > throw({uuidgen_error,Second}) > end > end, > erlang:list_to_binary(Result). > > Now, the reason I have this cascade of case is because on Linux (our > Ubuntu servers) it recognizes the -t option to generate the uuid and > on MacOSX (our dev machines) it does not. > Call me crazy, but I never feel confortable about using a totally > random uuid to identity medical records that might be copied from > system to system, exported, etc... even if the probability of > collision is lower than that of a large asteroid hitting earth! > > What is your strategy for this (maybe this is a question to ask the > couchdb developers, since they use guuids to identify the documents)? We use uuid values to manage jobs on our cluster processing infrastructure and have a very simple driver that calls the uuid library directly and sends the tuple back to the caller (e.g. {uuid, <<156,21,52,30,51,213,17,221,179,34,0,28,37,116,215,248>>}). The os:cmd() trick is just a quick and dirty solution because of the differences in systems. Short of keeping track of unique ids in a database read/write from all cooperating systems (and potentially introducing a bottleneck or point of failure in the processes), uuid values are the best approach. -- paul From erlang@REDACTED Fri Jun 6 17:00:35 2008 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 6 Jun 2008 17:00:35 +0200 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <48493D8E.9000003@bitfurnace.com> References: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> <54568.1212602052@snookles.snookles.com> <9b08084c0806050252wcda24a8q17cf55cdd8b0dc8@mail.gmail.com> <6i9h44lc45nu67im1huupbofetkbkova6g@4ax.com> <9b08084c0806060548x701d07b2w7380e04a2ad50946@mail.gmail.com> <48493D8E.9000003@bitfurnace.com> Message-ID: <9b08084c0806060800t364b127audc1d89b91e0b865f@mail.gmail.com> On Fri, Jun 6, 2008 at 3:37 PM, Damien Morton wrote: > On 6/6/2008 10:48 PM, Joe Armstrong wrote: >> >> Security has nothing to do with this argument - if a password is sent >> over the network it has been >> sent - nothing can alter that. >> >> A man in the middle might store the message forever so it would make >> no difference if the >> server stores the data for a millisecond or a trillion years. >> >> The reason for storing things in an appended log is to be able to >> replay the log later if things go wrong and >> recover from errors - it has nothing to do with security. Security and >> privacy has to do with the level of >> encryption that is applied to the items in the log. >> >> /Joe >> > > Yes and no - security against legal attacks depends on the information being > deleted (irrecoverable) after a certain point. > If the man in the middle took all your data then deleting the data is irrelevant - anyway most attacks are illegal /Joe From alexander.lamb@REDACTED Fri Jun 6 17:06:40 2008 From: alexander.lamb@REDACTED (Alexander Lamb) Date: Fri, 6 Jun 2008 17:06:40 +0200 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: <1212764332.7281.105.camel@localhost> References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> <1212764332.7281.105.camel@localhost> Message-ID: <9EF53800-0FCE-4F05-93A9-49AA2569431B@rodanotech.ch> I see, but it means you wrote a piece of code in C (probably) to call uuid on the underlying system? (maybe it's the word "drive" that I didn't fully understand. Alex Le 6 juin 08 ? 16:58, Paul Fisher a ?crit : > On Fri, 2008-06-06 at 16:43 +0200, Alexander Lamb wrote: >> make_guuid() -> >> First = os:cmd("uuidgen -t"), >> case verify_guuid(First) of >> true -> Result = First; >> false -> Second = os:cmd("uuidgen"), >> case verify_guuid(Second) of >> true -> Result = Second; >> false -> Result = "error", >> throw({uuidgen_error,Second}) >> end >> end, >> erlang:list_to_binary(Result). >> >> Now, the reason I have this cascade of case is because on Linux (our >> Ubuntu servers) it recognizes the -t option to generate the uuid and >> on MacOSX (our dev machines) it does not. >> Call me crazy, but I never feel confortable about using a totally >> random uuid to identity medical records that might be copied from >> system to system, exported, etc... even if the probability of >> collision is lower than that of a large asteroid hitting earth! >> >> What is your strategy for this (maybe this is a question to ask the >> couchdb developers, since they use guuids to identify the documents)? > > We use uuid values to manage jobs on our cluster processing > infrastructure and have a very simple driver that calls the uuid > library > directly and sends the tuple back to the caller (e.g. {uuid, > <<156,21,52,30,51,213,17,221,179,34,0,28,37,116,215,248>>}). The > os:cmd() trick is just a quick and dirty solution because of the > differences in systems. > > Short of keeping track of unique ids in a database read/write from all > cooperating systems (and potentially introducing a bottleneck or point > of failure in the processes), uuid values are the best approach. > > > -- > paul > From pfisher@REDACTED Fri Jun 6 17:12:10 2008 From: pfisher@REDACTED (Paul Fisher) Date: Fri, 06 Jun 2008 10:12:10 -0500 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: <9EF53800-0FCE-4F05-93A9-49AA2569431B@rodanotech.ch> References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> <1212764332.7281.105.camel@localhost> <9EF53800-0FCE-4F05-93A9-49AA2569431B@rodanotech.ch> Message-ID: <1212765130.7281.107.camel@localhost> On Fri, 2008-06-06 at 17:06 +0200, Alexander Lamb wrote: > I see, but it means you wrote a piece of code in C (probably) to call > uuid on the underlying system? > > (maybe it's the word "drive" that I didn't fully understand. Yes, a linked-in erlang driver, written in C. Once you write the first one it is not so bad after that. -- paul From andy@REDACTED Fri Jun 6 17:25:32 2008 From: andy@REDACTED (Andy Gross) Date: Fri, 6 Jun 2008 11:25:32 -0400 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: <1212765130.7281.107.camel@localhost> References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> <1212764332.7281.105.camel@localhost> <9EF53800-0FCE-4F05-93A9-49AA2569431B@rodanotech.ch> <1212765130.7281.107.camel@localhost> Message-ID: You can also use the following method for generating unique identifiers: 1> crypto:start(). ok 2> <> = crypto:sha(term_to_binary({make_ref(), now()})). <41,13,88,29,5,111,251,80,169,13,157,98,38,200,99,138,10, 166,160,63>> 3> UUIDString = erlang:integer_to_list(I, 16). "4EDB9BAAB77AF8612C14390672C9D9B9BFB3F32A" 4> - Andy On Jun 6, 2008, at 11:12 AM, Paul Fisher wrote: > On Fri, 2008-06-06 at 17:06 +0200, Alexander Lamb wrote: >> I see, but it means you wrote a piece of code in C (probably) to call >> uuid on the underlying system? >> >> (maybe it's the word "drive" that I didn't fully understand. > > Yes, a linked-in erlang driver, written in C. Once you write the > first > one it is not so bad after that. > > > -- > paul > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From anders.nygren@REDACTED Fri Jun 6 17:29:43 2008 From: anders.nygren@REDACTED (Anders Nygren) Date: Fri, 6 Jun 2008 10:29:43 -0500 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> Message-ID: On Fri, Jun 6, 2008 at 9:43 AM, Alexander Lamb wrote: > Hello, > > I need to generate ids to be used as mnesia keys which will be unique > not only withing a cluster of Erlang nodes but also globally (e.g. may > be read from a Java program). > > I initially thought of writing my own uuidgen function, but it seems > like reading the mac address of network interfaces can be rather > cumbersom depending on systems. > > So I decided to use uuidgen from the system and convert it to binary > (for efficiency). > > Is this the correct way of doing things: > > verify_guuid(Value) -> > A = lists:nth(9,Value), > B = lists:nth(14,Value), > C = lists:nth(19,Value), > case [A,B,C] of > "---" -> true; > _ -> false > end. > > make_guuid() -> > First = os:cmd("uuidgen -t"), > case verify_guuid(First) of > true -> Result = First; > false -> Second = os:cmd("uuidgen"), > case verify_guuid(Second) of > true -> Result = Second; > false -> Result = "error", > throw({uuidgen_error,Second}) > end > end, > erlang:list_to_binary(Result). > > Now, the reason I have this cascade of case is because on Linux (our > Ubuntu servers) it recognizes the -t option to generate the uuid and > on MacOSX (our dev machines) it does not. Wouldn't it be "nicer" to use os:type/0 and os:version/0 to detect what system You are running on? /Anders > Call me crazy, but I never feel confortable about using a totally > random uuid to identity medical records that might be copied from > system to system, exported, etc... even if the probability of > collision is lower than that of a large asteroid hitting earth! > > What is your strategy for this (maybe this is a question to ask the > couchdb developers, since they use guuids to identify the documents)? > > Thanks, > > Alex > -- > Alexander Lamb > Founding Associate > RODANOTECH S?rl > > 4 ch. de la Tour de Champel > 1206 Geneva > Switzerland > > Tel: 022 347 77 37 > Fax: 022 347 77 38 > > http://www.rodanotech.ch > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From pfisher@REDACTED Fri Jun 6 17:35:44 2008 From: pfisher@REDACTED (Paul Fisher) Date: Fri, 06 Jun 2008 10:35:44 -0500 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> <1212764332.7281.105.camel@localhost> <9EF53800-0FCE-4F05-93A9-49AA2569431B@rodanotech.ch> <1212765130.7281.107.camel@localhost> Message-ID: <1212766544.7281.116.camel@localhost> On Fri, 2008-06-06 at 11:25 -0400, Andy Gross wrote: > You can also use the following method for generating unique identifiers: > > 1> crypto:start(). > ok > 2> <> = crypto:sha(term_to_binary({make_ref(), now()})). > <41,13,88,29,5,111,251,80,169,13,157,98,38,200,99,138,10, > 166,160,63>> > 3> UUIDString = erlang:integer_to_list(I, 16). > "4EDB9BAAB77AF8612C14390672C9D9B9BFB3F32A" > 4> That is not guaranteed to be unique. make_ref() starts over each time the node restarts, so the uniqueness depends on two nodes not generating at the same microsecond (local clock perspective wise, that is.) Ultimately, this is only slightly better than using the value of now() directly. (uuid values work similar to this, but have some number of bits drawn from a random source to salt the result.) If you want to have this type of unique identifier, you would use whatever the crypto interface was to access random number generator. uuid values have the benefit of being able to use the embedded MAC address to tie the id back to the machine generating it (which come in handy in clusters of nodes.) -- paul From japerk@REDACTED Fri Jun 6 17:45:22 2008 From: japerk@REDACTED (Jacob Perkins) Date: Fri, 6 Jun 2008 08:45:22 -0700 Subject: [erlang-questions] Use of uuidgen with a mnesia Message-ID: Another way to generate uuids is to have a permanent gen_server process generate random strings. Having a long running process ensures good randomness since random:seed uses the process dictionary. Below is a random alphanumeric string generation function that you could plug in to a gen_server that calls random:seed at init time. random(Size) -> Chars = lists:seq($0, $9) ++ lists:seq($A, $Z) ++ lists:seq($a, $z), N = length(Chars), F = fun(_, S) -> [lists:nth(random:uniform(N), Chars) | S] end, lists:foldl(F, [], lists:seq(1, Size)). Date: Fri, 6 Jun 2008 11:25:32 -0400 > From: Andy Gross > Subject: Re: [erlang-questions] Use of uuidgen with a mnesia > application > To: erlang-questions@REDACTED > Message-ID: > Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes > > > You can also use the following method for generating unique identifiers: > > 1> crypto:start(). > ok > 2> <> = crypto:sha(term_to_binary({make_ref(), now()})). > <41,13,88,29,5,111,251,80,169,13,157,98,38,200,99,138,10, > 166,160,63>> > 3> UUIDString = erlang:integer_to_list(I, 16). > "4EDB9BAAB77AF8612C14390672C9D9B9BFB3F32A" > 4> > > - Andy > > On Jun 6, 2008, at 11:12 AM, Paul Fisher wrote: > > > On Fri, 2008-06-06 at 17:06 +0200, Alexander Lamb wrote: > >> I see, but it means you wrote a piece of code in C (probably) to call > >> uuid on the underlying system? > >> > >> (maybe it's the word "drive" that I didn't fully understand. > > > > Yes, a linked-in erlang driver, written in C. Once you write the > > first > > one it is not so bad after that. > > > > > > -- > > paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmercer@REDACTED Fri Jun 6 17:52:33 2008 From: dmercer@REDACTED (David Mercer) Date: Fri, 6 Jun 2008 10:52:33 -0500 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch><1212764332.7281.105.camel@localhost><9EF53800-0FCE-4F05-93A9-49AA2569431B@rodanotech.ch><1212765130.7281.107.camel@localhost> Message-ID: <008a01c8c7ed$561c4910$f21ea8c0@SSI.CORP> On Friday, June 06, 2008, Andy Gross wrote: > You can also use the following method for generating unique identifiers: > > 1> crypto:start(). > ok > 2> <> = crypto:sha(term_to_binary({make_ref(), now()})). > <41,13,88,29,5,111,251,80,169,13,157,98,38,200,99,138,10, > 166,160,63>> > 3> UUIDString = erlang:integer_to_list(I, 16). > "4EDB9BAAB77AF8612C14390672C9D9B9BFB3F32A" If you go with this approach, you need to handle leading zeroes. Change line 3 to: 3> lists:flatten(io_lib:fwrite("~40..0s", [erlang:integer_to_list(I, 16)])). DBM From alexander.lamb@REDACTED Fri Jun 6 17:53:52 2008 From: alexander.lamb@REDACTED (Alexander Lamb) Date: Fri, 6 Jun 2008 17:53:52 +0200 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> Message-ID: Yes, I agree it would be better. Actually, there is a mistake in my code: it is necessary to add a lists:sublist(Result,36) because cmd returns the command with the trailing carriage return! From what I see by googling uuid erlang, it seems like this could be a welcome addition to the basic erlang package. Alex Le 6 juin 08 ? 17:29, Anders Nygren a ?crit : > On Fri, Jun 6, 2008 at 9:43 AM, Alexander Lamb > wrote: >> Hello, >> >> I need to generate ids to be used as mnesia keys which will be unique >> not only withing a cluster of Erlang nodes but also globally (e.g. >> may >> be read from a Java program). >> >> I initially thought of writing my own uuidgen function, but it seems >> like reading the mac address of network interfaces can be rather >> cumbersom depending on systems. >> >> So I decided to use uuidgen from the system and convert it to binary >> (for efficiency). >> >> Is this the correct way of doing things: >> >> verify_guuid(Value) -> >> A = lists:nth(9,Value), >> B = lists:nth(14,Value), >> C = lists:nth(19,Value), >> case [A,B,C] of >> "---" -> true; >> _ -> false >> end. >> >> make_guuid() -> >> First = os:cmd("uuidgen -t"), >> case verify_guuid(First) of >> true -> Result = First; >> false -> Second = os:cmd("uuidgen"), >> case verify_guuid(Second) of >> true -> Result = >> Second; >> false -> Result = >> "error", >> >> throw({uuidgen_error,Second}) >> end >> end, >> erlang:list_to_binary(Result). >> >> Now, the reason I have this cascade of case is because on Linux (our >> Ubuntu servers) it recognizes the -t option to generate the uuid and >> on MacOSX (our dev machines) it does not. > > Wouldn't it be "nicer" to use os:type/0 and os:version/0 to detect > what system You are running on? > > /Anders > >> Call me crazy, but I never feel confortable about using a totally >> random uuid to identity medical records that might be copied from >> system to system, exported, etc... even if the probability of >> collision is lower than that of a large asteroid hitting earth! >> >> What is your strategy for this (maybe this is a question to ask the >> couchdb developers, since they use guuids to identify the documents)? >> >> Thanks, >> >> Alex >> -- >> Alexander Lamb >> Founding Associate >> RODANOTECH S?rl >> >> 4 ch. de la Tour de Champel >> 1206 Geneva >> Switzerland >> >> Tel: 022 347 77 37 >> Fax: 022 347 77 38 >> >> http://www.rodanotech.ch >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > From dnew@REDACTED Fri Jun 6 18:24:19 2008 From: dnew@REDACTED (Darren New) Date: Fri, 06 Jun 2008 09:24:19 -0700 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: <1212766544.7281.116.camel@localhost> References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> <1212764332.7281.105.camel@localhost> <9EF53800-0FCE-4F05-93A9-49AA2569431B@rodanotech.ch> <1212765130.7281.107.camel@localhost> <1212766544.7281.116.camel@localhost> Message-ID: <484964B3.8020509@san.rr.com> Paul Fisher wrote: > That is not guaranteed to be unique. I would hash the contents of the record itself. If you get a collision, it's probably because the records are identical anyway. :-) -- Darren New / San Diego, CA, USA (PST) "That's pretty. Where's that?" "It's the Age of Channelwood." "We should go there on vacation some time." From dnew@REDACTED Fri Jun 6 18:19:52 2008 From: dnew@REDACTED (Darren New) Date: Fri, 06 Jun 2008 09:19:52 -0700 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <9b08084c0806060800t364b127audc1d89b91e0b865f@mail.gmail.com> References: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> <54568.1212602052@snookles.snookles.com> <9b08084c0806050252wcda24a8q17cf55cdd8b0dc8@mail.gmail.com> <6i9h44lc45nu67im1huupbofetkbkova6g@4ax.com> <9b08084c0806060548x701d07b2w7380e04a2ad50946@mail.gmail.com> <48493D8E.9000003@bitfurnace.com> <9b08084c0806060800t364b127audc1d89b91e0b865f@mail.gmail.com> Message-ID: <484963A8.4080607@san.rr.com> Joe Armstrong wrote: >> Yes and no - security against legal attacks depends on the information being >> deleted (irrecoverable) after a certain point. > > If the man in the middle took all your data then deleting the data is > irrelevant - anyway most attacks > are illegal For MITM to work, the man has to be in the middle when the message is sent. If you save things forever on your server, you present years worth of potential value to anyone who breaks into your server. This is exactly why your credit card has those three or four extra printed digits on it. Merchants are contractually disallowed from storing those longer than it takes to process the transaction, so anyone who breaks into the server afterwards will be unable to use those cards without the company doing additional checking. This is why sshd changes its keys every few hours - old streams recorded by attackers become unbreakable even if they manage to break into the server. I believe a "legal" attack was meant to imply a subpoena. Mr Gates, for example, was quite brutalized by email messages he had written years earlier that would not have been available to his detractors had he a policy of purging any emails more than a few months old. Encrypting the data only helps if the court is unable to compel you to decrypt the data. And if the system is processing it, it must be decrypted, so a hacker is going to be able to see it anyway. -- Darren New / San Diego, CA, USA (PST) "That's pretty. Where's that?" "It's the Age of Channelwood." "We should go there on vacation some time." From pfisher@REDACTED Fri Jun 6 19:02:34 2008 From: pfisher@REDACTED (Paul Fisher) Date: Fri, 06 Jun 2008 12:02:34 -0500 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: <484964B3.8020509@san.rr.com> References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> <1212764332.7281.105.camel@localhost> <9EF53800-0FCE-4F05-93A9-49AA2569431B@rodanotech.ch> <1212765130.7281.107.camel@localhost> <1212766544.7281.116.camel@localhost> <484964B3.8020509@san.rr.com> Message-ID: <1212771754.7281.120.camel@localhost> On Fri, 2008-06-06 at 09:24 -0700, Darren New wrote: > Paul Fisher wrote: > > That is not guaranteed to be unique. > > I would hash the contents of the record itself. If you get a collision, > it's probably because the records are identical anyway. :-) That was my point, performing the SHA hash on term_to_binary( {make_ref(), now()} ) does not alter the uniqueness of the result. It only randomizes the distribution of the generated "id values". -- paul From tty.erlang@REDACTED Fri Jun 6 19:55:11 2008 From: tty.erlang@REDACTED (t ty) Date: Fri, 6 Jun 2008 13:55:11 -0400 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: <1212764332.7281.105.camel@localhost> References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> <1212764332.7281.105.camel@localhost> Message-ID: <290b3ba10806061055q54727ef2iebf3d8184325b202@mail.gmail.com> There is an Erlang UUID module called erlUUID which uses OSSP uuid. http://nixbit.com/cat//utilities/erluuid/ t From dnew@REDACTED Fri Jun 6 19:31:26 2008 From: dnew@REDACTED (Darren New) Date: Fri, 06 Jun 2008 10:31:26 -0700 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: <1212771754.7281.120.camel@localhost> References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> <1212764332.7281.105.camel@localhost> <9EF53800-0FCE-4F05-93A9-49AA2569431B@rodanotech.ch> <1212765130.7281.107.camel@localhost> <1212766544.7281.116.camel@localhost> <484964B3.8020509@san.rr.com> <1212771754.7281.120.camel@localhost> Message-ID: <4849746E.4060808@san.rr.com> Paul Fisher wrote: > On Fri, 2008-06-06 at 09:24 -0700, Darren New wrote: >> Paul Fisher wrote: >>> That is not guaranteed to be unique. >> I would hash the contents of the record itself. If you get a collision, >> it's probably because the records are identical anyway. :-) > > That was my point, performing the SHA hash on > term_to_binary( {make_ref(), now()} ) does not alter the uniqueness of > the result. It only randomizes the distribution of the generated "id > values". The important part of my statement was not "sha1" but "actual contents." For example, in the system I have now, I generate fingerprints for music and put them in a database. Since there are a number of independent servers in separate cities for which I'm generating fingerprints, the identification could be one of two things: The host name + pid + timestamp (which can be messy), or just the sha1 of the actual contents of the audio file getting fingerprinted. If the sha1 comes out the same both times, the most likely causes is that I fingerprinted the same digital version of the same song twice. Use the real life stuff-that-wont-change as the key for the record. (Or a hash thereof, if you want a single unique string.) As soon as you start making up arbitrary values and trying to map real-world items onto them, you start having to manage the arbitrary-value generation and the mapping. Of course, it's difficult to run a human being through SHA1, so at some point you need to pick something you think won't change or otherwise handle the primary key changing (say, by chaining to the records from before the patient changed names or something). -- Darren New / San Diego, CA, USA (PST) "That's pretty. Where's that?" "It's the Age of Channelwood." "We should go there on vacation some time." From hzerlang@REDACTED Fri Jun 6 20:27:41 2008 From: hzerlang@REDACTED (=?iso-8859-1?Q?Horv=E1th_zolt=E1n_-Erlang_lev=2Elist=E1hoz?=) Date: Fri, 6 Jun 2008 20:27:41 +0200 (CEST) Subject: [erlang-questions] Last CFP : Erlang Workshop 2008 - deadline: June 17 In-Reply-To: References: Message-ID: Second Call for papers Seventh ACM SIGPLAN Erlang Workshop Victoria, British Columbia, Canada, September 27, 2008 http://www.erlang.org/workshop/2008/ Satellite event of ACM SIGPLAN International Conference on Functional Programming, September 22-24, 2008 The Erlang workshop will take place during one day in connection with ICFP'08 (see http://www.icfpconference.org/icfp2008/) Important Dates Submission deadline (extended) - Midnight GMT June 17 Author notification - Midnight GMT June 30 Final submission - Midnight GMT July 14 Erlang is a concurrent, distributed functional programming language aimed at systems with requirements on massive concurrency, soft real time response, fault tolerance, and high availability. It has been available as open source for several years creating a community that actively contributes to its already existing rich set of libraries and applications. Originally created for telecom applications, its usage has spread to other domains including e-commerce, banking, and computer telephony. Erlang programs are today among the largest applications written in any functional programming language. These applications offer new opportunities to evaluate functional programming and functional programming methods on a very large scale and suggest new problems for the research community to solve. This workshop will bring together the open source, academic, and industrial programming communities of Erlang. It will enable participants to familiarize themselves with recent developments on new techniques and tools tailored to Erlang, novel applications, draw lessons from users' experiences and identify research problems and common areas relevant to the practice of Erlang and functional programming. Submission and publication information The workshop will have printed proceedings which will be distributed to the participants at the time of the workshop. Subsequently, the papers will be included in the ACM Digital Library. Authors are therefore advised to use the Author Information for SIGPLAN Conferences when preparing their submissions. Authors should submit provisional full papers. Submissions will be accepted electronically; the submission page is not yet ready. The length should be restricted to 12 pages in standard ACM format. In previous workshops we have had a mix of papers representing academic research and industrial experiences with Erlang. We particularly welcome papers describing new and novel application of Erlang - if you have any doubts about the suitability of a paper please feel free to contact the Program Chair for advice. Authors should submit their papers using the submission website: http://www-old.inf.elte.hu/rendezvenyek/erlangws08/openconf/ In case of technical problems, please contact erlangws08@REDACTED Workshop Chair Tee Teoh, Canadian Bank Note, Ottawa, Canada Program Chair Zolt?n Horv?th, Department of Programming Languages and Programming, Faculty of Informatics, E?tv?s Lor?nd University, Budapest, Hungary Program Committee Thomas Arts, IT University, G?teborg, Sweden Francesco Cesarini, Erlang Training and Consulting, London, UK Clara Benac Earle, University Carlos III, Madrid, Spain John Hughes, Chalmers University of Technology, G?teborg, Sweden Erik Stenman, Kreditor, Stockholm, Sweden Zolt?n Theisz, Ericsson, Ireland Simon Thompson, University of Kent,Canterbury, UK Rex Page, University of Oklahoma, USA For Venue and registration, please see the ICFP web site Related Links Erlang Workshop web site: http://www.erlang.org/workshop/2008/ ICFP 2008 web site: http://www.icfpconference.org/icfp2008/ Past ACM SIGPLAN Erlang workshops http://www.erlang.se/workshop Open Source Erlang http://www.erlang.org/ From frank@REDACTED Fri Jun 6 21:07:50 2008 From: frank@REDACTED (Frank Mueller) Date: Fri, 6 Jun 2008 21:07:50 +0200 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> Message-ID: <21B374D6-C3B7-4F53-8A4A-E0D2768A4775@mweb.de> Hi Alexander, in http://code.google.com/p/tideland-cel/source/browse/trunk/src/celutl.erl you'll find my process and functions for the generation of guids. Regards mue Am 06.06.2008 um 16:43 schrieb Alexander Lamb: > Hello, > > I need to generate ids to be used as mnesia keys which will be unique > not only withing a cluster of Erlang nodes but also globally (e.g. may > be read from a Java program). > > I initially thought of writing my own uuidgen function, but it seems > like reading the mac address of network interfaces can be rather > cumbersom depending on systems. > > So I decided to use uuidgen from the system and convert it to binary > (for efficiency). > > Is this the correct way of doing things: > > verify_guuid(Value) -> > A = lists:nth(9,Value), > B = lists:nth(14,Value), > C = lists:nth(19,Value), > case [A,B,C] of > "---" -> true; > _ -> false > end. > > make_guuid() -> > First = os:cmd("uuidgen -t"), > case verify_guuid(First) of > true -> Result = First; > false -> Second = os:cmd("uuidgen"), > case verify_guuid(Second) of > true -> Result = Second; > false -> Result = "error", > throw({uuidgen_error,Second}) > end > end, > erlang:list_to_binary(Result). > > Now, the reason I have this cascade of case is because on Linux (our > Ubuntu servers) it recognizes the -t option to generate the uuid and > on MacOSX (our dev machines) it does not. > Call me crazy, but I never feel confortable about using a totally > random uuid to identity medical records that might be copied from > system to system, exported, etc... even if the probability of > collision is lower than that of a large asteroid hitting earth! > > What is your strategy for this (maybe this is a question to ask the > couchdb developers, since they use guuids to identify the documents)? > > Thanks, > > Alex > -- > Alexander Lamb > Founding Associate > RODANOTECH S?rl > > 4 ch. de la Tour de Champel > 1206 Geneva > Switzerland > > Tel: 022 347 77 37 > Fax: 022 347 77 38 > > http://www.rodanotech.ch > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- ** ** Frank Mueller / Oldenburg / Germany ** ** http://frank.mweb.de and http://mue.tideland.biz ** From erlang@REDACTED Fri Jun 6 22:44:51 2008 From: erlang@REDACTED (Dominic Williams) Date: Fri, 06 Jun 2008 22:44:51 +0200 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <8268eea30806032346l463f42f4q798f47db08de84b3@mail.gmail.com> Message-ID: <4849A1C3.1040303@dominicwilliams.net> Hello Richard, Richard A. O'Keefe a ?crit : >> What if it would be possible to create Message Queues in addition to >> process "default" Message Queue? >> Where it would be possible to give options such as: >> * limit to the amount message to store >> [...] > > I'm beginning to sound like Little Johnny One-Note, > but we can already get the effect of multiple message queues by > having helper processes and giving out their PIDs instead. > [...] > counter(Self, Limit) when Limit > 0 -> > receive > {Self,Delta} -> counter(Self, Limit+Delta) > ; Msg -> Self ! Msg, > counter(Self, Limit-1) > end; > counter(Self, 0) -> > receive > {Self,Delta} -> counter(Self, Delta) > end. You are quite right that this could be used to achieve the semantics of a separate (and bounded) message queue. However, the original motivation was to avoid the occasional snowball effect that can result from selective receives, so I think using selective receive in the helper process defeats the purpose. I wonder whether we could still use a helper process to regulate messages for the handler process, but it would have to use the catch all receive. Maybe: 1) regulate by n? of messages per second (this can be done without communicating with the handler process) 2) use your counter idea, but something like this: counter (Self, Limit) -> receive Message -> counter (Self, Limit, Message) end. counter (Self, Limit, {Self, Delta}) -> counter (Self, Limit + Delta); counter (Self, 0, _) -> counter (Self, 0); counter (Self, Limit, Message) -> Self ! Message, counter (Self, Limit -1). Hmmm. I'm tempted to actually try that out... Regards, Dominic Williams http://dominicwilliams.net ---- From rvirding@REDACTED Fri Jun 6 23:23:39 2008 From: rvirding@REDACTED (Robert Virding) Date: Fri, 6 Jun 2008 23:23:39 +0200 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <8268eea30806040551r46127891hd2a9db0d8882c7e8@mail.gmail.com> References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <8268eea30806032346l463f42f4q798f47db08de84b3@mail.gmail.com> <95be1d3b0806040006q4ac93245x1c8405014398c5f7@mail.gmail.com> <8268eea30806040551r46127891hd2a9db0d8882c7e8@mail.gmail.com> Message-ID: <3dbc6d1c0806061423r57c17bc9y62f9c3130369adfe@mail.gmail.com> Do you envisage your queues as separate things or always bound to original process? Way back in the dark misty beginnings of Erlang we had a wonderful concept called the pipe to which messages were sent and received. Pipes has their own lives and were not bound to an original process. With these pipes you could do wonderful things like redirect them and join them together (muff_pipe/2). However, they were also by far the most complex and difficult part of the early Erlang interpreter so we were very glad to see them go. Joe wrote a bit about them in his HOPL paper. And at that stage we had not yet begun to think about distribution. Anyway the goal of this rather long discourse is "no, I don't think this is a good idea". It is extremely easy to create really weird interactions with this I think. I have a serious question which I ask out of ignorance. I know that really long message queues are possible in the existing language, it is easy to write a program to show this, but do they occur in real applications? Or is this theoretical worry? And if they do occur are they something which is property of the problem which can't easily be avoided or are they the result of some error in the logic or coding which shows up in message queue length? Robert 2008/6/4 Andreas Hillqvist : > Thank you for your answer. > > For sending I was thinking that it would be possible to use the Queue > Identifyer (somthing similar to a PID): > Queue ! Message > > And that Queue are only linked to process. > > > One possebility with Multiple Queues is that it would be possible to > Prioritize messages more efficiently: > * If there is a message in Queue read messsage, else repeat for next > Queue. > * If there is no message in any Queue, wait for a message from any Queue. > > It might be nice to have one Queue for system messages and such. > > > I do not know if the Multiple Queues is the right path. > It would be possible to build some of the functionality with helper > process for Queue. > > And that could be part of the framework you mentioned and let the > future decide if the Multiple Queues semantics will catch on. > > It might then be omre efficently be implemented in the Erlang Runtime. > > > Kind Regards > Andreas Hillqvist > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul-trapexit@REDACTED Sat Jun 7 04:38:11 2008 From: paul-trapexit@REDACTED (Paul Mineiro) Date: Fri, 6 Jun 2008 19:38:11 -0700 (PDT) Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <3dbc6d1c0806061423r57c17bc9y62f9c3130369adfe@mail.gmail.com> References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <8268eea30806032346l463f42f4q798f47db08de84b3@mail.gmail.com> <95be1d3b0806040006q4ac93245x1c8405014398c5f7@mail.gmail.com> <8268eea30806040551r46127891hd2a9db0d8882c7e8@mail.gmail.com> <3dbc6d1c0806061423r57c17bc9y62f9c3130369adfe@mail.gmail.com> Message-ID: We saw big message queues for the logger process in yaws. We upped the priority of the logger process and they went away. That was basically an N -> 1 problem. -- p On Fri, 6 Jun 2008, Robert Virding wrote: > Do you envisage your queues as separate things or always bound to original > process? > > Way back in the dark misty beginnings of Erlang we had a wonderful concept > called the pipe to which messages were sent and received. Pipes has their > own lives and were not bound to an original process. With these pipes you > could do wonderful things like redirect them and join them together > (muff_pipe/2). However, they were also by far the most complex and difficult > part of the early Erlang interpreter so we were very glad to see them go. > Joe wrote a bit about them in his HOPL paper. > > And at that stage we had not yet begun to think about distribution. > > Anyway the goal of this rather long discourse is "no, I don't think this is > a good idea". It is extremely easy to create really weird interactions with > this I think. > > I have a serious question which I ask out of ignorance. I know that really > long message queues are possible in the existing language, it is easy to > write a program to show this, but do they occur in real applications? Or is > this theoretical worry? And if they do occur are they something which is > property of the problem which can't easily be avoided or are they the result > of some error in the logic or coding which shows up in message queue length? > > Robert > > 2008/6/4 Andreas Hillqvist : > > > Thank you for your answer. > > > > For sending I was thinking that it would be possible to use the Queue > > Identifyer (somthing similar to a PID): > > Queue ! Message > > > > And that Queue are only linked to process. > > > > > > One possebility with Multiple Queues is that it would be possible to > > Prioritize messages more efficiently: > > * If there is a message in Queue read messsage, else repeat for next > > Queue. > > * If there is no message in any Queue, wait for a message from any Queue. > > > > It might be nice to have one Queue for system messages and such. > > > > > > I do not know if the Multiple Queues is the right path. > > It would be possible to build some of the functionality with helper > > process for Queue. > > > > And that could be part of the framework you mentioned and let the > > future decide if the Multiple Queues semantics will catch on. > > > > It might then be omre efficently be implemented in the Erlang Runtime. > > > > > > Kind Regards > > Andreas Hillqvist > > > In an artificial world, only extremists live naturally. -- Paul Graham From jilani@REDACTED Sat Jun 7 10:34:42 2008 From: jilani@REDACTED (Jilani Khaldi) Date: Sat, 07 Jun 2008 10:34:42 +0200 Subject: [erlang-questions] A math expression parser in Erlang In-Reply-To: <008a01c8c7ed$561c4910$f21ea8c0@SSI.CORP> References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch><1212764332.7281.105.camel@localhost><9EF53800-0FCE-4F05-93A9-49AA2569431B@rodanotech.ch><1212765130.7281.107.camel@localhost> <008a01c8c7ed$561c4910$f21ea8c0@SSI.CORP> Message-ID: <484A4822.3050806@cheapnet.it> I am looking for math expression parser in Erlang, did someone already write it? Otherwise, any hint to write one is appreciated. Thank you. Jilani From gleber.p@REDACTED Sat Jun 7 12:38:42 2008 From: gleber.p@REDACTED (Gleb Peregud) Date: Sat, 7 Jun 2008 12:38:42 +0200 Subject: [erlang-questions] cacherl memcached app startup problem question In-Reply-To: <1218d6a50806021819m354f65vddcdd659d40b90f4@mail.gmail.com> References: <1218d6a50805301550u50ddc883l4e07607c2e70f760@mail.gmail.com> <14f0e3620805310300t175e4ea0wb539e3c8f1a84786@mail.gmail.com> <1218d6a50806020755w263cb8a5o49f489ed2cc8b53c@mail.gmail.com> <1218d6a50806020813u6d5550e6oba58925cc8cc4b45@mail.gmail.com> <1218d6a50806021819m354f65vddcdd659d40b90f4@mail.gmail.com> Message-ID: <14f0e3620806070338s48f927fdh5546b373f4beceff@mail.gmail.com> I'm not sure what do you mean by this. Maybe because it's just a client? :) memchached_client.erl is implemented in proper way, by abstracting out internal structures. You just create new client, store it's "handle" (it doesn't matter what is inside this structure) and pass this "handle" to other API functions. I was not able to reproduce errors mentioned in your first mail on my system and i have no idea what is going on. Do this problem persists? Were you able to fix it? Have you checked permissions of /var/mnesia/* directories? On Tue, Jun 3, 2008 at 3:19 AM, db wrote: > Any reason why memcached_client doesn't use a gen_server? Seems like > I have to manually keep track of the states. > > -- > rk > > That which we persist in doing becomes easier for us to do; not that > the nature of the thing itself is changed, but that our power to do is > increased. > -Ralph Waldo Emerson > -- Gleb Peregud http://gleber.pl/ Every minute is to be grasped. Time waits for nobody. -- Inscription on a Zen Gong From klphanikar@REDACTED Sat Jun 7 13:13:18 2008 From: klphanikar@REDACTED (Phanikar.K) Date: Sat, 7 Jun 2008 16:43:18 +0530 Subject: [erlang-questions] Question Message-ID: Hi, I am trying to make an distributed application through which I will be able to run the same application using remote disk-less nodes. Regarding this issue I am facing some problems . I have created a target system and am able to run it on two nodes with distributed option in the .config file. But Now I wanted to add some more nodes where I don't want even to have any binaries of the modules in the application but should be able to work with the application running on the first two nodes. 1.Is there any way to push the application from any one of the first two nodes on to all the connected nodes as we do with the module using nl(module). Can i do the same for the application as nl(module) works only with the modules but not applications? 2. And if the first two nodes are running on two different machines then do I need to do the hot code loading on both the machines or is there any option that if I update the code on one machine that should be updated on the other machine as these two machines are using an distributed application? Please suggest me if there is any round about method also. Please answer me these questions as I am stuck up with my project at this point. Thanks, With Regards, Phanikar.K.L. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay@REDACTED Sat Jun 7 17:58:02 2008 From: jay@REDACTED (Jay Nelson) Date: Sat, 7 Jun 2008 08:58:02 -0700 Subject: [erlang-questions] Documentation code error Message-ID: <0C8B1BBB-5846-4F85-9693-AC4BF0BD8C1C@duomark.com> On file:////usr/local/lib/erlang/doc/system_principles/ part_frame.html the target_system code still calls file:rawopen/2 in the function copy_file/3. It should call file:open/2 and use a list for the options rather than a tuple. jay From jay@REDACTED Sat Jun 7 18:06:46 2008 From: jay@REDACTED (Jay Nelson) Date: Sat, 7 Jun 2008 09:06:46 -0700 Subject: [erlang-questions] [clarify] Limits and overhead of monitors Message-ID: Has anyone determined the answer to any of the following questions: 1) Is a practical limit to the number of monitors a single process can hold? - I assume they are fairly static, so just the memory to store them? - how are they stored? - any problems if I have 100K processes in an everyone monitoring everyone configuration? 2) How much overhead difference is there between a local monitor and remote node process monitor? - The pinging between nodes shouldn't need to propagate, just the failure to ping a node 3) Is there any difference between a peer-to-peer distributed mutual monitoring situation and a star network organization where the monitoring is located in the central pool of process? - Total number of monitors is fewer - Notification of down process should propagate in a similar amount of time (although I would have to manually chase down connections in the 2nd case by maintaining my own relationships) jay From valentin@REDACTED Sat Jun 7 16:03:21 2008 From: valentin@REDACTED (Valentin Micic) Date: Sat, 7 Jun 2008 16:03:21 +0200 Subject: [erlang-questions] On selective receive (Re: eep: multiplepatterns) References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com><8268eea30806032346l463f42f4q798f47db08de84b3@mail.gmail.com><95be1d3b0806040006q4ac93245x1c8405014398c5f7@mail.gmail.com><8268eea30806040551r46127891hd2a9db0d8882c7e8@mail.gmail.com> <3dbc6d1c0806061423r57c17bc9y62f9c3130369adfe@mail.gmail.com> Message-ID: <004801c8c8a7$40a44460$6501a8c0@moneymaker2> >From: Robert Virding > I have a serious question which I ask out of ignorance. ... > but do they occur in real applications? Or is this theoretical worry? Well, in my experience these things do happen during load-stress testing usually identifying some bottleneck overlooked during the functional testing. In one such case, this bottleneck got exposed only when we attempted tests with rates that were higher than, say, 1000 things(*) per second. In fact, the long message queue and performance degradation thereof, helped us optimise the system by pinpointing the bottleneck... with a relatively simple optimization we achieved 3000 things per second (target was 2000). I'm not sure, but another time when message queu emight grow is during the network outages -- but I do not have recent experiences in this regard ;-). V. (*) In this particular case TCAP transactions per second. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pfisher@REDACTED Sat Jun 7 18:29:12 2008 From: pfisher@REDACTED (Paul Fisher) Date: Sat, 07 Jun 2008 11:29:12 -0500 Subject: [erlang-questions] [clarify] Limits and overhead of monitors In-Reply-To: References: Message-ID: <1212856152.7281.145.camel@localhost> On Sat, 2008-06-07 at 09:06 -0700, Jay Nelson wrote: > Has anyone determined the answer to any of the following questions: > > 1) Is a practical limit to the number of monitors a single process > can hold? > - I assume they are fairly static, so just the memory to store > them? > - how are they stored? > - any problems if I have 100K processes in an everyone > monitoring everyone configuration? Do you really mean that each process would monitor the other 99,999 processes? That would be 9,999,900,000 monitor relationships, so at the very least you've gotta have a lot of memory on those systems. -- paul From chsu79@REDACTED Sat Jun 7 18:57:09 2008 From: chsu79@REDACTED (Christian S) Date: Sat, 7 Jun 2008 18:57:09 +0200 Subject: [erlang-questions] Mocking in Erlang In-Reply-To: References: <69d143cd0805231746p7f8ed4cdl91e0f6fdd7da8025@mail.gmail.com> <34687.217.128.75.198.1211802071.squirrel@www.geekisp.com> <69d143cd0805261846g46973594gbf58798c4b256330@mail.gmail.com> Message-ID: > True. And something I would like even better is if I could generate > that module in a few lines close to the eunit test function. (Cant be > impossible to create a module directly from a list of atoms and funs. > Doesnt need to be fast.) > > A downside is that the module namespace is node-wide, so you cant > safely run your tests in parallell on the same node. This is a reason > I consider this to be a very dirty way to slip in the mock. To follow up on myself. I hacked up something that almost fulfills my wishes: You load this module that follows the inteface of an error_handler: -module(mock). -compile(export_all). undefined_function(Mod, F, Args) -> case get(Mod) of undefined -> exit({undefined, Mod, F, Args}); MockDict -> Fun = dict:fetch(F, MockDict), apply(Fun, Args) end. undefined_lambda(_Mod, _Fun, _Args) -> exit(undef). Second, I put this in my process to install the above error_handler for this process: init() -> Mock = dict:from_list([{eat, fun() -> yum end}]), put(banana, Mock), process_flag(error_handler, mock), loop(). Whenever this process calls banana:eat(), it will get the atom 'yum' back. Completly process local. YES! It is dirty! The prettiest flowers grow in manure! PS. http://paste.lisp.org/display/61885 has more of the code and experiment, at least until it goes away in a couple of hours. From colm.dougan@REDACTED Sat Jun 7 22:04:32 2008 From: colm.dougan@REDACTED (Colm Dougan) Date: Sat, 7 Jun 2008 21:04:32 +0100 Subject: [erlang-questions] erlang module in apache cgi script not working In-Reply-To: <88f71dce0806010347h6c337854ic62a38191d50e5f3@mail.gmail.com> References: <88f71dce0806010347h6c337854ic62a38191d50e5f3@mail.gmail.com> Message-ID: <24d4f39c0806071304p5668de71qdf4bfca866b32cc1@mail.gmail.com> On Sun, Jun 1, 2008 at 11:47 AM, Jayakrishnan M wrote: > Hi, > > I am using Apache httpd 2 and have a shell script as a cgi script. I > am trying to call an > erlang module from the shell script for some processing. > The problem is that the erlang code is not getting executed. > The lines in the shell script which are above and below the call to > erlang binary are working. > I have been trying to figure out the problem for a long time, but > without any success. Without knowing more details, one possibility is that your shell scripts runtime does not have the required HOME environment variable set. If you haven't already set HOME, you could try manually setting it in your shell (CGI) script. Colm From rvirding@REDACTED Sat Jun 7 22:37:17 2008 From: rvirding@REDACTED (Robert Virding) Date: Sat, 7 Jun 2008 22:37:17 +0200 Subject: [erlang-questions] Mocking in Erlang In-Reply-To: References: <69d143cd0805231746p7f8ed4cdl91e0f6fdd7da8025@mail.gmail.com> <34687.217.128.75.198.1211802071.squirrel@www.geekisp.com> <69d143cd0805261846g46973594gbf58798c4b256330@mail.gmail.com> Message-ID: <3dbc6d1c0806071337y7a2049e2k4062fd79aeb0af91@mail.gmail.com> Actually this is not really a dirty hack. The reason why you can define your own error handler is to allow customising error handling in an application. We provided all the tools necessary to give you a wide range of options when shooting your own foot. :-) Seriously this was not the case, we wanted to provide tools to build systems and not packaged solutions. Robert 2008/6/7 Christian S : > > True. And something I would like even better is if I could generate > > that module in a few lines close to the eunit test function. (Cant be > > impossible to create a module directly from a list of atoms and funs. > > Doesnt need to be fast.) > > > > A downside is that the module namespace is node-wide, so you cant > > safely run your tests in parallell on the same node. This is a reason > > I consider this to be a very dirty way to slip in the mock. > > To follow up on myself. I hacked up something that almost fulfills my > wishes: > > You load this module that follows the inteface of an error_handler: > > -module(mock). > > -compile(export_all). > > undefined_function(Mod, F, Args) -> > case get(Mod) of > undefined -> > exit({undefined, Mod, F, Args}); > MockDict -> > Fun = dict:fetch(F, MockDict), > apply(Fun, Args) > end. > > undefined_lambda(_Mod, _Fun, _Args) -> > exit(undef). > > > Second, I put this in my process to install the above error_handler > for this process: > > init() -> > Mock = dict:from_list([{eat, fun() -> yum end}]), > put(banana, Mock), > process_flag(error_handler, mock), > loop(). > > > Whenever this process calls banana:eat(), it will get the atom 'yum' > back. Completly process local. > > > > YES! It is dirty! The prettiest flowers grow in manure! > > > PS. > http://paste.lisp.org/display/61885 has more of the code and > experiment, at least until it goes away in a couple of hours. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From japerk@REDACTED Sun Jun 8 04:19:50 2008 From: japerk@REDACTED (Jacob Perkins) Date: Sat, 7 Jun 2008 19:19:50 -0700 Subject: [erlang-questions] dynamic module compilation Message-ID: Hi all, I need some help understanding how to dynamically compile/generate an erlang module. I've seen other projects do this kind of thing, such as erlydtl, but my needs are much simpler. I have a few files that are basically lists of words. What I want to do is generate a module whose functions will return each list of words. So if I have two files named "adjectives" and "nouns", then the generated module (let's call it 'grammar') should have two functions, adjectives() and nouns(), that return their respective list of words. How can I do this? Thanks, Jacob -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay@REDACTED Sun Jun 8 06:15:51 2008 From: jay@REDACTED (Jay Nelson) Date: Sat, 7 Jun 2008 21:15:51 -0700 Subject: [erlang-questions] [clarify] Limits and overhead of monitors In-Reply-To: <1212856152.7281.145.camel@localhost> References: <1212856152.7281.145.camel@localhost> Message-ID: <833CF181-EF8D-4C42-B34F-B6175556903F@duomark.com> > Do you really mean that each process would monitor the other 99,999 > processes? That would be 9,999,900,000 monitor relationships, so > at the > very least you've gotta have a lot of memory on those systems. I got carried away as I was writing the email and added this without thinking. No only that, but if one process goes down I will get 99,999 messages. I think I need to come up with a partitioning scheme or a hierarchy of processes which have a single notifier watching them so that only one 'DOWN' message is signalled and then I can cascade it as needed. So, I'm back to my original question of how many monitors can a single process hold. I'll post the results of any benchmarking I do. jay From vychodil.hynek@REDACTED Sun Jun 8 10:25:00 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Sun, 8 Jun 2008 10:25:00 +0200 Subject: [erlang-questions] Mocking in Erlang In-Reply-To: References: <69d143cd0805231746p7f8ed4cdl91e0f6fdd7da8025@mail.gmail.com> <34687.217.128.75.198.1211802071.squirrel@www.geekisp.com> <69d143cd0805261846g46973594gbf58798c4b256330@mail.gmail.com> Message-ID: <4d08db370806080125s1a593df0qc41e59a112805dad@mail.gmail.com> Nice and elegant idea, but it is not solution for running non mocked and mocked code. When you run some code calling normal banana:eat, you can't run in same time process with mocked version because error_handler will not be raced. So for carefully wrote test it can be enough, but very fragile. You must disallow loading normal banana module by using embedded version of global error_handler. On Sat, Jun 7, 2008 at 6:57 PM, Christian S wrote: > > True. And something I would like even better is if I could generate > > that module in a few lines close to the eunit test function. (Cant be > > impossible to create a module directly from a list of atoms and funs. > > Doesnt need to be fast.) > > > > A downside is that the module namespace is node-wide, so you cant > > safely run your tests in parallell on the same node. This is a reason > > I consider this to be a very dirty way to slip in the mock. > > To follow up on myself. I hacked up something that almost fulfills my > wishes: > > You load this module that follows the inteface of an error_handler: > > -module(mock). > > -compile(export_all). > > undefined_function(Mod, F, Args) -> > case get(Mod) of > undefined -> > exit({undefined, Mod, F, Args}); > MockDict -> > Fun = dict:fetch(F, MockDict), > apply(Fun, Args) > end. > > undefined_lambda(_Mod, _Fun, _Args) -> > exit(undef). > > > Second, I put this in my process to install the above error_handler > for this process: > > init() -> > Mock = dict:from_list([{eat, fun() -> yum end}]), > put(banana, Mock), > process_flag(error_handler, mock), > loop(). > > > Whenever this process calls banana:eat(), it will get the atom 'yum' > back. Completly process local. > > > > YES! It is dirty! The prettiest flowers grow in manure! > > > PS. > http://paste.lisp.org/display/61885 has more of the code and > experiment, at least until it goes away in a couple of hours. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Sun Jun 8 11:01:13 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Sun, 8 Jun 2008 11:01:13 +0200 Subject: [erlang-questions] dynamic module compilation In-Reply-To: References: Message-ID: <4d08db370806080201i704b69c9m42bc7264cde36f1a@mail.gmail.com> compile:file or compile:forms with erl_parse and erl_scan than use code:purge and code:load or code:load_file But in your use case I think there could be better solution with ets from file or dets. 2008/6/8 Jacob Perkins : > Hi all, > I need some help understanding how to dynamically compile/generate an > erlang module. I've seen other projects do this kind of thing, such as > erlydtl, but my needs are much simpler. > > I have a few files that are basically lists of words. What I want to do is > generate a module whose functions will return each list of words. So if I > have two files named "adjectives" and "nouns", then the generated module > (let's call it 'grammar') should have two functions, adjectives() and > nouns(), that return their respective list of words. > > How can I do this? > > Thanks, > Jacob > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Sun Jun 8 11:13:29 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Sun, 8 Jun 2008 11:13:29 +0200 Subject: [erlang-questions] [clarify] Limits and overhead of monitors In-Reply-To: <833CF181-EF8D-4C42-B34F-B6175556903F@duomark.com> References: <1212856152.7281.145.camel@localhost> <833CF181-EF8D-4C42-B34F-B6175556903F@duomark.com> Message-ID: <4d08db370806080213mcf0c8a6wc4037666805aa8d@mail.gmail.com> No, no, your question was generally correct. It depend how monitor is implemented. Sure that tree organisation of controllers and partitioning is better solution for complexity. But how monitors work is good question anyway. I think it is one process per node with ets and linked to monitored processes, but I'm not sure. So anyway in your example it will be 9,999,900,000 records or 100,000 records with 99,999 list items each. If I remember there is timers described same way in this email list and I guess monitors can be implemented similar. On Sun, Jun 8, 2008 at 6:15 AM, Jay Nelson wrote: > > Do you really mean that each process would monitor the other 99,999 > > processes? That would be 9,999,900,000 monitor relationships, so > > at the > > very least you've gotta have a lot of memory on those systems. > > I got carried away as I was writing the email and added this without > thinking. > No only that, but if one process goes down I will get 99,999 messages. > > I think I need to come up with a partitioning scheme or a hierarchy of > processes which have a single notifier watching them so that only one > 'DOWN' message is signalled and then I can cascade it as needed. > > So, I'm back to my original question of how many monitors can a single > process hold. I'll post the results of any benchmarking I do. > > jay > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Sun Jun 8 12:03:20 2008 From: rvirding@REDACTED (Robert Virding) Date: Sun, 8 Jun 2008 12:03:20 +0200 Subject: [erlang-questions] Mocking in Erlang In-Reply-To: <4d08db370806080125s1a593df0qc41e59a112805dad@mail.gmail.com> References: <69d143cd0805231746p7f8ed4cdl91e0f6fdd7da8025@mail.gmail.com> <34687.217.128.75.198.1211802071.squirrel@www.geekisp.com> <69d143cd0805261846g46973594gbf58798c4b256330@mail.gmail.com> <4d08db370806080125s1a593df0qc41e59a112805dad@mail.gmail.com> Message-ID: <3dbc6d1c0806080303n19721a51mfcdb24b662ab12f2@mail.gmail.com> It is definitely a very powerful tool which should be used carefully. However, as this works on a per process basis you can use it in an app by using processes to split on which error handling method to use. Processes rule. My comment was really that it is not a dirty hack, but a valid way of doing things. Robert 2008/6/8 Hynek Vychodil : > Nice and elegant idea, but it is not solution for running non mocked and > mocked code. When you run some code calling normal banana:eat, you can't run > in same time process with mocked version because error_handler will not be > raced. So for carefully wrote test it can be enough, but very fragile. You > must disallow loading normal banana module by using embedded version of > global error_handler. > > > On Sat, Jun 7, 2008 at 6:57 PM, Christian S wrote: > >> > True. And something I would like even better is if I could generate >> > that module in a few lines close to the eunit test function. (Cant be >> > impossible to create a module directly from a list of atoms and funs. >> > Doesnt need to be fast.) >> > >> > A downside is that the module namespace is node-wide, so you cant >> > safely run your tests in parallell on the same node. This is a reason >> > I consider this to be a very dirty way to slip in the mock. >> >> To follow up on myself. I hacked up something that almost fulfills my >> wishes: >> >> You load this module that follows the inteface of an error_handler: >> >> -module(mock). >> >> -compile(export_all). >> >> undefined_function(Mod, F, Args) -> >> case get(Mod) of >> undefined -> >> exit({undefined, Mod, F, Args}); >> MockDict -> >> Fun = dict:fetch(F, MockDict), >> apply(Fun, Args) >> end. >> >> undefined_lambda(_Mod, _Fun, _Args) -> >> exit(undef). >> >> >> Second, I put this in my process to install the above error_handler >> for this process: >> >> init() -> >> Mock = dict:from_list([{eat, fun() -> yum end}]), >> put(banana, Mock), >> process_flag(error_handler, mock), >> loop(). >> >> >> Whenever this process calls banana:eat(), it will get the atom 'yum' >> back. Completly process local. >> >> >> >> YES! It is dirty! The prettiest flowers grow in manure! >> >> >> PS. >> http://paste.lisp.org/display/61885 has more of the code and >> experiment, at least until it goes away in a couple of hours. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > --Hynek (Pichi) Vychodil > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Sun Jun 8 12:10:07 2008 From: rvirding@REDACTED (Robert Virding) Date: Sun, 8 Jun 2008 12:10:07 +0200 Subject: [erlang-questions] dynamic module compilation In-Reply-To: References: Message-ID: <3dbc6d1c0806080310n15166488p4d34d925f10ab731@mail.gmail.com> There are basically two ways of doing this: - From you list of files generate a file, grammar.erl, containing the module grammar with functions adjectives/0 and nouns/0. You then compile with compile:file and load with code:load. - You can generate the code as AST in memory and then compile with compile:forms. This is a bit tricky the first time, but easy once you get the hang of it. - As someone else mentioned save them in a database on disk. Robert 2008/6/8 Jacob Perkins : > Hi all, > I need some help understanding how to dynamically compile/generate an > erlang module. I've seen other projects do this kind of thing, such as > erlydtl, but my needs are much simpler. > > I have a few files that are basically lists of words. What I want to do is > generate a module whose functions will return each list of words. So if I > have two files named "adjectives" and "nouns", then the generated module > (let's call it 'grammar') should have two functions, adjectives() and > nouns(), that return their respective list of words. > > How can I do this? > > Thanks, > Jacob > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hayeah@REDACTED Sun Jun 8 18:45:33 2008 From: hayeah@REDACTED (Howard Yeh) Date: Sun, 8 Jun 2008 09:45:33 -0700 Subject: [erlang-questions] dynamic module compilation In-Reply-To: <3dbc6d1c0806080310n15166488p4d34d925f10ab731@mail.gmail.com> References: <3dbc6d1c0806080310n15166488p4d34d925f10ab731@mail.gmail.com> Message-ID: see ERTS manual chapter 4 to see the abstract format. http://www.erlang.org/doc/apps/erts/part_frame.html there's also Core Erlang. But I am not sure where it's documented. On 6/8/08, Robert Virding wrote: > There are basically two ways of doing this: > > - From you list of files generate a file, grammar.erl, containing the module > grammar with functions adjectives/0 and nouns/0. You then compile with > compile:file and load with code:load. > > - You can generate the code as AST in memory and then compile with > compile:forms. This is a bit tricky the first time, but easy once you get > the hang of it. > > - As someone else mentioned save them in a database on disk. > > Robert > > 2008/6/8 Jacob Perkins : > > Hi all, > > > > > > > > I need some help understanding how to dynamically compile/generate an > erlang module. I've seen other projects do this kind of thing, such as > erlydtl, but my needs are much simpler. > > > > > > I have a few files that are basically lists of words. What I want to do is > generate a module whose functions will return each list of words. So if I > have two files named "adjectives" and "nouns", then the generated module > (let's call it 'grammar') should have two functions, adjectives() and > nouns(), that return their respective list of words. > > > > > > How can I do this? > > > > > > Thanks, > > Jacob > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- hayeah.wordpress.com --metacircular thinking From japerk@REDACTED Sun Jun 8 21:12:40 2008 From: japerk@REDACTED (Jacob Perkins) Date: Sun, 8 Jun 2008 12:12:40 -0700 Subject: [erlang-questions] dynamic module compilation In-Reply-To: References: <3dbc6d1c0806080310n15166488p4d34d925f10ab731@mail.gmail.com> Message-ID: So I decided to try generating the AST, and got most of the way there. I don't want to use ets because I don't want to have a process dependency. I was able to use compile:forms to generate a module and function, but my function clause isn't correct. The function and clause forms I can get to compile are: {function, LINE, strings, 0, [{clause, LINE, [], [], [{string, LINE, "foo"}, {string, LINE, "bar"}]}]} But the above form appears to generate strings() -> "foo", "bar". when what I really want is strings() -> ["foo", "bar"]. It's probably a simple fix in the forms, but the obvious change of enclosing the string literals in another list causes a compile error: {function, LINE, strings, 0, [{clause, LINE, [], [], [[{string, LINE, "foo"}, {string, LINE, "bar"}]]}]} Any help? Thanks, Jacob On Sun, Jun 8, 2008 at 9:45 AM, Howard Yeh wrote: > see ERTS manual chapter 4 to see the abstract format. > > http://www.erlang.org/doc/apps/erts/part_frame.html > > there's also Core Erlang. But I am not sure where it's documented. > > > On 6/8/08, Robert Virding wrote: > > There are basically two ways of doing this: > > > > - From you list of files generate a file, grammar.erl, containing the > module > > grammar with functions adjectives/0 and nouns/0. You then compile with > > compile:file and load with code:load. > > > > - You can generate the code as AST in memory and then compile with > > compile:forms. This is a bit tricky the first time, but easy once you get > > the hang of it. > > > > - As someone else mentioned save them in a database on disk. > > > > Robert > > > > 2008/6/8 Jacob Perkins : > > > Hi all, > > > > > > > > > > > > I need some help understanding how to dynamically compile/generate an > > erlang module. I've seen other projects do this kind of thing, such as > > erlydtl, but my needs are much simpler. > > > > > > > > > I have a few files that are basically lists of words. What I want to do > is > > generate a module whose functions will return each list of words. So if I > > have two files named "adjectives" and "nouns", then the generated module > > (let's call it 'grammar') should have two functions, adjectives() and > > nouns(), that return their respective list of words. > > > > > > > > > How can I do this? > > > > > > > > > Thanks, > > > Jacob > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > > > -- > hayeah.wordpress.com > --metacircular thinking > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Sun Jun 8 21:38:16 2008 From: ulf@REDACTED (Ulf Wiger) Date: Sun, 8 Jun 2008 21:38:16 +0200 Subject: [erlang-questions] dynamic module compilation In-Reply-To: References: <3dbc6d1c0806080310n15166488p4d34d925f10ab731@mail.gmail.com> Message-ID: <8209f740806081238l80d48cdo51d636375bd82b5d@mail.gmail.com> There are a few tricks you can use to quickly find out what abstract form a certain expression corresponds to: 1> erl_parse:abstract(["foo","bar"]). {cons,0,{string,0,"foo"},{cons,0,{string,0,"bar"},{nil,0}}} Another trick is to write a dummy module, then compile it with the option 'debug_info'. You can then read the debug info and view the abstract code that resulted from the source in your module. The command for reading debug info is beam_lib:chunks(BeamFile, [abstract_code]). BR, Ulf W 2008/6/8 Jacob Perkins : > So I decided to try generating the AST, and got most of the way there. I > don't want to use ets because I don't want to have a process dependency. > I was able to use compile:forms to generate a module and function, but my > function clause isn't correct. The function and clause forms I can get to > compile are: > {function, LINE, strings, 0, > [{clause, LINE, [], [], > [{string, LINE, "foo"}, {string, LINE, "bar"}]}]} > But the above form appears to generate > strings() -> "foo", "bar". > when what I really want is > strings() -> ["foo", "bar"]. > It's probably a simple fix in the forms, but the obvious change of enclosing > the string literals in another list causes a compile error: > {function, LINE, strings, 0, > [{clause, LINE, [], [], > [[{string, LINE, "foo"}, {string, LINE, "bar"}]]}]} > Any help? > Thanks, > Jacob > On Sun, Jun 8, 2008 at 9:45 AM, Howard Yeh wrote: >> >> see ERTS manual chapter 4 to see the abstract format. >> >> http://www.erlang.org/doc/apps/erts/part_frame.html >> >> there's also Core Erlang. But I am not sure where it's documented. >> >> >> On 6/8/08, Robert Virding wrote: >> > There are basically two ways of doing this: >> > >> > - From you list of files generate a file, grammar.erl, containing the >> > module >> > grammar with functions adjectives/0 and nouns/0. You then compile with >> > compile:file and load with code:load. >> > >> > - You can generate the code as AST in memory and then compile with >> > compile:forms. This is a bit tricky the first time, but easy once you >> > get >> > the hang of it. >> > >> > - As someone else mentioned save them in a database on disk. >> > >> > Robert >> > >> > 2008/6/8 Jacob Perkins : >> > > Hi all, >> > > >> > > >> > > >> > > I need some help understanding how to dynamically compile/generate an >> > erlang module. I've seen other projects do this kind of thing, such as >> > erlydtl, but my needs are much simpler. >> > > >> > > >> > > I have a few files that are basically lists of words. What I want to >> > > do is >> > generate a module whose functions will return each list of words. So if >> > I >> > have two files named "adjectives" and "nouns", then the generated module >> > (let's call it 'grammar') should have two functions, adjectives() and >> > nouns(), that return their respective list of words. >> > > >> > > >> > > How can I do this? >> > > >> > > >> > > Thanks, >> > > Jacob >> > > _______________________________________________ >> > > erlang-questions mailing list >> > > erlang-questions@REDACTED >> > > http://www.erlang.org/mailman/listinfo/erlang-questions >> > > >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://www.erlang.org/mailman/listinfo/erlang-questions >> > >> >> >> -- >> hayeah.wordpress.com >> --metacircular thinking > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From vychodil.hynek@REDACTED Sun Jun 8 21:40:54 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Sun, 8 Jun 2008 21:40:54 +0200 Subject: [erlang-questions] dynamic module compilation In-Reply-To: References: <3dbc6d1c0806080310n15166488p4d34d925f10ab731@mail.gmail.com> Message-ID: <4d08db370806081240k2e17ca25pa433c15328ff7ed1@mail.gmail.com> List constructor is AST node too. 1> erl_parse:parse(element(2, erl_scan:string("strings() -> [\"foo\", \"bar\"]."))). {ok,{function,1,strings,0, [{clause,1,[],[], [{cons,1, {string,1,"foo"}, {cons,1,{string,1,"bar"},{nil,1}}}]}]}} Anyway, ets is better solution for you, it will be faster for most of your tasks. It is easier and you don't need learn AST :-) 2008/6/8 Jacob Perkins : > So I decided to try generating the AST, and got most of the way there. I > don't want to use ets because I don't want to have a process dependency. > I was able to use compile:forms to generate a module and function, but my > function clause isn't correct. The function and clause forms I can get to > compile are: > > {function, LINE, strings, 0, > [{clause, LINE, [], [], > [{string, LINE, "foo"}, {string, LINE, "bar"}]}]} > > But the above form appears to generate > > strings() -> "foo", "bar". > > when what I really want is > > strings() -> ["foo", "bar"]. > > It's probably a simple fix in the forms, but the obvious change of > enclosing the string literals in another list causes a compile error: > > {function, LINE, strings, 0, > [{clause, LINE, [], [], > [[{string, LINE, "foo"}, {string, LINE, "bar"}]]}]} > > Any help? > Thanks, > Jacob > > On Sun, Jun 8, 2008 at 9:45 AM, Howard Yeh wrote: > >> see ERTS manual chapter 4 to see the abstract format. >> >> http://www.erlang.org/doc/apps/erts/part_frame.html >> >> there's also Core Erlang. But I am not sure where it's documented. >> >> >> On 6/8/08, Robert Virding wrote: >> > There are basically two ways of doing this: >> > >> > - From you list of files generate a file, grammar.erl, containing the >> module >> > grammar with functions adjectives/0 and nouns/0. You then compile with >> > compile:file and load with code:load. >> > >> > - You can generate the code as AST in memory and then compile with >> > compile:forms. This is a bit tricky the first time, but easy once you >> get >> > the hang of it. >> > >> > - As someone else mentioned save them in a database on disk. >> > >> > Robert >> > >> > 2008/6/8 Jacob Perkins : >> > > Hi all, >> > > >> > > >> > > >> > > I need some help understanding how to dynamically compile/generate an >> > erlang module. I've seen other projects do this kind of thing, such as >> > erlydtl, but my needs are much simpler. >> > > >> > > >> > > I have a few files that are basically lists of words. What I want to >> do is >> > generate a module whose functions will return each list of words. So if >> I >> > have two files named "adjectives" and "nouns", then the generated module >> > (let's call it 'grammar') should have two functions, adjectives() and >> > nouns(), that return their respective list of words. >> > > >> > > >> > > How can I do this? >> > > >> > > >> > > Thanks, >> > > Jacob >> > > _______________________________________________ >> > > erlang-questions mailing list >> > > erlang-questions@REDACTED >> > > http://www.erlang.org/mailman/listinfo/erlang-questions >> > > >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://www.erlang.org/mailman/listinfo/erlang-questions >> > >> >> >> -- >> hayeah.wordpress.com >> --metacircular thinking >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From japerk@REDACTED Sun Jun 8 22:22:06 2008 From: japerk@REDACTED (Jacob Perkins) Date: Sun, 8 Jun 2008 13:22:06 -0700 Subject: [erlang-questions] dynamic module compilation In-Reply-To: <8209f740806081238l80d48cdo51d636375bd82b5d@mail.gmail.com> References: <3dbc6d1c0806080310n15166488p4d34d925f10ab731@mail.gmail.com> <8209f740806081238l80d48cdo51d636375bd82b5d@mail.gmail.com> Message-ID: Thanks for the help everyone, I got things working using erl_parse:abstract. Final code turned out pretty simple: Strings = ["foo", "bar"], Clause = {clause, LINE, [], [], [erl_parse:abstract(Strings)]}, {function, LINE, strings, 0, [Clause]}. On Sun, Jun 8, 2008 at 12:38 PM, Ulf Wiger wrote: > There are a few tricks you can use to quickly find out what > abstract form a certain expression corresponds to: > > 1> erl_parse:abstract(["foo","bar"]). > {cons,0,{string,0,"foo"},{cons,0,{string,0,"bar"},{nil,0}}} > > Another trick is to write a dummy module, then compile > it with the option 'debug_info'. You can then read the > debug info and view the abstract code that resulted from > the source in your module. The command for reading > debug info is > > beam_lib:chunks(BeamFile, [abstract_code]). > > BR, > Ulf W > > 2008/6/8 Jacob Perkins : > > So I decided to try generating the AST, and got most of the way there. I > > don't want to use ets because I don't want to have a process dependency. > > I was able to use compile:forms to generate a module and function, but my > > function clause isn't correct. The function and clause forms I can get to > > compile are: > > {function, LINE, strings, 0, > > [{clause, LINE, [], [], > > [{string, LINE, "foo"}, {string, LINE, "bar"}]}]} > > But the above form appears to generate > > strings() -> "foo", "bar". > > when what I really want is > > strings() -> ["foo", "bar"]. > > It's probably a simple fix in the forms, but the obvious change of > enclosing > > the string literals in another list causes a compile error: > > {function, LINE, strings, 0, > > [{clause, LINE, [], [], > > [[{string, LINE, "foo"}, {string, LINE, "bar"}]]}]} > > Any help? > > Thanks, > > Jacob > > On Sun, Jun 8, 2008 at 9:45 AM, Howard Yeh wrote: > >> > >> see ERTS manual chapter 4 to see the abstract format. > >> > >> http://www.erlang.org/doc/apps/erts/part_frame.html > >> > >> there's also Core Erlang. But I am not sure where it's documented. > >> > >> > >> On 6/8/08, Robert Virding wrote: > >> > There are basically two ways of doing this: > >> > > >> > - From you list of files generate a file, grammar.erl, containing the > >> > module > >> > grammar with functions adjectives/0 and nouns/0. You then compile with > >> > compile:file and load with code:load. > >> > > >> > - You can generate the code as AST in memory and then compile with > >> > compile:forms. This is a bit tricky the first time, but easy once you > >> > get > >> > the hang of it. > >> > > >> > - As someone else mentioned save them in a database on disk. > >> > > >> > Robert > >> > > >> > 2008/6/8 Jacob Perkins : > >> > > Hi all, > >> > > > >> > > > >> > > > >> > > I need some help understanding how to dynamically compile/generate > an > >> > erlang module. I've seen other projects do this kind of thing, such as > >> > erlydtl, but my needs are much simpler. > >> > > > >> > > > >> > > I have a few files that are basically lists of words. What I want to > >> > > do is > >> > generate a module whose functions will return each list of words. So > if > >> > I > >> > have two files named "adjectives" and "nouns", then the generated > module > >> > (let's call it 'grammar') should have two functions, adjectives() and > >> > nouns(), that return their respective list of words. > >> > > > >> > > > >> > > How can I do this? > >> > > > >> > > > >> > > Thanks, > >> > > Jacob > >> > > _______________________________________________ > >> > > erlang-questions mailing list > >> > > erlang-questions@REDACTED > >> > > http://www.erlang.org/mailman/listinfo/erlang-questions > >> > > > >> > > >> > > >> > _______________________________________________ > >> > erlang-questions mailing list > >> > erlang-questions@REDACTED > >> > http://www.erlang.org/mailman/listinfo/erlang-questions > >> > > >> > >> > >> -- > >> hayeah.wordpress.com > >> --metacircular thinking > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay@REDACTED Sun Jun 8 23:11:44 2008 From: jay@REDACTED (Jay Nelson) Date: Sun, 8 Jun 2008 14:11:44 -0700 Subject: [erlang-questions] Message Receive Semantics (was eep:MultiplePatterns) Message-ID: <22A82CA1-E32A-4637-86C3-6AF207BC820A@duomark.com> Valentin wrote: > Back in 2000, when I just started with Erlang > Ok, you've got 2 years on me so your insights may be more grounded than mine. > (had a extensive commercial experience in procedural languages, > though), one of the first problems I tried to solve had to do with > priority handling. So I wrote something very ugly, but someting > that reflected a paridgm I had at the time -- I remember writing > something like this: > > loop_high( 0 ) -> loop_normal( 3 ); > loop_high( N ) -> > receive > {high, Msg} -> > process_msg( Msg ), > loop_high( N-1 ) > after 0 -> receive > {normal, Msg}-> > process_msg( Msg ), > loop_high( N ) > This is the first revelation that a beginner needs to understand. It sometimes takes a while to realize that the after clause of a receive can be another receive (and that the semantics of that are different than two successive receives). > loop_low( 0 ) -> loop_high( 5 ); > loop_low( N ) -> ... > ... > ... > The next revelation is that this is polling. You will consume CPU time when there are no messages available. This may or may not matter to your application. As you say below, it feels grungy and it doesn't look like a very erlangish approach. > > The above code would do the trick, however, being ugly as it may -- > would eventually trigger refactoring to the tune of: > > loop_high( 0 ) -> loop_normal( 3 ); > loop_hihg(N) -> > receive > {high, Msg} -> > process_msg( Msg ), > loop_high( N-1 ); > {normal, Msg} -> > process_msg( Msg ), > loop_high( N ); > {low, Msg} -> > process_msg( Msg ), > loop_high(N); > ANY -> > case process_msg( ANY ) of > stop -> exit( normal ); > _ -> loop_high( N ) > end > after 1000 -> loop_high( N ) > end. > Are you sure the above works? Suppose you have a Low message followed by a High message. Won't the Low message get handled first? The same applies to the other clauses. I also don't see the purpose of the 1000 timeout above, but again you end up with a polling. > > What I'm trying to say, I guess -- it does not really matter if one > is novice or not. We all know how to think, and only through > thinking we can arrive to a point where we could articulate the > "right" kind of questions. Nothing wrong in making a few mistakes > in a process. > I may have mis-stated my objection by phrasing it as a challenge to beginners. I conflated two issues: 1) advanced message handling requires one to become fluent and familiar with simpler message handling first, and 2) priority messaging is a pathological case for erlang's sequential mailbox approach. #1 is true of any new feature of any new language, however, I would bet the majority of new programmers to erlang have never used another language that has as simple a mechanism for message handling, and therefore are likely to be unfamiliar with the various options that receive enables. The same would not be true of lists or hash tables. #2 is a result of the sequential, and wisely chosen simple nature, of the receive mechanism. I don't believe it is possible for a single process to implement priority without polling, unless the message queue is unloaded and maintained using standard data structures. That's what I mean by pathological -- normal use of the receive statement leads to bad results and unorthodox means to handle things efficiently. Although the choice that the language designers chose makes a lot of other cases very easy to implement and very easy to change your architecture. Overall it is a win, but always be aware of the worst case scenario so you can avoid it. (My general recommendation on priority messaging is to avoid it by changing the architecture. Often times it is just a side effect of the previous techniques for handling messages. The lightweightedness of erlang processes allows alternative approaches to standard priorities.) Suppose the requirements change to add one more level of priority. It ripples through all the receive statements. Not a desirable outcome. > > BTW, how easy one may make a mistake depends largely on how the > problem was stated. If one says: > > > > 3) Always handle messages in the following order: > a) 5 messages of {reply, high, Msg} > b) 3 messages of {reply, normal, Msg} > c) 1 message of {reply, low, Msg} > d) 'EXIT' message > > > > one should not get surprised if the resulting code blocks until 5 > high priority messages have been processed. However, if one > indicates that preference should be given to high priority > messages, than normal, followed by low priority message, utilising > a ratio of 5:3:1, and stipulate that should there no higher > priority message, lower one should be processed -- one may close a > semantic gap, and thus prevent a novice from making a mistake. All > of that in far better English, of course ;-). > > Well, I did botch the explanation, but it seemed a rather arbitrary requirement to start with. Did you really mean 5:3:1 or was that an easy way to approximate priority because you can't easily do it with a normal receive? Wouldn't you really want to process as many High priority messages as quickly as possible, and then normal and low as you get a chance, maybe in parallel? Why not let the erlang process time-slicing manage it for you, using the erlang process priority to give preference to high, normal and low messages in that order? Since there is one message queue and one process, you can't use any of the built in process time-sharing or parallelism. Even if you unload the mailbox and put items in a priority queue, you will still have one process to handle them. The only way that relies on the VM semantics of fair scheduling is to use a router process to receive items and to forward them to one of 3 different processes for handling. Using that approach also allows the flexibility to add new priorities easily (one new pattern and one new process), although it is still a little tricky to manage the balance of high vs normal vs low if you don't just want the VM to treat them equally -- there are two choices: 1) use process priorities and 2) send a series of messages followed by a request to ack them so that the router controls the flow by waiting for acks before adding messages to the process queues. Doing it with a router process is the most flexible (it allows both selective receive and unloading the mailbox to a priority queue, plus new priority levels have less impact). It is also arguably much easier to understand the resulting code. It is unfortunately a forced solution to simplify the complexity caused by serial queues. jay From jay@REDACTED Sun Jun 8 23:54:12 2008 From: jay@REDACTED (Jay Nelson) Date: Sun, 8 Jun 2008 14:54:12 -0700 Subject: [erlang-questions] Message Receive Semantics (was eep:MultiplePatterns) Message-ID: <91B68D87-ABD0-4487-B66E-5CDD338F6195@duomark.com> David Mercer posted a submission that performed as the requirements stated. In between my post and David's, Valentin pointed out that requirements documents may not be my calling. The solution David posted is a cleanly written separation of the various priorities, and it avoids polling by quitting after the message queue is cleared. Of course, he perfectly satisfied my requirements as restated from Valentin's original. In a real operational situation a few things would be different: 1) New messages can arrive at any time 2) Newly arriving higher priority messages should interrupt a lesser priority loop 3) The 5:3:1 is an artificial substitute for true process priorities To allow new messages, I believe your clean separation has to be violated. This typically occurs by nesting receives as Valentin did (the after clause of high priority contains a receive that includes both high and normal). The 5:3:1 is just a way to say that given a single message queue and a single process, allow some of each message type to get processed to avoid constant priority pre-emption without exhibiting priority inversion (too many low messages when there are higher priority messages). To leverage erlang strengths, use a router on your single queue and a separate process for each of the priorities. This accomplishes the following: 1) One scan of the message queue is possible (although selective receive may still be useful depending on the application). * [Using your approach, there will be one scan per priority level -- suppose we have 8 priorities and the queue has 1000 low priority messages. There will be 7 scans of all the messages to do nothing followed by a scan that processes them all, unless we have to check for a newly arriving higher priority message in which case there will be 999 more unsuccessful scans (x 7 higher priorities) of a queue that decreases in length by 1 each time.] 2) You now have uniform priority queues per process. You can process in order, or use selective receive to thread a single conversation, but you don't have to worry about priority issues. 3) You now have separate processes which can be bumped or reduced in priority, or fed messages quickly or slowly as the router wishes to balance the relative amount of processing. 4) Adding a few new priority levels doesn't ripple through receive statements, nor does it multiply the number of message scans. The new priorities require a few new spawns and pattern to route messages. The real message is: be careful if you find yourself wishing for priority message handling. Look for alternative ways to route messages to accomplish the intended effect. Priority is a hack for single channels to work as if they had more smarts, more horsepower and to simulate multiple channel communications. In a threaded world, I constantly come across architectures with callbacks on events. Eventually, the implementer wants priority because the message queues are too long and they can't get to the messages that need handling immediately. It is a symptom of an architectural issue, and a patch to make it work, rather than reconsidering the whole communications structure. With erlang, it is easy to reorganize the architecture or communications structure without impacting the application logic. It is much harder in other languages, so the lazy way is a quick patch rather than an architectural refactoring. There is a completely separate discussion about how to recognize priority properly (we had this a few months back). In our examples here, the message was magically marked with the correct priority. To do that requires all senders to have global knowledge and logic to set the proper level. A better approach is to mark the messages with the data needed to make a decision, but allow the router process to use its own logic to determine the priorities based on the content provided to it. It enables a simple logic change in the router to affect the throughput of the entire system. If the logic is spread throughout the system, or is globally determined, the impact of priority change is large. jay From jay@REDACTED Mon Jun 9 00:03:52 2008 From: jay@REDACTED (Jay Nelson) Date: Sun, 8 Jun 2008 15:03:52 -0700 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) Message-ID: <2C323C2F-A57D-4BEC-A8C9-125747F5ECDF@duomark.com> There was a suggestion to have multiple channels and a request of whether this is useful. I previously suggested this (and was pre-dated by Ulf suggesting something along those lines as well). Neither of us got much support. Here is the background of my old post: http://www.duomark.com/erlang/proposals/outofband.html In general, I like the idea of channels. The problem comes in the implementation. It would be enough just to have a control channel so that a command to pause or exit could jump in front of a backlog of messages, but a one off approach of a second channel is not a very functional (as in fp) concept. An fp approach would be to have an arbitrary number of channels per message link and to be able to map across them or selectively filter them. The problem in erlang is two-fold: reasonable syntax and reasonable standard performance. Adding this feature could complicate most code by requiring additional syntax, and could also inhibit performance of non-complicated messaging. As ROK indicated, there are ways to use multiple processes to do the same thing. It would be nice if someone came up with an approach like ROKs that solved a useful set of problems, and then encapsulated that in a parse transform that demonstrated how to enhance the syntax to support such an approach. That would give a concrete solution to discuss the merits as a model for a language enhancement. jay From dmorton@REDACTED Thu Jun 5 21:40:11 2008 From: dmorton@REDACTED (Damien Morton) Date: Fri, 06 Jun 2008 05:40:11 +1000 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: References: <4848237A.7090103@bitfurnace.com> Message-ID: <4848411B.4090804@bitfurnace.com> Makes one wonder where the database refactoring tools are :) On 6/6/2008 5:18 AM, Christian S wrote: >> That's true - relational databases have ossified. The original Date book >> on SQL databases, IIRC, seemed to cast them in the light of being a tool >> for executives to quickly and arbitrarily query the company's >> information repository. I cant quite remember what the orginal IBM SQL >> database was called, but Query-By-Example seemed to be an essential element. >> > > My experience with rdbms told me that it is a really bad idea to write > queries that depend on table names and layouts. That is what makes it > so hard to evolve a db. All the applications using it depend on the > current schema. > > So stored procedures are a really good idea. > > > I assume there is a similar lesson to mnesia, put all your transaction > funs for a given mnesia-using application in a single module. > > > From dmorton@REDACTED Fri Jun 6 01:30:24 2008 From: dmorton@REDACTED (Damien Morton) Date: Fri, 06 Jun 2008 09:30:24 +1000 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <2CB2D37B-386B-4078-A83E-D391716A31B2@cs.otago.ac.nz> References: <6e65495c-5a20-446d-8017-3c43ebe55518@x41g2000hsb.googlegroups.com> <54568.1212602052@snookles.snookles.com> <9b08084c0806050252wcda24a8q17cf55cdd8b0dc8@mail.gmail.com> <2CB2D37B-386B-4078-A83E-D391716A31B2@cs.otago.ac.nz> Message-ID: <48487710.4040501@bitfurnace.com> Umm, I think I would prefer it if my tweets didnt persist too long. There's no expectation on the part of the users that they will be retained, so imagine the surprise if one of their users ended up in court only to be presented with a complete history of all their tweets. No, in this case deletion is the right policy. On 6/6/2008 8:55 AM, Richard A. O'Keefe wrote: > On 5 Jun 2008, at 9:52 pm, Joe Armstrong wrote: > >> Appending things to a input queue *and never deleting them* seems to >> me a perfect way to deal with things. >> If you every want to delete things it can only be for pragmatic >> reasons and should be done years later >> in a garbage collection sweep. If you never delete things you can >> always go back later an fix errors! >> > > I note that this is one of the reasons why SAP is successful. > Businesses use SAP because they *can* use SAP; their national tax > departments are happy with SAP. Their national tax departments > are happy with SAP because they can *audit* business data held in > SAP. And they can do *that* because (ta-dah!) SAP never deletes > anything! If there is an incorrect data entry, for example, you > can mark it as incorrect and replace it with a corrected record, > but you cannot remove the old record from the system. > > I note that you can now buy a > Western Digital WD10EACS 1 Terabyte 7200 RPM disc drive for > USD 253 (which is currently NZD 330), price from pixelUSA.com. > Deleting stuff makes a lot less sense than it used to. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > > From paul-trapexit@REDACTED Mon Jun 9 06:44:01 2008 From: paul-trapexit@REDACTED (Paul Mineiro) Date: Sun, 8 Jun 2008 21:44:01 -0700 (PDT) Subject: [erlang-questions] how: prefix matching on external term format In-Reply-To: References: <50549.1212598296@snookles.snookles.com> Message-ID: Hey, I have an initial release on google code: http://code.google.com/p/tcerl/ I'm just now starting on integration with mnesia so there will be additional changes as problems are exposed via that experience. In the interim I'm very interested in feedback from mnesia / dets experts. Thanks, -- p On Wed, 4 Jun 2008, Paul Mineiro wrote: > Oh hey since there's interest, I got this working by forking > erl_compare_ext and adding special tags for smallest and largest erlang > term. My original fear of forking was unfounded since 1) the canonical > erl_compare_ext has several bugs and 2) the canonical erl_compare_ext does > not support ERL_NEW_FLOAT_EXT. That last one was pretty funny since my > main fear of forking was missing out on new type tags. Well, the > directory *does* say "legacy" ... > > I'm in the end game now, within a week I should have something on google > code that sports the same interface as dets but supports ordered_set and > ordered_bag (and 8EB [9.22e18 bytes] byte file size limit). > > -- p > > On Wed, 4 Jun 2008, Scott Lystig Fritchie wrote: > > > Paul Mineiro wrote: > > > > pm> so i've envisioned the following choices: > > > > Hrm, I wonder if it's a viable option to avoid the official Erlang > > external format? If I recall correctly(*), any tuple stored in ETS is > > not serialized using the external term format. Instead, the tuple is > > copied into a separate heap. That heap is big enough to store only the > > tuple. But the data inside that heap uses the same tagged pointer > > scheme as any Erlang term inside a process's heap, so all the term > > comparison macros & functions work, regardless of type (process heap or > > ETS mini-heap). > > > > (*) If I'm wrong, someone please yell loudly at me and the list. I > > wouldn't want the falsehood to live unrefuted forever. > > > > If my memory is correct, then storing an ETS mini-heap onto disk will > > suffer from pointer offset problems: the VM's tagged pointers for use in > > RAM won't point to the necessary byte offset within a file on disk. The > > simplest (?) kludge would be to write an ETS mini-heap with all pointers > > rewritten to be relative to the start of the mini-heap, and then re-set > > those pointers to RAM-valid values after reading from disk but before > > any comparisons. (Simplest = avoid reimplementing term comparison and > > match spec functions.) > > > > -Scott > > > > In an artificial world, only extremists live naturally. > > -- Paul Graham > In an artificial world, only extremists live naturally. -- Paul Graham From dbarker@REDACTED Wed Jun 4 21:57:10 2008 From: dbarker@REDACTED (Deryk Barker) Date: Wed, 4 Jun 2008 12:57:10 -0700 Subject: [erlang-questions] Reia - Python/Ruby mix targeting Beam/HiPE In-Reply-To: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com>; from gleber.p@gmail.com on Wed, Jun 04, 2008 at 09:42:49PM +0200 References: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com> Message-ID: <20080604125710.A6153@hal.cs.camosun.bc.ca> Thus spake Gleb Peregud (gleber.p@REDACTED): > I've stumbled upon this link few moments ago: > > http://reia-lang.org/ > > And thought it will be interesing for fellow erlangers. > > > Reia (pronounced RAY-uh) is a Python/Ruby-like mixed-paradigm language > > targeting the Erlang virtual machine (BEAM) and high-performance > > native compiler (HiPE). Reia aims to expose all the features and functionality > > of Erlang in a language more familiar to programmers of scripting > > languages, while improving string handling, regular expressions, > > linking with external libraries, and other tasks which are generally > > considered outside the scope of Erlang. I was quite interested until I saw that it supports multiple assignments to a variable. -- |Deryk Barker, Computer Science Dept. | Music does not have to be understood| |Camosun College, Victoria, BC, Canada| It has to be listened to. | |email: dbarker@REDACTED | | |phone: +1 250 370 4452 | Hermann Scherchen. | From bertil.karlsson@REDACTED Mon Jun 9 08:15:00 2008 From: bertil.karlsson@REDACTED (Bertil Karlsson) Date: Mon, 09 Jun 2008 08:15:00 +0200 Subject: [erlang-questions] Parsing a DTD file with xmerl In-Reply-To: <1212700706.9015.2.camel@tigerella> References: <1212700706.9015.2.camel@tigerella> Message-ID: <484CCA64.9090501@ericsson.com> Hi, > > Hi everyone. Just a quick question about xmerl - is it possible to > parse a DTD file and receive back something useful? > no > For example, running xmerl_scan:file("CIM_DTD_V22.dtd") gives back the > following error: > > 1698- fatal: {error,{wfc_PEs_In_Internal_Subset}} > ** exited: {fatal,{{error,{wfc_PEs_In_Internal_Subset}}, > {file,"CIM_DTD_V22.dtd"}, > {line,102}, > {col,10}}} ** > > Tracing down through the error a bit it appears that xmerl is > complaining about a parameter-entity reference appearing within a markup > declaration which I don't think is the case here. > > This DTD file is parsed by other XML parsers, for example Python. > > Parsing an xml file and validating it with your DTD should be fine. /Bertl From alexander.lamb@REDACTED Mon Jun 9 10:31:36 2008 From: alexander.lamb@REDACTED (Alexander Lamb) Date: Mon, 9 Jun 2008 10:31:36 +0200 Subject: [erlang-questions] Beginner's question on Erlang compilation Message-ID: <9F3DCCB2-98D6-4521-83B2-4CF645C71456@rodanotech.ch> Hello, Ok, this is probably a stupid question, but somehow I have to ask! When I compile my modules, the compiler doesn't check the existence of functions in other modules. In other words if I do: lists:do_something_non_existent() or my_module:do_something_else(abc) If those functions don't exist, I will only know it when my code runs and crashes. Ok, so I understand what late binding means, but couldn't the compiler at least test the existence of functions with a given arity? I compile using erl -make The makefile is generated by automake (I didn't manage, or want to dive into the traditional makefiles, see post a few weeks ago:-) Thanks for any hints (and please be indulgent, my first "real" programming language was ADA so in terms of compilation, the warnings are really different) Alex -- Alexander Lamb Founding Associate RODANOTECH S?rl 4 ch. de la Tour de Champel 1206 Geneva Switzerland Tel: 022 347 77 37 Fax: 022 347 77 38 http://www.rodanotech.ch From bengt.kleberg@REDACTED Mon Jun 9 11:36:16 2008 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Mon, 09 Jun 2008 11:36:16 +0200 Subject: [erlang-questions] Beginner's question on Erlang compilation In-Reply-To: <9F3DCCB2-98D6-4521-83B2-4CF645C71456@rodanotech.ch> References: <9F3DCCB2-98D6-4521-83B2-4CF645C71456@rodanotech.ch> Message-ID: <1213004176.7209.15.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> Greetings, The compiler does not check functions in other modules. Look at xref (http://www.erlang.org/doc/man/xref.html) if you want to do that. It is a good idea, I have a script that does some xref checks after each compile. bengt On Mon, 2008-06-09 at 10:31 +0200, Alexander Lamb wrote: > Hello, > > Ok, this is probably a stupid question, but somehow I have to ask! > > When I compile my modules, the compiler doesn't check the existence of > functions in other modules. > > In other words if I do: > > lists:do_something_non_existent() > > or > > my_module:do_something_else(abc) > > If those functions don't exist, I will only know it when my code runs > and crashes. > Ok, so I understand what late binding means, but couldn't the compiler > at least test the existence of functions with a given arity? > > I compile using erl -make > > The makefile is generated by automake (I didn't manage, or want to > dive into the traditional makefiles, see post a few weeks ago:-) > > Thanks for any hints (and please be indulgent, my first "real" > programming language was ADA so in terms of compilation, the warnings > are really different) > > Alex > -- > Alexander Lamb > Founding Associate > RODANOTECH S?rl > > 4 ch. de la Tour de Champel > 1206 Geneva > Switzerland > > Tel: 022 347 77 37 > Fax: 022 347 77 38 > > http://www.rodanotech.ch > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From chsu79@REDACTED Mon Jun 9 11:44:50 2008 From: chsu79@REDACTED (Christian S) Date: Mon, 9 Jun 2008 11:44:50 +0200 Subject: [erlang-questions] Beginner's question on Erlang compilation In-Reply-To: <9F3DCCB2-98D6-4521-83B2-4CF645C71456@rodanotech.ch> References: <9F3DCCB2-98D6-4521-83B2-4CF645C71456@rodanotech.ch> Message-ID: There is a tool called dialyzer that does whole program static type analysis. On 6/9/08, Alexander Lamb wrote: > Hello, > > Ok, this is probably a stupid question, but somehow I have to ask! > > When I compile my modules, the compiler doesn't check the existence of > functions in other modules. > > In other words if I do: > > lists:do_something_non_existent() > > or > > my_module:do_something_else(abc) > > If those functions don't exist, I will only know it when my code runs > and crashes. > Ok, so I understand what late binding means, but couldn't the compiler > at least test the existence of functions with a given arity? > > I compile using erl -make > > The makefile is generated by automake (I didn't manage, or want to > dive into the traditional makefiles, see post a few weeks ago:-) > > Thanks for any hints (and please be indulgent, my first "real" > programming language was ADA so in terms of compilation, the warnings > are really different) > > Alex > -- > Alexander Lamb > Founding Associate > RODANOTECH S?rl > > 4 ch. de la Tour de Champel > 1206 Geneva > Switzerland > > Tel: 022 347 77 37 > Fax: 022 347 77 38 > > http://www.rodanotech.ch > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From kostis@REDACTED Mon Jun 9 11:55:21 2008 From: kostis@REDACTED (Kostis Sagonas) Date: Mon, 09 Jun 2008 11:55:21 +0200 Subject: [erlang-questions] Beginner's question on Erlang compilation In-Reply-To: <9F3DCCB2-98D6-4521-83B2-4CF645C71456@rodanotech.ch> References: <9F3DCCB2-98D6-4521-83B2-4CF645C71456@rodanotech.ch> Message-ID: <484CFE09.2000008@cs.ntua.gr> Alexander Lamb wrote: > Hello, > > When I compile my modules, the compiler doesn't check the existence of > functions in other modules. > ... > If those functions don't exist, I will only know it when my code runs > and crashes. You should take a look at and put a call to dialyzer at the end of your make. It performs the checks you want and many more. Nowadays it is part of the Erlang/OTP system and it is really easy to use. Kostis From bjorn@REDACTED Mon Jun 9 15:28:02 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 09 Jun 2008 15:28:02 +0200 Subject: [erlang-questions] Documentation code error In-Reply-To: <0C8B1BBB-5846-4F85-9693-AC4BF0BD8C1C@duomark.com> References: <0C8B1BBB-5846-4F85-9693-AC4BF0BD8C1C@duomark.com> Message-ID: Jay Nelson writes: > On file:////usr/local/lib/erlang/doc/system_principles/ > part_frame.html the target_system code still calls file:rawopen/2 in > the function copy_file/3. It should call file:open/2 and use a list > for the options rather than a tuple. Thanks! It will be changed in R12B-3. /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From klphanikar@REDACTED Mon Jun 9 15:37:41 2008 From: klphanikar@REDACTED (Phanikar.K) Date: Mon, 9 Jun 2008 19:07:41 +0530 Subject: [erlang-questions] Question Message-ID: Hi, I am trying to make a distributed system which uses the same binary files. I have found that with the help of erl_prim_loader we can set an diskless nodes to acess the files from another node which contains all the files. I have tried it but whenever I use -loader inet with all the options like -id -hosts -setcookie it is saying me that {erl_prim_loader,'no server found'} . Can any one please explain me what the error means and explain me the way to acess the files on one node from a diskless nodes.. Thanks, Regards Phanikar. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay@REDACTED Mon Jun 9 15:42:39 2008 From: jay@REDACTED (Jay Nelson) Date: Mon, 9 Jun 2008 06:42:39 -0700 Subject: [erlang-questions] R12B-3 features Message-ID: <62391BE9-32F8-4888-A31B-9A89CFF2BB48@duomark.com> Any chance on either of my prior patches making R12B-3 or a future version? Any comments from the OTP team on either of them? http://www.erlang.org/pipermail/erlang-questions/2008-May/035065.html http://www.erlang.org/pipermail/erlang-questions/2008-May/035066.html jay From camuig@REDACTED Mon Jun 9 15:44:24 2008 From: camuig@REDACTED (Andrey Shnayder) Date: Mon, 9 Jun 2008 17:44:24 +0400 Subject: [erlang-questions] gen_udp and Wake On Lan (wol) Message-ID: <3df44f150806090644r7dd0c6b7m16cd795cbff05799@mail.gmail.com> Hello, Erlang team! I try to send magic pack of wake on lan feature ( http://en.wikipedia.org/wiki/Wake_on_lan) in erlang. code below: > {ok,S} = gen_udp:open(2025). {ok,#Port<0.275>} > gen_udp:send(S, Pack). {error,einval} > gen_udp:close(S). Pack is the magic packet. i.e. I have an error in gen_udp:send(S, Pack). Please, help me. PS: sorry for my english, I'm from Russia =) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Mon Jun 9 16:01:57 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 09 Jun 2008 16:01:57 +0200 Subject: [erlang-questions] [clarify] Limits and overhead of monitors In-Reply-To: References: Message-ID: Jay Nelson writes: > Has anyone determined the answer to any of the following questions: > > 1) Is a practical limit to the number of monitors a single process > can hold? There is no theoretical limit. The practical limit probably depends on the application and the amount of memory in the computer. > - I assume they are fairly static, so just the memory to store > them? > - how are they stored? The endpoints for each monitor are stored in AVL trees. A monitor requires about 10 words at each endpoint. > - any problems if I have 100K processes in an everyone > monitoring everyone configuration? If one process dies, 100,000 processes will each get one DOWN message and become runnable. It could take some time for the system run all those processes. > 2) How much overhead difference is there between a local monitor and > remote node process monitor? > > - The pinging between nodes shouldn't need to propagate, just > the failure to ping a node Correct. /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From ignatios@REDACTED Mon Jun 9 15:37:16 2008 From: ignatios@REDACTED (Ignatios Souvatzis) Date: Mon, 9 Jun 2008 15:37:16 +0200 Subject: [erlang-questions] mnesia:delete on nonexistent key: ok? Message-ID: <20080609133716.GD23806@cs.uni-bonn.de> Hi, does mnesia:delete on a tab/key that's not there return ok? (I seem to observe this, but I'm can't find documentation on whether this is the intended behaviour.) -is From vances@REDACTED Mon Jun 9 17:00:59 2008 From: vances@REDACTED (Vance Shipley) Date: Mon, 9 Jun 2008 11:00:59 -0400 Subject: [erlang-questions] [clarify] Limits and overhead of monitors In-Reply-To: References: Message-ID: <20080609150058.GC3253@h216-235-12-173.host.egate.net> I'm seeing a paralell between this discussion of 100K monitors and the selective receive thread and the question therein of practical reasons why a mailbox would grow large. If a process had 100K monitors and there was a cascading crash within that population of processes you would have a very large number of messages arriving in one mailbox. Furthermore you may well want to deal with the monitor messages on a priority basis. If you are running a marshalling process you'd probably rather know the consumer process is gone before forwarding a message to it. -Vance From tsuraan@REDACTED Mon Jun 9 17:11:54 2008 From: tsuraan@REDACTED (tsuraan) Date: Mon, 9 Jun 2008 10:11:54 -0500 Subject: [erlang-questions] How long for the FFI EEP to go through? In-Reply-To: References: <62F3426C-094F-4A61-9E15-496B7AC3E069@gmail.com> Message-ID: <84fb38e30806090811y28b4c768wc93f31b7607aebb6@mail.gmail.com> > On 12/11/07, Joel Reymont wrote: >> Is there an estimate for when the FFI EEP will be part of the Erlang >> distribution? >> > No, there is no estimate. It's been about six months, so I figured I'd raise the question again. Has any progress been made on the libffi integration for Erlang? I'm very interested in this, so is there anything I can do to help, or are the details with blocking calls and stability still just being ironed out? Is there a good place to get news on active EEPs? The page for EEP7 isn't exactly active, and it just says to ping this news group for information. From hakan@REDACTED Mon Jun 9 17:12:22 2008 From: hakan@REDACTED (Hakan Mattsson) Date: Mon, 9 Jun 2008 17:12:22 +0200 (CEST) Subject: [erlang-questions] mnesia:delete on nonexistent key: ok? In-Reply-To: <20080609133716.GD23806@cs.uni-bonn.de> References: <20080609133716.GD23806@cs.uni-bonn.de> Message-ID: On Mon, 9 Jun 2008, Ignatios Souvatzis wrote: IS> does mnesia:delete on a tab/key that's not there return ok? IS> IS> (I seem to observe this, but I'm can't find documentation on whether IS> this is the intended behaviour.) Yes, it is the intended behavior. Mnesia, ets and dets all regards this as a normal case (not an error). /H?kan --- H?kan Mattsson Erlang/OTP, Ericsson From matthias@REDACTED Mon Jun 9 17:59:13 2008 From: matthias@REDACTED (Matthias Lang) Date: Mon, 9 Jun 2008 17:59:13 +0200 Subject: [erlang-questions] gen_udp and Wake On Lan (wol) In-Reply-To: <3df44f150806090644r7dd0c6b7m16cd795cbff05799@mail.gmail.com> References: <3df44f150806090644r7dd0c6b7m16cd795cbff05799@mail.gmail.com> Message-ID: <18509.21329.481977.72870@antilipe.corelatus.se> Andrey Shnayder writes: > code below: > > {ok,S} = gen_udp:open(2025). > {ok,#Port<0.275>} > > gen_udp:send(S, Pack). > {error,einval} > > gen_udp:close(S). > > Pack is the magic packet. > i.e. I have an error in gen_udp:send(S, Pack). Please, help me. Why are you calling send with two arguments? The documented interface has four: http://erlang.org/doc/man/gen_udp.html 1> {ok,S} = gen_udp:open(2025). {ok,#Port<0.106>} 2> gen_udp:send(S, {172,16,2,1}, 1234, <<"not a magic wake on LAN packet">>). ok This at least sends a packet, but it still isn't what you want---you almost certainly want to broadcast that packet, so maybe something along the lines of 8> {ok,S} = gen_udp:open(0, [{broadcast, true}]). {ok,#Port<0.114>} 9> gen_udp:send(S, {172,16,255,255}, 9, <<"not a magic wake on LAN packet">>). would be more like it, though I don't know enough about wake-on-lan to be sure that this fulfils all the requirements. Matt From dmercer@REDACTED Mon Jun 9 18:23:14 2008 From: dmercer@REDACTED (David Mercer) Date: Mon, 9 Jun 2008 11:23:14 -0500 Subject: [erlang-questions] dynamic module compilation In-Reply-To: References: <3dbc6d1c0806080310n15166488p4d34d925f10ab731@mail.gmail.com> Message-ID: <003a01c8ca4d$1e6a61c0$f21ea8c0@SSI.CORP> On Sunday, June 08, 2008, Howard Yeh wrote: > see ERTS manual chapter 4 to see the abstract format. > > http://www.erlang.org/doc/apps/erts/part_frame.html > > there's also Core Erlang. But I am not sure where it's documented. What are the arguments for using one over the other? DBM From dmercer@REDACTED Mon Jun 9 18:27:35 2008 From: dmercer@REDACTED (David Mercer) Date: Mon, 9 Jun 2008 11:27:35 -0500 Subject: [erlang-questions] dynamic module compilation In-Reply-To: <4d08db370806080201i704b69c9m42bc7264cde36f1a@mail.gmail.com> References: <4d08db370806080201i704b69c9m42bc7264cde36f1a@mail.gmail.com> Message-ID: <003b01c8ca4d$ba7288e0$f21ea8c0@SSI.CORP> This reminds me of a suggestion months ago to compile regular expressions into modules "on-the-fly" for better performance. I thought that was a neat idea and put in on my list as a possible project for myself. Have not yet gotten around to it, but planned to it and benchmark against the regexp module to see if it is any better. Thought I might learn something from it, too. DBM _____ From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Hynek Vychodil Sent: Sunday, June 08, 2008 04:01 To: Jacob Perkins Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] dynamic module compilation compile:file or compile:forms with erl_parse and erl_scan than use code:purge and code:load or code:load_file But in your use case I think there could be better solution with ets from file or dets. 2008/6/8 Jacob Perkins : Hi all, I need some help understanding how to dynamically compile/generate an erlang module. I've seen other projects do this kind of thing, such as erlydtl, but my needs are much simpler. I have a few files that are basically lists of words. What I want to do is generate a module whose functions will return each list of words. So if I have two files named "adjectives" and "nouns", then the generated module (let's call it 'grammar') should have two functions, adjectives() and nouns(), that return their respective list of words. How can I do this? Thanks, Jacob _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://www.erlang.org/mailman/listinfo/erlang-questions -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From hayeah@REDACTED Mon Jun 9 20:03:50 2008 From: hayeah@REDACTED (Howard Yeh) Date: Mon, 9 Jun 2008 11:03:50 -0700 Subject: [erlang-questions] dynamic module compilation In-Reply-To: <003a01c8ca4d$1e6a61c0$f21ea8c0@SSI.CORP> References: <3dbc6d1c0806080310n15166488p4d34d925f10ab731@mail.gmail.com> <003a01c8ca4d$1e6a61c0$f21ea8c0@SSI.CORP> Message-ID: On 6/9/08, David Mercer wrote: > On Sunday, June 08, 2008, Howard Yeh wrote: > > see ERTS manual chapter 4 to see the abstract format. > > > > http://www.erlang.org/doc/apps/erts/part_frame.html > > > > there's also Core Erlang. But I am not sure where it's documented. > > > What are the arguments for using one over the other? I started working with Serl (the "other" Lisp front end one should not speak of) not knowing the erlang compiler takes Core Erlang. Reading Robert's LFE, it looks like using Core Erlang makes cleaner code, as it's just a bunch of records. Building erlang abstract syntax isn't very friendly. On the other hand, erl_syntax and erl_syntax_lib are very helpful. To do lambda lifting for letrec I just used the binding analysis included in erl_syntax. There's also map tree, fold tree, etc in erl_syntax. I think Core Erlang also has binding analysis included, but not the mapping, folding stuff. And Core Erlang isn't "extensible". erl_syntax is purportedly "extensible", but most of the functions don't work on custom syntax types. then again, Core Erlang has letrec, so had I used it, I wouldn't have to get into all the hairiness that is lambda lifting. On yet another hand, Core Erlang is pretty minimal. I think you have to do your own list comprehension, for example. just my 2 cents. howard > > DBM > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- hayeah.wordpress.com --metacircular thinking From fig@REDACTED Mon Jun 9 21:00:09 2008 From: fig@REDACTED (Michael FIG) Date: Mon, 9 Jun 2008 13:00:09 -0600 (CST) Subject: [erlang-questions] Erlide tutorial? Message-ID: <9754549.225131213038009752.JavaMail.root@zimbra> Hi, Could anybody out there please point me at a very simple "hello world" tutorial in using Erlide? I'm an Emacs user with absolutely no experience with Eclipse, and I can't for the life of me figure out how to run my newly-created Erlang projects, evaluate something in the console, or debug a project. I don't know if that's because my Erlide installation is broken, or if it's just because I don't know where to find these features. Thanks, -- Michael FIG , PMP MarkeTel Multi-Line Dialing Systems, Ltd. Phone: (306) 359-6893 ext. 528 From rvirding@REDACTED Tue Jun 10 01:10:46 2008 From: rvirding@REDACTED (Robert Virding) Date: Tue, 10 Jun 2008 01:10:46 +0200 Subject: [erlang-questions] dynamic module compilation In-Reply-To: References: <3dbc6d1c0806080310n15166488p4d34d925f10ab731@mail.gmail.com> <003a01c8ca4d$1e6a61c0$f21ea8c0@SSI.CORP> Message-ID: <3dbc6d1c0806091610t224bb86ck8ea2176b4db1ba56@mail.gmail.com> 2008/6/9 Howard Yeh : > On 6/9/08, David Mercer wrote: > > On Sunday, June 08, 2008, Howard Yeh wrote: > > > see ERTS manual chapter 4 to see the abstract format. > > > > > > http://www.erlang.org/doc/apps/erts/part_frame.html > > > > > > there's also Core Erlang. But I am not sure where it's documented. > > > > > > What are the arguments for using one over the other? > > I started working with Serl (the "other" Lisp front end one should not > speak of) not knowing the erlang compiler takes Core Erlang. > > Reading Robert's LFE, it looks like using Core Erlang makes cleaner > code, as it's just a bunch of records. Building erlang abstract syntax > isn't very friendly. > > On the other hand, erl_syntax and erl_syntax_lib are very helpful. To > do lambda lifting for letrec I just used the binding analysis included > in erl_syntax. There's also map tree, fold tree, etc in erl_syntax. I > think Core Erlang also has binding analysis included, but not the > mapping, folding stuff. And Core Erlang isn't "extensible". erl_syntax > is purportedly "extensible", but most of the functions don't work on > custom syntax types. > > then again, Core Erlang has letrec, so had I used it, I wouldn't have > to get into all the hairiness that is lambda lifting. On yet another > hand, Core Erlang is pretty minimal. I think you have to do your own > list comprehension, for example. > > just my 2 cents. Yes, Core erlang is a very simple, minimal functional language. You really have to do most things yourself. The benefit of using it is that it is simple and minimal so it is really very easy to generate. Strangely the only thing missing is non-recursive local functions. Well, not so strange really as Core was developed to compile erlang and only recursive local functions are really needed for that. Actually for just list/binary comprehensions. Comprehensions are easy to compile by the way. One very useful thing which Core does handle is patterns, these are in the Core language so you don't have to worry about compiling them. The only documentation I know of is a paper written by Richard Carlsson. This together with the record definitions and some examples will be enough for most things. Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.burri@REDACTED Tue Jun 10 10:37:36 2008 From: greg.burri@REDACTED (Greg Burri) Date: Tue, 10 Jun 2008 10:37:36 +0200 Subject: [erlang-questions] process_info(Pid, memory). Message-ID: <60ed8a460806100137o271c37c3jf0e227e01f2638f8@mail.gmail.com> Hi everybody, I'm coding (just for fun) a little chat system with Erlang based on Yaws[1]. I'm using some "server push" (also know as COMET) to make a real time chat. Basically the client send its state with a XMLHttpRequest() and the server decide if this state is up to date or not. If not then the server send the delta and if yes then an event is waiting from Mnesia and the connection is held. You can see the actual work here (in french) : http://www.euphorik.ch Please, don't use MS Internet Explorer, I have some little issues with it. (you can try to speak in french if you want ;) ) I try to know how much memory each connection have. I'm using process_info, I have these results : >process_info(pid(0, 29961, 0), memory). {memory,142736} (~139kB) >From where these 139kB come ? I think it's a bit huge. More details : > process_info(pid(0, 29961, 0)). [{current_function, {euphorik_protocole,wait_event_bd_page_chat,0}}, {initial_call,{proc_lib,init_p,5}}, {status,waiting}, {message_queue_len,0}, {messages,[]}, {links,[<0.65.0>,<0.80.0>,#Port<0.20824>]}, {dictionary,[..]} {trap_exit,false}, {error_handler,error_handler}, {priority,normal}, {group_leader,<0.36.0>}, {total_heap_size,17711}, {heap_size,6765}, {stack_size,58}, {reductions,3700}, {garbage_collection,[{fullsweep_after,65535}]}, {suspending,[]}] When I suspend (with a simple "receive _ -> ok end") my "COMET" process before the mnesia event waiting I have variables sizes of process_info(Pid, memory) from 50kB to 1.5MB depends where I suspend.. It's a bit amazing. For exemple, if I suspend my process at the begining of euphorik_protocole:wait_event([{page, "chat"} | Data]) the process size is around 1.5MB. The source code[2] is available under GPLv3 (the comments are sadly in french). So... my questions are : 1. What exactly means process_info(Pid, memory) ? 2. Why my process size varying in a such large difference ? [1] : http://www.euphorik.ch [2] : http://svn.euphorik.ch/index.cgi/euphorik/browse/tags/1.0.1/modules/erl The beginning of the "COMET" process is euphorik_requests:out/1 TIA /Greg From 0x6e6562@REDACTED Tue Jun 10 11:30:50 2008 From: 0x6e6562@REDACTED (Ben Hood) Date: Tue, 10 Jun 2008 10:30:50 +0100 Subject: [erlang-questions] Using QLC to select unique rows Message-ID: Hi, I have a question about to most efficiency query a large table for unique rows. For example, if I have a table with 3 attributes and I want to retrieve all unique combinations of 2 of the attributes, the cost of this appears to be linear in the amount of rows (see code below). In this example I am inserting N identical rows and 1 different row and then querying for the 1 row that is different from the rest. As an alternative, I had considered decomposing the table into a separate table that would only store unique combinations of the attributes I want to query and then perform a join on the 2 tables. However, doing so changes my entity model and would introduce more CRUD maintenance into the app. Are there any good practices for doing this with QLC/mnesia or does my code have a glaring mistake in it? Thx, Ben -module(foo). -include_lib("stdlib/include/qlc.hrl"). -compile(export_all). -record(a, {id,first,second}). insert(N) -> mnesia:create_schema([node()]), mnesia:start(), mnesia:delete_table(a), mnesia:create_table(a, [{attributes, record_info(fields, a)}]), mnesia:add_table_index(a,first), mnesia:add_table_index(a,second), F = fun() -> write(#a{first = N,second = N},N) end, mnesia:ets(F). q() -> Q = qlc:q([A || A <- mnesia:table(a), A#a.first == "something", A#a.second == "else"]), F = fun() -> qlc:e(Q,[unique_all]) end, timer:tc(mnesia,transaction,[F]). write(_,0) -> mnesia:write(#a{id = 0, first = "something",second = "else"}); write(X,N) -> mnesia:write(X#a{id = N}), write(X,N-1). From alceste@REDACTED Tue Jun 10 12:27:08 2008 From: alceste@REDACTED (Alceste Scalas) Date: Tue, 10 Jun 2008 12:27:08 +0200 Subject: [erlang-questions] Erlang FFI: status report In-Reply-To: <84fb38e30806090811y28b4c768wc93f31b7607aebb6@mail.gmail.com> References: <62F3426C-094F-4A61-9E15-496B7AC3E069@gmail.com> <84fb38e30806090811y28b4c768wc93f31b7607aebb6@mail.gmail.com> Message-ID: <1213093633.10156.54.camel@gnatziu.crs4.it> Il giorno lun, 09/06/2008 alle 10.11 -0500, tsuraan ha scritto: > Has any progress been made on the libffi integration for Erlang? I'm > very interested in this, so is there anything I can do to help, or are > the details with blocking calls and stability still just being ironed > out? Is there a good place to get news on active EEPs? The page for > EEP7 isn't exactly active, and it just says to ping this news group > for information. The news is that there are no news :-) The FFI patches published on [1] are the last version I've developed, and we're using them daily here at CRS4. Based on our experience, they are stable. Blocking calls are a feature, and not a bug (the FFI does *not* want to replace the Erlang driver interface, but just simplify some common cases of native function calls). Since we are still using OTP R11B-5, there are no FFI patches for OTP R12B (yet). However, I've received some feedback, and one voluntary developer is (was?) actually working on porting the patches --- but nothing has been released so far. If nothing changes, I'll work on it when we'll switch to OTP R12B. Regarding a possible timeline, the FFI has been defined a "tentative goal for 2008" from the Erlang/OTP Development Team at Ericsson [2]. If I receive some feedback about the EEP 7 status, we (at CRS4) may decide to change our priorities and work on the R12B FFI patches sooner than later. Regards, alceste References: [1] Foreign Function Interface for Erlang/OTP http://muvara.org/crs4/erlang/ffi [2] Kenneth Lundin "News from the Erlang/OTP Development team at Ericsson" Presentation at the Erlang/OTP User Conference 2007, pag.15 http://www.erlang.se/euc/07/papers/1630OTPupdateEUC07.pdf -- Alceste Scalas CRS4 - http://www.crs4.it/ From 0x6e6562@REDACTED Tue Jun 10 13:51:16 2008 From: 0x6e6562@REDACTED (Ben Hood) Date: Tue, 10 Jun 2008 12:51:16 +0100 Subject: [erlang-questions] Index Overhead In Mnesia Message-ID: Hi, I'm looking into the rate of inserting rows in mnesia. Having written the attached test (that can be parameterized to insert an arbitrary amount of rows in arbitrary chunk sizes), I've found out so far that the highest throughput seems to be somebody where between 50 and 200 per transaction. What surprised me a bit is the magnitude of the effect that index maintenance has on the rate of insertion. If I place secondary indexes on two non-key attributes, the throughput drops off considerably. For example, inserting 10000 rows in batches of 1000 whilst maintaining 2 non-key indexes produces the following rates of insertion per batch: rate:insert(10000,1000). Batch rate = 10688 Batch rate = 7182 Batch rate = 5001 Batch rate = 4072 Batch rate = 3300 Batch rate = 2866 Batch rate = 2377 Batch rate = 2166 Batch rate = 1807 Batch rate = 1303 The Batch rate is the amount of inserts per second in each batch. This tallies up with the idea that at the beginning the index overhead is tiny, but grows on each insertion, which is normal. I just didn't think that the throughput would drop off so sharply. Does anybody know if I'm doing something completely wrong or if there is a much better way to use mnesia with large tables? Thanks, Ben -module(rate). -compile(export_all). -record(a, {id,first,second}). init() -> mnesia:create_schema([node()]), mnesia:start(), mnesia:delete_table(a), mnesia:create_table(a, [{attributes, record_info(fields, a)}]), mnesia:add_table_index(a,first), mnesia:add_table_index(a,second), ok. insert(N,BatchSize) -> mnesia:clear_table(a), batch(N, BatchSize). batch(0,_) -> ok; batch(N,BS) -> F = fun() -> write(#a{first = BS,second = BS},BS) end, {Time,_} = timer:tc(mnesia,transaction,[F]), io:format("Batch rate = ~p~n",[round(BS / Time * 1000000)]), batch(N - BS, BS). write(_,0) -> ok; write(X,N) -> mnesia:write(X#a{id = now()}), write(X,N-1). From christoph.mewes@REDACTED Tue Jun 10 14:12:15 2008 From: christoph.mewes@REDACTED (Christoph Mewes) Date: Tue, 10 Jun 2008 14:12:15 +0200 Subject: [erlang-questions] Erlang London / Stockholm UG meetings this week In-Reply-To: <48327F7A.1020303@erlang-consulting.com> References: <4831A6FB.6090007@erlang-consulting.com> <290b3ba10805191807m18f7e730ica029a96cf28e06c@mail.gmail.com> <48327F7A.1020303@erlang-consulting.com> Message-ID: <484E6F9F.7030703@freiheit.com> Hi. Did the recording happen? I can't find it on google yet, so i thought i'd ask back if there is a recording to wait for. Greets, Fossi Francesco Cesarini schrieb: > Yes, both events will be taped. > > Francesco > -- > http://www.erlang-consulting.com > > t ty wrote: >> Hello, >> >> Will both events be taped for those of us unfortunately enough to be >> on the wrong side of the Atlantic ? >> >> Thanks, >> >> Tee >> >> On Mon, May 19, 2008 at 12:12 PM, Francesco Cesarini >> wrote: >> >>> Hi All, >>> >>> a note to remind you that if you are planing to attend Robert Virding's >>> presentation on Lisp Flavored Erlang on the 22nd of May in Stockholm, >>> you need to register. There are plenty of spaces available, so spread >>> the word and bring your friends, but it is good if we know in advance >>> how many you are just in case we need to move to an even larger lecture >>> hall. You can sign up following the register link: >>> http://www.erlang-consulting.com/erlang/events.html#58 >>> Stockholms >>> Universitet is kindly providing the lecture room. Fredrik Thulin will >>> introduce the event and give a brief presentation on the usage of Erlang >>> within the University. To subscribe to the Stockholm UG mailing list, >>> send a blank email to erlangstockholm-subscribe@REDACTED >>> >>> For those in London, the UG meeting will be on the 21st of May. Hunter >>> Morris, CTO and co-founder of Smarkets, a London-based betting exchange >>> startup with a primarily Erlang code base will be presenting their >>> Erlang Web Framework. For more informaiton on this event and to >>> register, visit http://www.erlang-consulting.com/erlang/events.html#56 >>> To subscribe to the London UG mailing list, send a blank email to >>> erlang-london-subscribe@REDACTED >>> >>> See you there, >>> >>> Francesco >>> -- >>> http://www.erlang-consulting.com >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- Christoph Mewes Dipl.-Wi.-Inform. freiheit.com technologies gmbh Stra?enbahnring 22 / 20251 Hamburg, Germany fon +49 (0)40 / 890584-0 fax +49 (0)40 / 890584-20 HRB Hamburg 70814 FE97 1705 44D3 ED72 1E59 7F3E 2D93 C469 042A 43CD Gesch?ftsf?hrer: Claudia Dietze, Stefan Richter, J?rg Kirchhof -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature URL: From vychodil.hynek@REDACTED Tue Jun 10 14:46:22 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Tue, 10 Jun 2008 14:46:22 +0200 Subject: [erlang-questions] Index Overhead In Mnesia In-Reply-To: References: Message-ID: <4d08db370806100546p3f76740aj2bbc36d27ac40e5f@mail.gmail.com> It is not obvious, but try change your write: write(_,0) -> ok; write(X,N) -> T = now(), mnesia:write(X#a{id = T, first = T, second = T}), write(X,N-1). and think what happened to indexes in previous version :-) On Tue, Jun 10, 2008 at 1:51 PM, Ben Hood <0x6e6562@REDACTED> wrote: > Hi, > > I'm looking into the rate of inserting rows in mnesia. > > Having written the attached test (that can be parameterized to insert > an arbitrary amount of rows in arbitrary chunk sizes), I've found out > so far that the highest throughput seems to be somebody where between > 50 and 200 per transaction. > > What surprised me a bit is the magnitude of the effect that index > maintenance has on the rate of insertion. > > If I place secondary indexes on two non-key attributes, the throughput > drops off considerably. > > For example, inserting 10000 rows in batches of 1000 whilst > maintaining 2 non-key indexes produces the following rates of > insertion per batch: > > rate:insert(10000,1000). > Batch rate = 10688 > Batch rate = 7182 > Batch rate = 5001 > Batch rate = 4072 > Batch rate = 3300 > Batch rate = 2866 > Batch rate = 2377 > Batch rate = 2166 > Batch rate = 1807 > Batch rate = 1303 > > The Batch rate is the amount of inserts per second in each batch. > > This tallies up with the idea that at the beginning the index overhead > is tiny, but grows on each insertion, which is normal. > > I just didn't think that the throughput would drop off so sharply. > > Does anybody know if I'm doing something completely wrong or if there > is a much better way to use mnesia with large tables? > > Thanks, > > Ben > > -module(rate). > > -compile(export_all). > > -record(a, {id,first,second}). > > init() -> > mnesia:create_schema([node()]), > mnesia:start(), > mnesia:delete_table(a), > mnesia:create_table(a, > [{attributes, record_info(fields, a)}]), > mnesia:add_table_index(a,first), > mnesia:add_table_index(a,second), > ok. > > insert(N,BatchSize) -> > mnesia:clear_table(a), > batch(N, BatchSize). > > batch(0,_) -> ok; > batch(N,BS) -> > F = fun() -> write(#a{first = BS,second = BS},BS) end, > {Time,_} = timer:tc(mnesia,transaction,[F]), > io:format("Batch rate = ~p~n",[round(BS / Time * 1000000)]), > batch(N - BS, BS). > > write(_,0) -> ok; > write(X,N) -> > mnesia:write(X#a{id = now()}), > write(X,N-1). > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From icfp.publicity@REDACTED Tue Jun 10 14:47:04 2008 From: icfp.publicity@REDACTED (Matthew Fluet (ICFP Publicity Chair)) Date: Tue, 10 Jun 2008 07:47:04 -0500 Subject: [erlang-questions] DEFUN08: Final Call for Talks & Tutorials (deadline: June 27) Message-ID: <53ff55480806100547x737a8daag30661c7959d9bd1b@mail.gmail.com> Final Call for Talks and Tutorials ACM SIGPLAN 2008 Developer Tracks on Functional Programming http://www.deinprogramm.de/defun-2008/ Victoria, BC, Canada, 25, 27 September, 2008 The workshop will be held in conjunction with ICFP 2008. http://www.icfpconference.org/icfp2008/ Important dates Proposal Deadline: June 27, 2008, 0:00 UTC Notification: July 14, 2008 DEFUN 2008 invites functional programmers who know how to solve problems with functional progamming to give talks and lead tutorials at the The ICFP Developer Tracks. We want to know about your favorite programming techniques, powerful libraries, and engineering approaches you've used that the world should know about and apply to other projects. We want to know how to be productive using functional programming, write better code, and avoid common pitfalls. We invite proposals for presentations in the following categories: How-to talks: 45-minute "how-to" talks that provide specific information on how to solve specific problems using functional programming. These talks focus on concrete examples, but provide useful information for developers working on different projects or in different contexts. Examples: - "How I made Haskell an extension language for SAP R/3." - "How I replaced /sbin/init by a Scheme program." - "How I hooked up my home appliances to an Erlang control system." - "How I got an SML program to drive my BMW." General language tutorials: Half-day general language tutorials for specific functional languages, given by recognized experts for the respective languages. Technology tutorials: Half-day tutorials on techniques, technologies, or solving specific problems in functional programming. Examples: - how to make the best use of specific FP programming techniques - how to inject FP into a development team used to more conventional technologies - how to connect FP to existing libraries / frameworks / platforms - how to deliver high-performance systems with FP - how to deliver high-reliability systems with FP Remember that your audience will include computing professionals who are not academics and who may not already be experts on functional programming. Presenters of tutorials will receive free registration to ICFP 2008. Submission guidelines Submit a proposal of 150 words or less for either a 45-minute talk with a short Q&A session at the end, or a 300-word-or-less proposal for a 3-hour tutorial, where you present your material, but also give participants a chance to practice it on their own laptops. Some advice: - Give it a simple and straightforward title or name; avoid fancy titles or puns that would make it harder for attendees to figure out what you'll be talking about. - Clearly identify the level of the talk: What knowledge should people have when they come to the presentation or tutorial? - Explain why people will want to attend: is the language or library useful for a wide range of attendees? Is the pitfall you're identifying common enough that a wide range of attendees is likely to encounter it? - Explain what benefits attendees are expected to take home to their own projects. - For a tutorial, explain how you want to structure the time, and what you expect to have attendees to do on their laptops. List what software you'll expect attendees to have installed prior to coming. Submit your proposal in plain text electronically to defun-2008-submission-AT-deinprogramm.de by the beginning of Friday, June 27, Universal Coordinated Time. Organizers Kathleen Fisher AT&T Labs Simon Peyton Jones Microsoft Research Mike Sperber (co-chair) DeinProgramm Don Stewart (co-chair) Galois From gijsbert_de_haan@REDACTED Tue Jun 10 15:39:07 2008 From: gijsbert_de_haan@REDACTED (Gijsbert de Haan) Date: Tue, 10 Jun 2008 09:39:07 -0400 Subject: [erlang-questions] terminate() not finishing after application:stop() Message-ID: <1213105147.14614.1257699575@webmail.messagingengine.com> Hi, I have an application that I want to stop. In terminate() of one of the processes I save some state to an xml file (with xmerl:export() and file:write_file()). When this process terminates by itself all is well, and the file is saved. But when I call application:stop(myapp) the call to xmerl:export() (in terminate) never returns and the state is not saved. There are no error reports in the log, just the usual info report about myapp having exited. I have put a try/catch (throw/exit/error) around the xmerl:export() with some io:format() calls but nothing comes out. Anybody have a suggestion? Thanks, Gijsbert From ulf.wiger@REDACTED Tue Jun 10 15:45:51 2008 From: ulf.wiger@REDACTED (Ulf Wiger (TN/EAB)) Date: Tue, 10 Jun 2008 15:45:51 +0200 Subject: [erlang-questions] terminate() not finishing after application:stop() In-Reply-To: <1213105147.14614.1257699575@webmail.messagingengine.com> References: <1213105147.14614.1257699575@webmail.messagingengine.com> Message-ID: <484E858F.9010808@ericsson.com> Make sure your process is trapping exits. If it isn't, the process will die immediately when the supervisor calls exit(Pid, shutdown). BR, Ulf W Gijsbert de Haan skrev: > Hi, > > I have an application that I want to stop. In terminate() of one of the > processes I save some state to an xml file (with xmerl:export() and > file:write_file()). When this process terminates by itself all is well, > and the file is saved. > But when I call application:stop(myapp) the call to xmerl:export() (in > terminate) never returns and the state is not saved. There are no error > reports in the log, just the usual info report about myapp having > exited. I have put a try/catch (throw/exit/error) around the > xmerl:export() with some io:format() calls but nothing comes out. > > Anybody have a suggestion? > Thanks, > Gijsbert > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From bekesa@REDACTED Tue Jun 10 15:53:20 2008 From: bekesa@REDACTED (Andras Georgy Bekes) Date: Tue, 10 Jun 2008 15:53:20 +0200 Subject: [erlang-questions] mnesia:delete on nonexistent key: ok? In-Reply-To: References: <20080609133716.GD23806@cs.uni-bonn.de> Message-ID: <200806101553.20215.bekesa@sch.bme.hu> > IS> does mnesia:delete on a tab/key that's not there return ok? > IS> (I seem to observe this, but I'm can't find documentation on > whether IS> this is the intended behaviour.) > Yes, it is the intended behavior. Mnesia, ets and dets > all regards this as a normal case (not an error). Just like lists, dict, sets, orddict, ordsets, and in some sense, the process dictionary. It seems only gb_sets and gb_trees are different: they have two delete operations, one that crashes if the element (key) not found, and one that doesn't. Georgy From sebastian.bello@REDACTED Tue Jun 10 14:59:13 2008 From: sebastian.bello@REDACTED (Sebastian Bello) Date: Tue, 10 Jun 2008 09:59:13 -0300 Subject: [erlang-questions] SNMP questions Message-ID: <484E7AA1.8020605@inswitch.us> Hi list, we are currently trying to use the Erlang SNMP toolkit and have some questions: 1- we are using /snmpa:send_trap/ to send traps from within the master agent node. How should this be done from a different node? Maybe using Erlang communication to send the traps' information to the master agent node and then issuing a /snmpa:send_trap/ from it? Or is there a way to specify a remote node in /snmpa:send_trap/? 2- in a distributed application, how should the MIB represent data from applications running on different nodes? Thanks, Sebastian- -------------- next part -------------- An HTML attachment was scrubbed... URL: From gijsbert_de_haan@REDACTED Tue Jun 10 16:12:33 2008 From: gijsbert_de_haan@REDACTED (Gijsbert de Haan) Date: Tue, 10 Jun 2008 10:12:33 -0400 Subject: [erlang-questions] terminate() not finishing after application:stop() In-Reply-To: <484E858F.9010808@ericsson.com> References: <1213105147.14614.1257699575@webmail.messagingengine.com> <484E858F.9010808@ericsson.com> Message-ID: <1213107153.26224.1257703723@webmail.messagingengine.com> The process is indeed trapping exits. I see my debug io:format() from my terminate() function before the call to xmerl:export/3, but not the ones after. In the following code (edited) I see 'before_xmerl_export', but nothing after (related to that process): terminate(Reason, State) -> ?D(Reason), {memory, Memory} = erlang:process_info(self(), memory), error_logger:info_msg( "~p session '~s' memory usage ~B\n", [self(), Name, Memory]), Prolog = [""], ?D(Prolog), Xml = getXml(State), ?D(before_xmerl_export), try xmerl:export( [Xml], xmerl_xml, [#xmlAttribute{name = prolog, value = Prolog}]) of StrData -> ?D(before_list_to_binary), Data = list_to_binary(StrData), case file:write_file(Name2, Data) of ok -> error_logger:info_msg("~p session '~s' saved\n", [self(), Name2]); {error, Error} -> error_logger:error_msg( "~p session '~s' NOT saved\nError ~p\n", [self(), Name2, Error]) end catch throw:X -> ?D(throw), ?D(X); exit:X -> ?D(exit), ?D(X); error:X -> ?D(error), ?D(X) end, ok. On Tue, 10 Jun 2008 15:45:51 +0200, "Ulf Wiger (TN/EAB)" said: > > Make sure your process is trapping exits. > If it isn't, the process will die immediately when > the supervisor calls exit(Pid, shutdown). > > BR, > Ulf W > > Gijsbert de Haan skrev: > > Hi, > > > > I have an application that I want to stop. In terminate() of one of the > > processes I save some state to an xml file (with xmerl:export() and > > file:write_file()). When this process terminates by itself all is well, > > and the file is saved. > > But when I call application:stop(myapp) the call to xmerl:export() (in > > terminate) never returns and the state is not saved. There are no error > > reports in the log, just the usual info report about myapp having > > exited. I have put a try/catch (throw/exit/error) around the > > xmerl:export() with some io:format() calls but nothing comes out. > > > > Anybody have a suggestion? > > Thanks, > > Gijsbert > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions From tonyg@REDACTED Tue Jun 10 16:50:33 2008 From: tonyg@REDACTED (Tony Garnock-Jones) Date: Tue, 10 Jun 2008 15:50:33 +0100 Subject: [erlang-questions] Twoorl: an open source Twitter clone In-Reply-To: <4848411B.4090804@bitfurnace.com> References: <4848237A.7090103@bitfurnace.com> <4848411B.4090804@bitfurnace.com> Message-ID: <484E94B9.3030701@lshift.net> Damien Morton wrote: > Makes one wonder where the database refactoring tools are :) http://www.dabbledb.com/ :-) Tony From 0x6e6562@REDACTED Tue Jun 10 17:00:28 2008 From: 0x6e6562@REDACTED (Ben Hood) Date: Tue, 10 Jun 2008 16:00:28 +0100 Subject: [erlang-questions] Index Overhead In Mnesia In-Reply-To: <4d08db370806100546p3f76740aj2bbc36d27ac40e5f@mail.gmail.com> References: <4d08db370806100546p3f76740aj2bbc36d27ac40e5f@mail.gmail.com> Message-ID: <045CF58E-4EB1-4683-8C64-5D4394171B0F@gmail.com> Hynek, On 10 Jun 2008, at 13:46, Hynek Vychodil wrote: > It is not obvious, but try change your write: > > write(_,0) -> ok; > write(X,N) -> > T = now(), > mnesia:write(X#a{id = T, first = T, second = T}), > write(X,N-1). > > and think what happened to indexes in previous version :-) Very interesting! So, if I understand you correctly, what I conclude is that the index performance degrades when you have lots of identical keys. Do you know why this is the case? Does the index perhaps maintain a list of values for each key and hence the set of values has to be updated continuously? If so, is this a bug or a feature? Thx, Ben From vychodil.hynek@REDACTED Tue Jun 10 17:25:49 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Tue, 10 Jun 2008 17:25:49 +0200 Subject: [erlang-questions] Index Overhead In Mnesia In-Reply-To: <045CF58E-4EB1-4683-8C64-5D4394171B0F@gmail.com> References: <4d08db370806100546p3f76740aj2bbc36d27ac40e5f@mail.gmail.com> <045CF58E-4EB1-4683-8C64-5D4394171B0F@gmail.com> Message-ID: <4d08db370806100825q36d4269fy5031b58ce71aa9eb@mail.gmail.com> I think this is feature of bag and duplicate_bag ets table types and is mentioned in manual. On Tue, Jun 10, 2008 at 5:00 PM, Ben Hood <0x6e6562@REDACTED> wrote: > Hynek, > > On 10 Jun 2008, at 13:46, Hynek Vychodil wrote: > > It is not obvious, but try change your write: >> >> write(_,0) -> ok; >> write(X,N) -> >> T = now(), >> mnesia:write(X#a{id = T, first = T, second = T}), >> write(X,N-1). >> >> and think what happened to indexes in previous version :-) >> > > Very interesting! > > So, if I understand you correctly, what I conclude is that the index > performance degrades when you have lots of identical keys. > > Do you know why this is the case? > > Does the index perhaps maintain a list of values for each key and hence the > set of values has to be updated continuously? > > If so, is this a bug or a feature? > > Thx, > > Ben > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Tue Jun 10 17:30:03 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Tue, 10 Jun 2008 17:30:03 +0200 Subject: [erlang-questions] Index Overhead In Mnesia In-Reply-To: <045CF58E-4EB1-4683-8C64-5D4394171B0F@gmail.com> References: <4d08db370806100546p3f76740aj2bbc36d27ac40e5f@mail.gmail.com> <045CF58E-4EB1-4683-8C64-5D4394171B0F@gmail.com> Message-ID: <4d08db370806100830n5cc0e98ye671534d84feec31@mail.gmail.com> I think this is feature of bag and duplicate_bag ets table types and is mentioned in manual. On Tue, Jun 10, 2008 at 5:00 PM, Ben Hood <0x6e6562@REDACTED> wrote: > Hynek, > > On 10 Jun 2008, at 13:46, Hynek Vychodil wrote: > > It is not obvious, but try change your write: >> >> write(_,0) -> ok; >> write(X,N) -> >> T = now(), >> mnesia:write(X#a{id = T, first = T, second = T}), >> write(X,N-1). >> >> and think what happened to indexes in previous version :-) >> > > Very interesting! > > So, if I understand you correctly, what I conclude is that the index > performance degrades when you have lots of identical keys. > > Do you know why this is the case? > > Does the index perhaps maintain a list of values for each key and hence the > set of values has to be updated continuously? > > If so, is this a bug or a feature? > > Thx, > > Ben > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimo.cesaro@REDACTED Tue Jun 10 17:53:48 2008 From: massimo.cesaro@REDACTED (Massimo Cesaro) Date: Tue, 10 Jun 2008 17:53:48 +0200 Subject: [erlang-questions] On selective receive (Re: eep: multiple patterns) In-Reply-To: <3dbc6d1c0806061423r57c17bc9y62f9c3130369adfe@mail.gmail.com> References: <8209f740805302244o2aa53054s856523553506aee8@mail.gmail.com> <8268eea30806032346l463f42f4q798f47db08de84b3@mail.gmail.com> <95be1d3b0806040006q4ac93245x1c8405014398c5f7@mail.gmail.com> <8268eea30806040551r46127891hd2a9db0d8882c7e8@mail.gmail.com> <3dbc6d1c0806061423r57c17bc9y62f9c3130369adfe@mail.gmail.com> Message-ID: <7ae16d50806100853u70d7263dx1bd2745740318e4b@mail.gmail.com> 2008/6/6 Robert Virding : > > I have a serious question which I ask out of ignorance. I know that really > long message queues are possible in the existing language, it is easy to > write a program to show this, but do they occur in real applications? Or is > this theoretical worry? And if they do occur are they something which is > property of the problem which can't easily be avoided or are they the result > of some error in the logic or coding which shows up in message queue length? > > Robert > In a very complex TLC application that we released on field a couple of years ago, and which was soak tested in our labs, we experienced massive message queued do to disk I/O. The system was performing quite well for most of the time, but on rare occasions, writing to a sequential file on the file system provoked a buildup of messages on a couple of processes that in turn slowed everything to a crawl. So this is not a theoretical worry, and in our case it was not an error in the coding logic. We fixed it by completely bypassing the writing to the file system step in favor of a different approach involving a RAM based configuration mechanism. Massimo -- - Is there anything I can do? - Computer says "no". -------------- next part -------------- An HTML attachment was scrubbed... URL: From 0x6e6562@REDACTED Tue Jun 10 18:06:00 2008 From: 0x6e6562@REDACTED (Ben Hood) Date: Tue, 10 Jun 2008 17:06:00 +0100 Subject: [erlang-questions] Index Overhead In Mnesia In-Reply-To: <269388e30806100844n4aaa14ddkb4c88be5f24859fc@mail.gmail.com> References: <4d08db370806100546p3f76740aj2bbc36d27ac40e5f@mail.gmail.com> <045CF58E-4EB1-4683-8C64-5D4394171B0F@gmail.com> <4d08db370806100830n5cc0e98ye671534d84feec31@mail.gmail.com> <269388e30806100844n4aaa14ddkb4c88be5f24859fc@mail.gmail.com> Message-ID: <269388e30806100906m33e25619kd6f1ec66400e6092@mail.gmail.com> Hynek, On Tue, Jun 10, 2008 at 4:30 PM, Hynek Vychodil wrote: > I think this is feature of bag and duplicate_bag ets table types and is > mentioned in manual. Maybe, but my table does not specify a type, so it be a set by default. According to the mnesia manual, the three types of table are set, ordered_set or bag, whereas in ets/dets you can have a duplicate_bag as well. In the mnesia man page it states: "Indices do not come free, they occupy space which is proportional to the size of the table. They also cause insertions into the table to execute slightly slower." But I can't see anything specific about using bags and indices. I googled this post by Ulf a few years ago, which may shed some light onto the situation: http://www.erlang.org/pipermail/erlang-questions/2005-July/016083.html There seems to be a reference to a (now unmaintained) version of a mnesia index, which I assume to be the rdbms module in jungerl. Does anybody know if that addresses the issue, or even exactly what the underlying problem is? Ben From sean.hinde@REDACTED Tue Jun 10 19:03:52 2008 From: sean.hinde@REDACTED (Sean Hinde) Date: Tue, 10 Jun 2008 19:03:52 +0200 Subject: [erlang-questions] Index Overhead In Mnesia In-Reply-To: <045CF58E-4EB1-4683-8C64-5D4394171B0F@gmail.com> References: <4d08db370806100546p3f76740aj2bbc36d27ac40e5f@mail.gmail.com> <045CF58E-4EB1-4683-8C64-5D4394171B0F@gmail.com> Message-ID: <7082733C-07D7-4572-9ADB-02119FCB6C01@gmail.com> Hi, > So, if I understand you correctly, what I conclude is that the index > performance degrades when you have lots of identical keys. Yep! > Do you know why this is the case? > > Does the index perhaps maintain a list of values for each key and > hence the set of values has to be updated continuously? Yes > If so, is this a bug or a feature? It is a long standing "feature" of mnesia secondary indexes. It is one that I also ran into a few weeks into my use of Erlang/mnesia. I guess it is in the same category as repeated queue scanning for selective receive - easy to workaround in most cases, so no major incentive to change the behaviour. Sean From 0x6e6562@REDACTED Tue Jun 10 21:44:20 2008 From: 0x6e6562@REDACTED (Ben Hood) Date: Tue, 10 Jun 2008 20:44:20 +0100 Subject: [erlang-questions] Index Overhead In Mnesia In-Reply-To: <7082733C-07D7-4572-9ADB-02119FCB6C01@gmail.com> References: <4d08db370806100546p3f76740aj2bbc36d27ac40e5f@mail.gmail.com> <045CF58E-4EB1-4683-8C64-5D4394171B0F@gmail.com> <7082733C-07D7-4572-9ADB-02119FCB6C01@gmail.com> Message-ID: <269388e30806101244n46f316f0u6388970dbf5c8d0d@mail.gmail.com> > I guess it is in the same category as repeated queue scanning for selective > receive - easy to workaround in most cases, so no major incentive to change > the behaviour. So you're saying that one should decompose the duplicates into another table that just contains a canonical representation of each unique combination and just reference these from the original table? Ben From raould@REDACTED Tue Jun 10 22:27:58 2008 From: raould@REDACTED (Raoul Duke) Date: Tue, 10 Jun 2008 13:27:58 -0700 Subject: [erlang-questions] clueless performance question Message-ID: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> (i mean this with an eye towards getting more arguments for Erlang, not against it. i googled around but haven't hit anything super relevant to this question yet.) Erlang excels at concurrency and distribution. What aspects of the current Erlang VMs (HiPE and non-HiPE) make them possibly slower for serial computation vs. some other well-known VMs? If somebody wanted to work on tweaking the serial computation part, what might be the top things to examine (e.g. from personal experience, or from profiler usage)? This could be anything, like doing work on VM inner loops, or adding some more automatic optimization abilities. sincerely. From chsu79@REDACTED Tue Jun 10 23:13:36 2008 From: chsu79@REDACTED (Christian S) Date: Tue, 10 Jun 2008 23:13:36 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> References: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> Message-ID: On Tue, Jun 10, 2008 at 10:27 PM, Raoul Duke wrote: > Erlang excels at concurrency and distribution. What aspects of the > current Erlang VMs (HiPE and non-HiPE) make them possibly slower for > serial computation vs. some other well-known VMs? If somebody wanted > to work on tweaking the serial computation part, what might be the top > things to examine (e.g. from personal experience, or from profiler > usage)? This could be anything, like doing work on VM inner loops, or > adding some more automatic optimization abilities. Get the papers produced by the hipe group to see what they have done to increase the sequential performance. It's an interesting read, exposing what one have to win by using hipe. I would find research into what can be won by doing JIT interesting, things like avoiding external-call lookups by compiling it into a direct call (constraint it so it needs to be re-done when modules are replaced). I imagine these small book-keeping costs in dynamic languages sum up, if not else, there would be no reason for java jits to do it. :) Of course, for the forseeable future, the winner in raw computation will be those that know how to distribute the computation to make best use of many simultaneous processing units. PS. Possibly nobody in research wants to disclose their current work before they can get it into a paper to publish. It is how funded research tend to work. From ulf.wiger@REDACTED Wed Jun 11 00:54:45 2008 From: ulf.wiger@REDACTED (Ulf Wiger (TN/EAB)) Date: Wed, 11 Jun 2008 00:54:45 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: References: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> Message-ID: <484F0635.2020003@ericsson.com> Christian S skrev: > I would find research into what can be won by doing JIT interesting, > things like avoiding external-call lookups by compiling it into a > direct call (constraint it so it needs to be re-done when modules are > replaced). You could e.g. read Thomas Lindgren's paper on cross-module optimization: http://citeseer.ist.psu.edu/458295.html One of the things debated in connection with that work was the idea that we could opt for larger units of upgrade (normally, it makes sense to upgrade whole applications), and this could possibly open up for more agressive cross- module optimizations. Having a more expressive way of providing type restrictions also seems to be a way to speed up the code. Even today, speedups can be had by adding type guards at the right places (most notably perhaps for floating-point math). Of course, in many of the micro benchmarks where Erlang suffers, the key to high performance seems to be the ability to perform blocking IO. This problem may become less of an issue if we start benchmarking on multicores, where it may be possible to do such things and still not jeopardize real-time characteristics. BR, Ulf W From exta7@REDACTED Wed Jun 11 01:11:05 2008 From: exta7@REDACTED (Zvi) Date: Tue, 10 Jun 2008 16:11:05 -0700 (PDT) Subject: [erlang-questions] clueless performance question In-Reply-To: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> References: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> Message-ID: <17766840.post@talk.nabble.com> from my PI serial benchmarks, floating point performance is weak point of Erlang, since float values always allocated on the Heap. So, if you need to iterate over range [0:0.001:1] using tail recursion, it's much faster to iterate over integers [0:1000] and multiply it inside the function to the step 0.001 . The solution to this is itroducing APL/K/J/Q/Matlab - like vector/matrix of doubles datatype with possible SIMD and SMP optimizations for element-wise and reduction operators. This was discussed on this list several months ago. Zvi Raoul Duke wrote: > > (i mean this with an eye towards getting more arguments for Erlang, > not against it. i googled around but haven't hit anything super > relevant to this question yet.) > > Erlang excels at concurrency and distribution. What aspects of the > current Erlang VMs (HiPE and non-HiPE) make them possibly slower for > serial computation vs. some other well-known VMs? If somebody wanted > to work on tweaking the serial computation part, what might be the top > things to examine (e.g. from personal experience, or from profiler > usage)? This could be anything, like doing work on VM inner loops, or > adding some more automatic optimization abilities. > > sincerely. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > -- View this message in context: http://www.nabble.com/clueless-performance-question-tp17764097p17766840.html Sent from the Erlang Questions mailing list archive at Nabble.com. From thomasl_erlang@REDACTED Wed Jun 11 01:06:23 2008 From: thomasl_erlang@REDACTED (Thomas Lindgren) Date: Tue, 10 Jun 2008 16:06:23 -0700 (PDT) Subject: [erlang-questions] clueless performance question In-Reply-To: Message-ID: <315656.29951.qm@web38806.mail.mud.yahoo.com> --- Christian S wrote: > I would find research into what can be won by doing > JIT interesting, > things like avoiding external-call lookups by > compiling it into a > direct call (constraint it so it needs to be re-done > when modules are > replaced). I think both Hipe and Beam already link calls when modules are loaded. At least they used to do that. Part of the 'problem' (one view is that it's a good thing) is that builtins begin to dominate execution, and those are mostly out of the compiler's reach today. For instance, you may basically be spending your time inside the loops implementing ets operations or suchlike. Hipe can't help you there. Another problem is that all the sophisticated optimizations tend to yield little extra performance on real programs (ie, big nasty sprawling messes of code rather than neat inner loops). For whatever reasons; for instance, native code is nominally faster than bytecode but causes more cache misses with a large working set, so it evens out ... This experience is fairly common across the spectrum of programming languages. Worse, nowadays users want stuff like reliability and tracing :-) Nevertheless, I think there's plenty more performance on the table. It's just difficult to get at it. And finally, it's not like we're standing still. I measured the performance between OTP releases a few years back, running the same code on the same hardware, and found that Beam had improved by about 30% over time on my benchmarks. That's not bad. Best, Thomas From vances@REDACTED Wed Jun 11 02:22:34 2008 From: vances@REDACTED (Vance Shipley) Date: Tue, 10 Jun 2008 20:22:34 -0400 Subject: [erlang-questions] clueless performance question In-Reply-To: <315656.29951.qm@web38806.mail.mud.yahoo.com> References: <315656.29951.qm@web38806.mail.mud.yahoo.com> Message-ID: <20080611002233.GE3253@h216-235-12-173.host.egate.net> ... and no question about performance from a newcomer to Erlang should not be answered without including the advice from rule number 3.8 of the Erlang Programming rules: http://www.erlang.se/doc/programming_rules.shtml#HDR6 "Performance" is the least of your worries. From ok@REDACTED Wed Jun 11 02:39:50 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 11 Jun 2008 12:39:50 +1200 Subject: [erlang-questions] clueless performance question In-Reply-To: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> References: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> Message-ID: <490D297E-0151-489D-BF26-355E9AB486B2@cs.otago.ac.nz> On 11 Jun 2008, at 8:27 am, Raoul Duke wrote: > Erlang excels at concurrency and distribution. What aspects of the > current Erlang VMs (HiPE and non-HiPE) make them possibly slower for > serial computation vs. some other well-known VMs? I can put this very simply indeed: early binding => goes faster, much harder to change late binding => goes slower, much easier to change If you want object oriented extensions to C that have the potential to go very fast (but thanks to pervasive use of "smart pointers", will almost certainly go like a slug in treacle), use C++. C++ requires *major* compiler technology to go at all, let alone fast. If you want object oriented extensions to C and care much more about post-delivery extensibility, use Objective-C. You've got to love the way things like F-Script (roughly speaking, a Smalltalk dialect) can just plug into Objective-C and make Mac OS X look like Java on steroids. The compiler changes required at least for the first versions of ObjC were really very small. An ObjC compiler has much less type information to exploit, and practically everything (even, as a matter of factor, the dynamic dispatch function!) can be changed. Erlang is designed to support hot loading, the post-delivery replacement/upgrade of modules. That means late binding, which is *always* going to be less efficient than early binding. Of course, efficiency is not just a matter of micro-level language efficiency. It's a matter of algorithmic choice, and of system level design. And what really counts in practice is how well the system does what you want it to do NOW, not what you wanted it to do last year when the interfaces were frozen. From adam@REDACTED Wed Jun 11 03:07:13 2008 From: adam@REDACTED (Adam Wagner) Date: Tue, 10 Jun 2008 18:07:13 -0700 Subject: [erlang-questions] US DOD Certification of Erlang? Message-ID: I currently work for a US defense contractor, but am relatively new to government work. From what I've learned so far about Erlang I can see programs on the horizon where it would be a great fit. Whenever I've mentioned the idea to those higher up, I'm told that it would never happen because the DOD would never "certify" it. Has anybody else gone down this road with Erlang that could offer advice? Thanks, Adam From ulf.wiger@REDACTED Wed Jun 11 07:49:29 2008 From: ulf.wiger@REDACTED (Ulf Wiger (TN/EAB)) Date: Wed, 11 Jun 2008 07:49:29 +0200 Subject: [erlang-questions] US DOD Certification of Erlang? In-Reply-To: References: Message-ID: <484F6769.7000600@ericsson.com> Adam Wagner skrev: > I currently work for a US defense contractor, but am relatively new to > government work. From what I've learned so far about Erlang I can see > programs on the horizon where it would be a great fit. Whenever I've > mentioned the idea to those higher up, I'm told that it would never > happen because the DOD would never "certify" it. Has anybody else > gone down this road with Erlang that could offer advice? The DOD uses lots of tools that aren't necessarily certified, as long as they are specific solutions to some important problem. You'll find it difficult to view the DOD as one entity. There will be people who don't think you can do anything without first certifying all the steps, and there will be others who focus on solving the really interesting problems - the ones where, if things don't work right, people die. Pretty much like any big organization, really, but the DOD does have a core mission that's a bit more serious than most. Both extremes (the bureaucrats and the doers) will use that to justify their own position, of course. ;) In my personal dealings with the DOD, I've come across systems written in PowerHouse, Pascal, C, DECMessageQ, and even HyperCard (a full-blown command and control system, no less). Even back then, the official programming language was Ada. The official desktop computer was an ugly 386 PC, but of course the general had a nice Mac IIx on his desk (I'm sure it's been upgraded since...) Prototyping is usually permitted, and if you solve an operational problem, chances are it will get used, while others figure out how to do it with certified technology. At least that's how things used to get done in the places I got in touch with in the 90s. BR, Ulf W From vychodil.hynek@REDACTED Wed Jun 11 09:03:36 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Wed, 11 Jun 2008 09:03:36 +0200 Subject: [erlang-questions] Index Overhead In Mnesia In-Reply-To: <269388e30806101244n46f316f0u6388970dbf5c8d0d@mail.gmail.com> References: <4d08db370806100546p3f76740aj2bbc36d27ac40e5f@mail.gmail.com> <045CF58E-4EB1-4683-8C64-5D4394171B0F@gmail.com> <7082733C-07D7-4572-9ADB-02119FCB6C01@gmail.com> <269388e30806101244n46f316f0u6388970dbf5c8d0d@mail.gmail.com> Message-ID: <4d08db370806110003i10a61392wc645512aafc7f29@mail.gmail.com> On Tue, Jun 10, 2008 at 9:44 PM, Ben Hood <0x6e6562@REDACTED> wrote: > > I guess it is in the same category as repeated queue scanning for > selective > > receive - easy to workaround in most cases, so no major incentive to > change > > the behaviour. > > So you're saying that one should decompose the duplicates into another > table that just contains a canonical representation of each unique > combination and just reference these from the original table? > > Ben > or don't use index if cardinality of element is far less than cardinality of table. Index is useless in this case anyway. -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From valentin@REDACTED Wed Jun 11 07:43:12 2008 From: valentin@REDACTED (Valentin Micic) Date: Wed, 11 Jun 2008 07:43:12 +0200 Subject: [erlang-questions] US DOD Certification of Erlang? References: Message-ID: <00d501c8cb86$15eba4c0$6501a8c0@moneymaker2> IMHO, open source may introduce security risks that might be very dificult to quantify. Maybe a sound approach would be to see if they ever certified any open-source environment (how's Java doin', for example) and see if it is even possible. But I have a strong feeling that DoD would not certify something that does not have a "Made in U.S.A." tag, woulld they? ;-) I mean, just think about it -- if they can run it, so can Russia, Iran... where's the competitive advantage? V. ----- Original Message ----- From: "Adam Wagner" To: Sent: Wednesday, June 11, 2008 3:07 AM Subject: [erlang-questions] US DOD Certification of Erlang? >I currently work for a US defense contractor, but am relatively new to > government work. From what I've learned so far about Erlang I can see > programs on the horizon where it would be a great fit. Whenever I've > mentioned the idea to those higher up, I'm told that it would never > happen because the DOD would never "certify" it. Has anybody else > gone down this road with Erlang that could offer advice? > > Thanks, > Adam > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From alexander.lamb@REDACTED Wed Jun 11 10:23:28 2008 From: alexander.lamb@REDACTED (Alexander Lamb) Date: Wed, 11 Jun 2008 10:23:28 +0200 Subject: [erlang-questions] Use of Records in the Java API Message-ID: <59F222DE-8A8B-4011-91EE-DBEB1920C14C@rodanotech.ch> Hello, My Erlang modules are mainly a server to Java Web applications (for the time being). This means that I am making a big use of the Erlang Java API which works fine. Since my Erlang modules use Mnesia to store data coming from the Java side (and vice versa), I need to implement functions with many parameters for example to insert new data in a Mnesia table. I also need to convert tuples into actual Java objects when reading data. This would be simpler if the Java API had an OtpErlangRecord. It would make it possible to subclass it and could even be the prefered way of bridging the Erlang world with the Java world. It would also avoid breaking code each time a new attribute is needed for a Mnesia table (making it today necessary to add a parameter to the complete chain of function calls). I didn't find any mention of OtpErlangRecord in the documentation. I suspect it is because the Java API was written before records were introduced in Erlang. In a more general sense, is there any plan to enhance the functionalities of the Java API to handle more gracefully the mismatch between the loosely typed functional world of Erlang and the object oriented world of Java? I could very well imagine a Java class to handle marshalling and unmarshalling of terms at a high level using records. You would add a type description to the attributes and on the Java side you would immediately receive objects (subclasses of OtpErlangRecord for example). It could look somewhat like tools such as Cayenne or Hibernate in the SQL world. Just some thoughts. And since in two weeks Dennis Byrne is making a presentation at Erlang Exchange on that subject, maybe if Dennis is reading this post, he could consider preparing a few words in that direction for his London speech:-) That comment actually also applies if the client is in ActionScript for a Flash/Flex application or even JavaScript. Alex From ulf@REDACTED Wed Jun 11 10:54:23 2008 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 11 Jun 2008 10:54:23 +0200 Subject: [erlang-questions] US DOD Certification of Erlang? In-Reply-To: <00d501c8cb86$15eba4c0$6501a8c0@moneymaker2> References: <00d501c8cb86$15eba4c0$6501a8c0@moneymaker2> Message-ID: <8209f740806110154of9cfeeci56283917b06e4855@mail.gmail.com> The DOD was AFAIK a major user of the Nortel SSL offload accelerator, which was written in Erlang. BR, Ulf W 2008/6/11, Valentin Micic : > IMHO, open source may introduce security risks that might be very dificult > to quantify. Maybe a sound approach would be to see if they ever certified > any open-source environment (how's Java doin', for example) and see if it is > even possible. But I have a strong feeling that DoD would not certify > something that does not have a "Made in U.S.A." tag, woulld they? ;-) I > mean, just think about it -- if they can run it, so can Russia, Iran... > where's the competitive advantage? > > V. > > ----- Original Message ----- > From: "Adam Wagner" > To: > Sent: Wednesday, June 11, 2008 3:07 AM > Subject: [erlang-questions] US DOD Certification of Erlang? > > >>I currently work for a US defense contractor, but am relatively new to >> government work. From what I've learned so far about Erlang I can see >> programs on the horizon where it would be a great fit. Whenever I've >> mentioned the idea to those higher up, I'm told that it would never >> happen because the DOD would never "certify" it. Has anybody else >> gone down this road with Erlang that could offer advice? >> >> Thanks, >> Adam >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From valentin@REDACTED Wed Jun 11 11:13:14 2008 From: valentin@REDACTED (Valentin Micic) Date: Wed, 11 Jun 2008 11:13:14 +0200 Subject: [erlang-questions] US DOD Certification of Erlang? References: <00d501c8cb86$15eba4c0$6501a8c0@moneymaker2> <8209f740806110154of9cfeeci56283917b06e4855@mail.gmail.com> Message-ID: <00f901c8cba3$66b69a50$6501a8c0@moneymaker2> Wow! Two birds with one stone ("Made in U.S.A" tag and open-source! ((-:||). It seems that "those higher up" are just using "certification" as an excuse. V. ----- Original Message ----- From: "Ulf Wiger" To: "Valentin Micic" ; "Adam Wagner" ; Sent: Wednesday, June 11, 2008 10:54 AM Subject: Re: [erlang-questions] US DOD Certification of Erlang? > The DOD was AFAIK a major user of the Nortel SSL offload accelerator, > which was written in Erlang. > > BR, > Ulf W > > 2008/6/11, Valentin Micic : >> IMHO, open source may introduce security risks that might be very >> dificult >> to quantify. Maybe a sound approach would be to see if they ever >> certified >> any open-source environment (how's Java doin', for example) and see if it >> is >> even possible. But I have a strong feeling that DoD would not certify >> something that does not have a "Made in U.S.A." tag, woulld they? ;-) I >> mean, just think about it -- if they can run it, so can Russia, Iran... >> where's the competitive advantage? >> >> V. >> >> ----- Original Message ----- >> From: "Adam Wagner" >> To: >> Sent: Wednesday, June 11, 2008 3:07 AM >> Subject: [erlang-questions] US DOD Certification of Erlang? >> >> >>>I currently work for a US defense contractor, but am relatively new to >>> government work. From what I've learned so far about Erlang I can see >>> programs on the horizon where it would be a great fit. Whenever I've >>> mentioned the idea to those higher up, I'm told that it would never >>> happen because the DOD would never "certify" it. Has anybody else >>> gone down this road with Erlang that could offer advice? >>> >>> Thanks, >>> Adam >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> From bjorn@REDACTED Wed Jun 11 11:49:51 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 11 Jun 2008 11:49:51 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: <17766840.post@talk.nabble.com> References: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> <17766840.post@talk.nabble.com> Message-ID: Zvi writes: > from my PI serial benchmarks, floating point performance is weak point of > Erlang, since float values always allocated on the Heap. Actually, the Beam compiler does some limited optimization of floating point expressions to avoid storing temporary floating point values on the heap. The catch is that the compiler can only do the optimization if it can figure out that a variable is guaranteed to be a float at run-time. Currently, the compiler is not very smart about figuring out types by itself, so you'll need to use is_float/1 guards. Here is an example from Wings: http://wings.svn.sourceforge.net/viewvc/wings/trunk/e3d/e3d_vec.erl?view=markup /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From alexander.lamb@REDACTED Wed Jun 11 12:36:18 2008 From: alexander.lamb@REDACTED (Alexander Lamb) Date: Wed, 11 Jun 2008 12:36:18 +0200 Subject: [erlang-questions] Mnesia transactions and error handling good practices Message-ID: Hello, I read in the Erlang manual about avoiding using too many exceptions as well as making the initial assumption that functions work. I have a practicle (simple) case: Each time I save a user (a record in a table), I want to archive the previous values of that record. For that purpose, I have a users table and a users_archive table. To archive the user, it is simply: archive_user(User_Id) -> F = fun() -> case mnesia:read({users,User_Id}) of [User] -> mnesia:write(users_archive,User#users{timestamp = calendar:universal_time()},write); _Any -> nothing end, case mnesia:transaction(F) of {atomic, _Result} -> {ok, archive}; {aborted, Reason} -> {error, Reason} end. Now, to insert the new values, I do: set_user (User_Id ,System_Name ,Production_Type ,Email ,Last_Name,First_Name,Country,User_Status,Save_User_Id,User_Details) -> F = fun() -> archive_user(User_Id), mnesia:write(email, #users{user_id = User_Id, system_name = System_Name, production_type = Production_Type, email = Email, last_name = Last_Name, first_name = First_Name, country = Country, user_status = User_Status, timestamp = calendar:universal_time(), save_user_id = Save_User_Id, user_details = User_Details}, write) end, case mnesia:transaction(F) of {atomic, _Any} -> {ok, erlang:binary_to_list(User_Id)}; {aborted, Reason} -> {error, Reason} end. Now, obviously, this works as long as the "archive_user" function does not return {aborted, Reason}. Indeed, I don't test it. What is the good practice: 1) Not test "obvious" things (like in this example) 2) Test the response in cascade with case statements 3) Raise an exception in nested function in order to have the outer transaction fail Thanks, Alex From gleber.p@REDACTED Wed Jun 11 12:58:57 2008 From: gleber.p@REDACTED (Gleb Peregud) Date: Wed, 11 Jun 2008 12:58:57 +0200 Subject: [erlang-questions] Use of Records in the Java API In-Reply-To: <59F222DE-8A8B-4011-91EE-DBEB1920C14C@rodanotech.ch> References: <59F222DE-8A8B-4011-91EE-DBEB1920C14C@rodanotech.ch> Message-ID: <14f0e3620806110358td345347j506b8ac5787fd401@mail.gmail.com> On Wed, Jun 11, 2008 at 10:23 AM, Alexander Lamb wrote: > Hello, > > My Erlang modules are mainly a server to Java Web applications (for > the time being). > > This means that I am making a big use of the Erlang Java API which > works fine. > Since my Erlang modules use Mnesia to store data coming from the Java > side (and vice versa), I need to implement functions with many > parameters for example to insert new data in a Mnesia table. > I also need to convert tuples into actual Java objects when reading > data. > > This would be simpler if the Java API had an OtpErlangRecord. It would > make it possible to subclass it and could even be the prefered way of > bridging the Erlang world with the Java world. It would also avoid > breaking code each time a new attribute is needed for a Mnesia table > (making it today necessary to add a parameter to the complete chain of > function calls). > > I didn't find any mention of OtpErlangRecord in the documentation. I > suspect it is because the Java API was written before records were > introduced in Erlang. > > In a more general sense, is there any plan to enhance the > functionalities of the Java API to handle more gracefully the > mismatch between the loosely typed functional world of Erlang and the > object oriented world of Java? > > I could very well imagine a Java class to handle marshalling and > unmarshalling of terms at a high level using records. You would add a > type description to the attributes and on the Java side you would > immediately receive objects (subclasses of OtpErlangRecord for > example). It could look somewhat like tools such as Cayenne or > Hibernate in the SQL world. > > Just some thoughts. And since in two weeks Dennis Byrne is making a > presentation at Erlang Exchange on that subject, maybe if Dennis is > reading this post, he could consider preparing a few words in that > direction for his London speech:-) > That comment actually also applies if the client is in ActionScript > for a Flash/Flex application or even JavaScript. > > Alex Hello, There's a problem with records on the Java side, since records are comple-time module-specific features. As far i know erlang VM does not have any runtime information about records, hence it is unable to send it to Java side. Though probably it is not possible to implement OtpErlangRecord in general way. Feel free to correct me if i'm wrong. Best regards, -- Gleb Peregud http://gleber.pl/ Every minute is to be grasped. Time waits for nobody. -- Inscription on a Zen Gong From raimo+erlang-questions@REDACTED Wed Jun 11 13:31:37 2008 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 11 Jun 2008 13:31:37 +0200 Subject: [erlang-questions] : Use of Records in the Java API In-Reply-To: <14f0e3620806110358td345347j506b8ac5787fd401@mail.gmail.com> References: <59F222DE-8A8B-4011-91EE-DBEB1920C14C@rodanotech.ch> <14f0e3620806110358td345347j506b8ac5787fd401@mail.gmail.com> Message-ID: <20080611113137.GA21281@erix.ericsson.se> On Wed, Jun 11, 2008 at 12:58:57PM +0200, Gleb Peregud wrote: > On Wed, Jun 11, 2008 at 10:23 AM, Alexander Lamb > wrote: > > Hello, > > > > My Erlang modules are mainly a server to Java Web applications (for > > the time being). > > > > This means that I am making a big use of the Erlang Java API which > > works fine. > > Since my Erlang modules use Mnesia to store data coming from the Java > > side (and vice versa), I need to implement functions with many > > parameters for example to insert new data in a Mnesia table. > > I also need to convert tuples into actual Java objects when reading > > data. > > > > This would be simpler if the Java API had an OtpErlangRecord. It would > > make it possible to subclass it and could even be the prefered way of > > bridging the Erlang world with the Java world. It would also avoid > > breaking code each time a new attribute is needed for a Mnesia table > > (making it today necessary to add a parameter to the complete chain of > > function calls). > > > > I didn't find any mention of OtpErlangRecord in the documentation. I > > suspect it is because the Java API was written before records were > > introduced in Erlang. > > > > In a more general sense, is there any plan to enhance the > > functionalities of the Java API to handle more gracefully the > > mismatch between the loosely typed functional world of Erlang and the > > object oriented world of Java? > > > > I could very well imagine a Java class to handle marshalling and > > unmarshalling of terms at a high level using records. You would add a > > type description to the attributes and on the Java side you would > > immediately receive objects (subclasses of OtpErlangRecord for > > example). It could look somewhat like tools such as Cayenne or > > Hibernate in the SQL world. > > > > Just some thoughts. And since in two weeks Dennis Byrne is making a > > presentation at Erlang Exchange on that subject, maybe if Dennis is > > reading this post, he could consider preparing a few words in that > > direction for his London speech:-) > > That comment actually also applies if the client is in ActionScript > > for a Flash/Flex application or even JavaScript. > > > > Alex > > Hello, > > There's a problem with records on the Java side, since records are > comple-time module-specific features. As far i know erlang VM does not > have any runtime information about records, hence it is unable to send > it to Java side. Though probably it is not possible to implement > OtpErlangRecord in general way. > > Feel free to correct me if i'm wrong. > http://www.erlang.org/doc/reference_manual/part_frame.html 8.7 Internal Representation of Records Record expressions are translated to tuple expressions during compilation. A record defined as -record(Name, {Field1,...,FieldN}). is internally represented by the tuple {Name,Value1,...,ValueN} where each ValueI is the default value for FieldI. To each module using records, a pseudo function is added during compilation to obtain information about records: record_info(fields, Record) -> [Field] record_info(size, Record) -> Size Size is the size of the tuple representation, that is one more than the number of fields. > Best regards, > -- > Gleb Peregud > http://gleber.pl/ > > Every minute is to be grasped. > Time waits for nobody. > -- Inscription on a Zen Gong > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From mike.shaver@REDACTED Wed Jun 11 13:57:35 2008 From: mike.shaver@REDACTED (Mike Shaver) Date: Wed, 11 Jun 2008 07:57:35 -0400 Subject: [erlang-questions] clueless performance question In-Reply-To: <490D297E-0151-489D-BF26-355E9AB486B2@cs.otago.ac.nz> References: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> <490D297E-0151-489D-BF26-355E9AB486B2@cs.otago.ac.nz> Message-ID: On Tue, Jun 10, 2008 at 8:39 PM, Richard A. O'Keefe wrote: > Erlang is designed to support hot loading, the post-delivery > replacement/upgrade of modules. That means late binding, which is > *always* going to be less efficient than early binding. It's probably[*] going to be less efficient than early binding on first call, but there's no reason that it can't be equivalent to early binding on every subsequent call, until a module replacement triggers the flushing of a JIT cache or what have you. Work on tracing JITs indicates that the overhead of late binding can be made to disappear completely in the loops and functions that comprise virtually all of a program's work -- even in languages that don't have Erlang's functional goodness to assist the compilers, and permit mutation of objects in more difficult-to-handle ways than module reload. [*] unless you know that a region is going to be hot, and therefore choose to compile away its binding overhead in the background before the first call; I hear Erlang is pretty good at parallel operations. :) Mike From thomasl_erlang@REDACTED Wed Jun 11 14:09:59 2008 From: thomasl_erlang@REDACTED (Thomas Lindgren) Date: Wed, 11 Jun 2008 05:09:59 -0700 (PDT) Subject: [erlang-questions] clueless performance question In-Reply-To: <484F0635.2020003@ericsson.com> Message-ID: <194477.83170.qm@web38804.mail.mud.yahoo.com> --- "Ulf Wiger (TN/EAB)" wrote: > Of course, in many of the micro benchmarks where > Erlang > suffers, the key to high performance seems to be the > ability to perform blocking IO. This problem may > become > less of an issue if we start benchmarking on > multicores, > where it may be possible to do such things and still > not > jeopardize real-time characteristics. Micro benchmarks are seductive but normally not very important beyond bragging rights. (Blocking I/O might be important anyway, I don't know.) On the other hand, I thought the "wide finder" work of last fall was pretty interesting. (Likewise for the WF2 getting started now.) Best, Thomas From alexander.lamb@REDACTED Wed Jun 11 14:24:31 2008 From: alexander.lamb@REDACTED (Alexander Lamb) Date: Wed, 11 Jun 2008 14:24:31 +0200 Subject: [erlang-questions] : Use of Records in the Java API In-Reply-To: <20080611113137.GA21281@erix.ericsson.se> References: <59F222DE-8A8B-4011-91EE-DBEB1920C14C@rodanotech.ch> <14f0e3620806110358td345347j506b8ac5787fd401@mail.gmail.com> <20080611113137.GA21281@erix.ericsson.se> Message-ID: <13724ECC-9180-41C4-8F0F-557DAB1B954E@rodanotech.ch> Ok, I should have remembered the "records are tuples in disguise" from Joe's book. So, since there is no actual runtime documentation of a record, this makes programming from Java an issue of "conventions". It is like puting in place a protocole were client and server agree on fields and data types. Therefore, it is not possible to build the java records by introspection. What would be possible though is to "by convention" always send and receive meta-model type of information with any data exchanged between java and erlang. Thanks, Alex Le 11 juin 08 ? 13:31, Raimo Niskanen a ?crit : > On Wed, Jun 11, 2008 at 12:58:57PM +0200, Gleb Peregud wrote: >> On Wed, Jun 11, 2008 at 10:23 AM, Alexander Lamb >> wrote: >>> Hello, >>> >>> My Erlang modules are mainly a server to Java Web applications (for >>> the time being). >>> >>> This means that I am making a big use of the Erlang Java API which >>> works fine. >>> Since my Erlang modules use Mnesia to store data coming from the >>> Java >>> side (and vice versa), I need to implement functions with many >>> parameters for example to insert new data in a Mnesia table. >>> I also need to convert tuples into actual Java objects when reading >>> data. >>> >>> This would be simpler if the Java API had an OtpErlangRecord. It >>> would >>> make it possible to subclass it and could even be the prefered way >>> of >>> bridging the Erlang world with the Java world. It would also avoid >>> breaking code each time a new attribute is needed for a Mnesia table >>> (making it today necessary to add a parameter to the complete >>> chain of >>> function calls). >>> >>> I didn't find any mention of OtpErlangRecord in the documentation. I >>> suspect it is because the Java API was written before records were >>> introduced in Erlang. >>> >>> In a more general sense, is there any plan to enhance the >>> functionalities of the Java API to handle more gracefully the >>> mismatch between the loosely typed functional world of Erlang and >>> the >>> object oriented world of Java? >>> >>> I could very well imagine a Java class to handle marshalling and >>> unmarshalling of terms at a high level using records. You would >>> add a >>> type description to the attributes and on the Java side you would >>> immediately receive objects (subclasses of OtpErlangRecord for >>> example). It could look somewhat like tools such as Cayenne or >>> Hibernate in the SQL world. >>> >>> Just some thoughts. And since in two weeks Dennis Byrne is making a >>> presentation at Erlang Exchange on that subject, maybe if Dennis is >>> reading this post, he could consider preparing a few words in that >>> direction for his London speech:-) >>> That comment actually also applies if the client is in ActionScript >>> for a Flash/Flex application or even JavaScript. >>> >>> Alex >> >> Hello, >> >> There's a problem with records on the Java side, since records are >> comple-time module-specific features. As far i know erlang VM does >> not >> have any runtime information about records, hence it is unable to >> send >> it to Java side. Though probably it is not possible to implement >> OtpErlangRecord in general way. >> >> Feel free to correct me if i'm wrong. >> > http://www.erlang.org/doc/reference_manual/part_frame.html > > > > 8.7 Internal Representation of Records > > Record expressions are translated to tuple expressions during > compilation. A record defined as > > -record(Name, {Field1,...,FieldN}). > > is internally represented by the tuple > > {Name,Value1,...,ValueN} > > where each ValueI is the default value for FieldI. > > To each module using records, a pseudo function is added during > compilation to obtain information about records: > > record_info(fields, Record) -> [Field] > record_info(size, Record) -> Size > > Size is the size of the tuple representation, that is one more than > the number of fields. > > > >> Best regards, >> -- >> Gleb Peregud >> http://gleber.pl/ >> >> Every minute is to be grasped. >> Time waits for nobody. >> -- Inscription on a Zen Gong >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From mats.cronqvist@REDACTED Wed Jun 11 14:35:21 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Wed, 11 Jun 2008 14:35:21 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: <194477.83170.qm@web38804.mail.mud.yahoo.com> References: <194477.83170.qm@web38804.mail.mud.yahoo.com> Message-ID: <484FC689.7090807@gmail.com> Thomas Lindgren wrote: > > ...I thought the "wide finder" work of > last fall was pretty interesting. (Likewise for the > WF2 getting started now.) > > i tuned out of that discussion, but i was under the impression that erlang pretty much... sucked. so it was interesting to see Jeff Atwood's summary(*); Slowest Fastest Perl 44.29 1.51 Erlang 37.58 3.54 Python 41.04 4.38 OCaml 49.69 14.64 Ruby 1:43.71 50.16 mats (*) http://www.codinghorror.com/blog/archives/001131.html From vychodil.hynek@REDACTED Wed Jun 11 15:24:14 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Wed, 11 Jun 2008 15:24:14 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: <484FC689.7090807@gmail.com> References: <194477.83170.qm@web38804.mail.mud.yahoo.com> <484FC689.7090807@gmail.com> Message-ID: <4d08db370806110624s452be6edjc921c6d01754daa9@mail.gmail.com> if you look at http://wikis.sun.com/display/WideFinder/Results, there is OCaml version with 00:08:12 elapsed time and using regexp instead Boyer-Moore. This is far far away what erlang can do except with specialized mmap ad hoc linked in driver I guess. If OCaml version upgraded to Boyer-Moore prefix match I guess it can be better but not so much because this task is strong I/O bound. On Wed, Jun 11, 2008 at 2:35 PM, Mats Cronqvist wrote: > Thomas Lindgren wrote: > > > > ...I thought the "wide finder" work of > > last fall was pretty interesting. (Likewise for the > > WF2 getting started now.) > > > > > i tuned out of that discussion, but i was under the impression that > erlang pretty much... sucked. > so it was interesting to see Jeff Atwood's summary(*); > > Slowest Fastest > Perl 44.29 1.51 > Erlang > > 37.58 3.54 > Python > > 41.04 4.38 > OCaml 49.69 14.64 > Ruby > 1:43.71 50.16 > > mats > > (*) http://www.codinghorror.com/blog/archives/001131.html > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Wed Jun 11 15:42:23 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 11 Jun 2008 15:42:23 +0200 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released Message-ID: Bug fix release : otp_src_R12B-3 Build date : 2008-06-11 This is bug fix release 3 for the R12B release. You can find the README file for the release at http://www.erlang.org/download/otp_src_R12B-3.readme (this file) The source distribution and binary distribution for Windows can be downloaded from http://www.erlang.org/download/otp_src_R12B-3.tar.gz http://www.erlang.org/download/otp_win32_R12B-3.exe The distribution can also be downloaded using the BitTorrent protocol. Use the following torrent files to download the source distribution and binary distribution for Windows: http://www.erlang.org/download/otp_src_R12B-3.tar.gz.torrent http://www.erlang.org/download/otp_win32_R12B-3.exe.torrent Note: To unpack the TAR archive you need a GNU TAR compatible program. For installation instructions please read the README file that is part of the distribution. The on-line documentation can be found at: http://www.erlang.org/doc/ You can also download the complete HTML documentation or the Unix manual files http://www.erlang.org/download/otp_doc_html_R12B-3.tar.gz http://www.erlang.org/download/otp_doc_man_R12B-3.tar.gz We also want to thank those that sent us patches, suggestions and bug reports, The OTP Team -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From andreas.hillqvist@REDACTED Wed Jun 11 15:57:53 2008 From: andreas.hillqvist@REDACTED (Andreas Hillqvist) Date: Wed, 11 Jun 2008 15:57:53 +0200 Subject: [erlang-questions] Mnesia transactions and error handling good practices In-Reply-To: References: Message-ID: <8268eea30806110657y3cbfa16el4963336c1fba0c02@mail.gmail.com> 4) Test and let it crash? (The erlang way) {ok, archive} = archive_user(User_Id), My guess is that the call to mnesia:transaction(F) returns something like {aborted, badmatch} By the way, what is the cost of nested transaction in mnesia? Would it be "better practice" to keep this archive and set operation within only on transaction? Kind regards Andreas Hillqvist 2008/6/11, Alexander Lamb : > Hello, > > I read in the Erlang manual about avoiding using too many exceptions > as well as making the initial assumption that functions work. > > I have a practicle (simple) case: > > Each time I save a user (a record in a table), I want to archive the > previous values of that record. For that purpose, I have a users table > and a users_archive table. > > To archive the user, it is simply: > > archive_user(User_Id) -> > F = fun() -> > case mnesia:read({users,User_Id}) of > [User] -> mnesia:write(users_archive,User#users{timestamp = > calendar:universal_time()},write); > _Any -> nothing > end, > case mnesia:transaction(F) of > {atomic, _Result} -> {ok, archive}; > {aborted, Reason} -> {error, Reason} > end. > > Now, to insert the new values, I do: > > set_user > (User_Id > ,System_Name > ,Production_Type > ,Email > ,Last_Name,First_Name,Country,User_Status,Save_User_Id,User_Details) -> > F = fun() -> > archive_user(User_Id), > mnesia:write(email, > #users{user_id = User_Id, system_name = System_Name, > production_type = Production_Type, email = Email, last_name = > Last_Name, first_name = First_Name, country = Country, user_status = > User_Status, timestamp = calendar:universal_time(), save_user_id = > Save_User_Id, user_details = User_Details}, > write) > end, > case mnesia:transaction(F) of > {atomic, _Any} -> {ok, erlang:binary_to_list(User_Id)}; > {aborted, Reason} -> {error, Reason} > end. > > Now, obviously, this works as long as the "archive_user" function does > not return {aborted, Reason}. Indeed, I don't test it. > > What is the good practice: > > 1) Not test "obvious" things (like in this example) > 2) Test the response in cascade with case statements > 3) Raise an exception in nested function in order to have the outer > transaction fail > > Thanks, > > Alex > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From james.hague@REDACTED Wed Jun 11 16:29:35 2008 From: james.hague@REDACTED (James Hague) Date: Wed, 11 Jun 2008 09:29:35 -0500 Subject: [erlang-questions] clueless performance question In-Reply-To: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> References: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> Message-ID: I find Erlang to be impressively fast in comparison with other dynamic languages. For general computation Erlang typically outperforms Python by a wide margin, for example. The weak points--performance-wise--in Erlang tend to be: * Algorithms involving heavy use of destructive array updates * Floating point math * Regular expressions * Line-oriented I/O James From thomasl_erlang@REDACTED Wed Jun 11 16:33:14 2008 From: thomasl_erlang@REDACTED (Thomas Lindgren) Date: Wed, 11 Jun 2008 07:33:14 -0700 (PDT) Subject: [erlang-questions] clueless performance question In-Reply-To: <4d08db370806110623p709ffb5ehc52d835c657df4db@mail.gmail.com> Message-ID: <969357.86477.qm@web38804.mail.mud.yahoo.com> (Just to check, Mats is writing about the WF1 results and Hynek about WF2, right?) Best, Thomas --- Hynek Vychodil wrote: > if you look at > http://wikis.sun.com/display/WideFinder/Results, > there is > OCaml version with 00:08:12 elapsed time > and using regexp instead Boyer-Moore. This is far > far away what erlang can > do except with specialized mmap ad hoc linked in > driver I guess. > If OCaml version upgraded to Boyer-Moore prefix > match I guess it can be > better but not so much because this task is strong > I/O bound. > > On Wed, Jun 11, 2008 at 2:35 PM, Mats Cronqvist > > wrote: > > > Thomas Lindgren wrote: > > > > > > ...I thought the "wide finder" work of > > > last fall was pretty interesting. (Likewise for > the > > > WF2 getting started now.) > > > > > > > > i tuned out of that discussion, but i was under > the impression that > > erlang pretty much... sucked. > > so it was interesting to see Jeff Atwood's > summary(*); > > > > Slowest Fastest > > Perl > 44.29 1.51 > > Erlang > > > > > 37.58 3.54 > > Python > > > > > 41.04 4.38 > > OCaml > 49.69 14.64 > > Ruby > > > 1:43.71 50.16 > > > > mats > > > > (*) > http://www.codinghorror.com/blog/archives/001131.html > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > > > > -- > --Hynek (Pichi) Vychodil > From luna@REDACTED Wed Jun 11 16:38:30 2008 From: luna@REDACTED (Daniel Luna) Date: Wed, 11 Jun 2008 16:38:30 +0200 (CEST) Subject: [erlang-questions] Mnesia transactions and error handling good practices In-Reply-To: <8268eea30806110657y3cbfa16el4963336c1fba0c02@mail.gmail.com> References: <8268eea30806110657y3cbfa16el4963336c1fba0c02@mail.gmail.com> Message-ID: On Wed, 11 Jun 2008, Andreas Hillqvist wrote: > By the way, what is the cost of nested transaction in mnesia? > Would it be "better practice" to keep this archive and set operation > within only on transaction? The cost of a nested transaction is proportional to the size of both of the transactions. We rewrote some code at Kreditor into using nested transactions (which is more readable), but had to throw it away. A test which usually takes about 20-30 minutes on a desktop machine wasn't finished after 10 hours (when the test was aborted). /Luna -- Daniel Luna | Top reasons that I have a beard: luna@REDACTED | a) Laziness. http://www.update.uu.se/~luna/ | b) I can. Don't look at my homepage (it stinks).| c) I can get away with it. From ulf.wiger@REDACTED Wed Jun 11 16:44:35 2008 From: ulf.wiger@REDACTED (Ulf Wiger (TN/EAB)) Date: Wed, 11 Jun 2008 16:44:35 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: References: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> Message-ID: <484FE4D3.7020807@ericsson.com> James Hague skrev: > > The weak points--performance-wise--in Erlang tend to be: > > * Algorithms involving heavy use of destructive array updates > * Floating point math > * Regular expressions > * Line-oriented I/O ...and the problem with line-oriented I/O is that it is either too inefficient (io:get_line()), or way, way too efficient (the user driver in line-oriented mode). Specifically, the second alternative suffers from lack of flow control, which means that even the tightest of loops in erlang code is unable to keep up with a sufficiently large stream of very short lines (which of course was exactly the case in the shootout). http://www.erlang.org/pipermail/erlang-questions/2007-June/027557.html BR, Ulf W From thomasl_erlang@REDACTED Wed Jun 11 16:51:18 2008 From: thomasl_erlang@REDACTED (Thomas Lindgren) Date: Wed, 11 Jun 2008 07:51:18 -0700 (PDT) Subject: [erlang-questions] clueless performance question In-Reply-To: Message-ID: <334820.98238.qm@web38803.mail.mud.yahoo.com> --- Mike Shaver wrote: > Work on tracing JITs indicates that the overhead of > late binding can > be made to disappear completely in the loops and > functions that > comprise virtually all of a program's work -- even > in languages that > don't have Erlang's functional goodness to assist > the compilers, and > permit mutation of objects in more > difficult-to-handle ways than > module reload. On one hand, I'd say Erlang is probably more straightforward to compile than at least some OO-languages, and Hipe thoughtfully has provided the infrastructure for JIT native compilation, etc etc. Java has already made great strides regarding performance. So there's clearly hope for further progress. On the other hand, these techniques require serious manpower to implement and in practice there's just a handful of people actually working on Erlang implementation. I don't even know of _anyone_ currently working on general compiler issues except Bj?rn? So clearly things will get prioritized, and I guess progress will be slow ... (That said, I don't hear many complaints about performance these days. Perhaps time is better spent on other issues.) Best, Thomas From mats.cronqvist@REDACTED Wed Jun 11 17:02:51 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Wed, 11 Jun 2008 17:02:51 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: <969357.86477.qm@web38804.mail.mud.yahoo.com> References: <969357.86477.qm@web38804.mail.mud.yahoo.com> Message-ID: <484FE91B.5090109@gmail.com> Thomas Lindgren wrote: > (Just to check, Mats is writing about the WF1 results > and Hynek about WF2, right?) i was indeed referring to a WF1 writeup. mats From vychodil.hynek@REDACTED Wed Jun 11 17:10:49 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Wed, 11 Jun 2008 17:10:49 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: <969357.86477.qm@web38804.mail.mud.yahoo.com> References: <4d08db370806110623p709ffb5ehc52d835c657df4db@mail.gmail.com> <969357.86477.qm@web38804.mail.mud.yahoo.com> Message-ID: <4d08db370806110810q7aecda5chf0524fea89280b0c@mail.gmail.com> On Wed, Jun 11, 2008 at 4:33 PM, Thomas Lindgren wrote: > (Just to check, Mats is writing about the WF1 results > and Hynek about WF2, right?) Yes, I wrote about WF2 and Mats about WF1. I was attended to WF1 but for WF2 I'm so busy, but may be :-) > > > Best, > Thomas > > --- Hynek Vychodil wrote: > > > if you look at > > http://wikis.sun.com/display/WideFinder/Results, > > there is > > OCaml version with 00:08:12 elapsed time > > and using regexp instead Boyer-Moore. This is far > > far away what erlang can > > do except with specialized mmap ad hoc linked in > > driver I guess. > > If OCaml version upgraded to Boyer-Moore prefix > > match I guess it can be > > better but not so much because this task is strong > > I/O bound. > > > > On Wed, Jun 11, 2008 at 2:35 PM, Mats Cronqvist > > > > wrote: > > > > > Thomas Lindgren wrote: > > > > > > > > ...I thought the "wide finder" work of > > > > last fall was pretty interesting. (Likewise for > > the > > > > WF2 getting started now.) > > > > > > > > > > > i tuned out of that discussion, but i was under > > the impression that > > > erlang pretty much... sucked. > > > so it was interesting to see Jeff Atwood's > > summary(*); > > > > > > Slowest Fastest > > > Perl > > 44.29 1.51 > > > Erlang > > > > > > > > > 37.58 3.54 > > > Python > > > > > > > > > 41.04 4.38 > > > OCaml > > 49.69 14.64 > > > Ruby > > > > > > 1:43.71 50.16 > > > > > > mats > > > > > > (*) > > > http://www.codinghorror.com/blog/archives/001131.html > > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > > > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > > > -- > > --Hynek (Pichi) Vychodil > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew@REDACTED Wed Jun 11 17:27:39 2008 From: matthew@REDACTED (Matthew Dempsky) Date: Wed, 11 Jun 2008 08:27:39 -0700 Subject: [erlang-questions] [erlang-announce] Erlang/OTP R12B-3 has been released In-Reply-To: References: Message-ID: WHOOO!! On Wed, Jun 11, 2008 at 6:42 AM, Bjorn Gustavsson wrote: > Bug fix release : otp_src_R12B-3 > Build date : 2008-06-11 > > This is bug fix release 3 for the R12B release. > > You can find the README file for the release at > > http://www.erlang.org/download/otp_src_R12B-3.readme (this file) > > The source distribution and binary distribution for Windows can be > downloaded from > > http://www.erlang.org/download/otp_src_R12B-3.tar.gz > http://www.erlang.org/download/otp_win32_R12B-3.exe > > The distribution can also be downloaded using the BitTorrent > protocol. Use the following torrent files to download the source > distribution and binary distribution for Windows: > > http://www.erlang.org/download/otp_src_R12B-3.tar.gz.torrent > http://www.erlang.org/download/otp_win32_R12B-3.exe.torrent > > Note: To unpack the TAR archive you need a GNU TAR compatible program. > > For installation instructions please read the README file that is part > of the distribution. > > The on-line documentation can be found at: http://www.erlang.org/doc/ > You can also download the complete HTML documentation or the Unix manual files > > http://www.erlang.org/download/otp_doc_html_R12B-3.tar.gz > http://www.erlang.org/download/otp_doc_man_R12B-3.tar.gz > > We also want to thank those that sent us patches, suggestions and bug > reports, > > The OTP Team > > -- > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-announce mailing list > erlang-announce@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-announce > From kevin@REDACTED Wed Jun 11 17:32:49 2008 From: kevin@REDACTED (Kevin Scaldeferri) Date: Wed, 11 Jun 2008 08:32:49 -0700 Subject: [erlang-questions] US DOD Certification of Erlang? In-Reply-To: <00d501c8cb86$15eba4c0$6501a8c0@moneymaker2> References: <00d501c8cb86$15eba4c0$6501a8c0@moneymaker2> Message-ID: <78C38FC1-3F7C-4703-9597-36626CA794B1@scaldeferri.com> On Jun 10, 2008, at 10:43 PM, Valentin Micic wrote: > IMHO, open source may introduce security risks that might be very > dificult > to quantify. Isn't the same true of proprietary systems? > Maybe a sound approach would be to see if they ever certified > any open-source environment (how's Java doin', for example) and see > if it is > even possible. But I have a strong feeling that DoD would not certify > something that does not have a "Made in U.S.A." tag, woulld they? ;-) Haskell / GHC is open source and not really "Made in the USA". I'm not sure of the exact status currently, but I know there are people working on putting it through the DOD certification process. -kevin From kevin@REDACTED Wed Jun 11 17:41:40 2008 From: kevin@REDACTED (Kevin Scaldeferri) Date: Wed, 11 Jun 2008 08:41:40 -0700 Subject: [erlang-questions] clueless performance question In-Reply-To: <484FC689.7090807@gmail.com> References: <194477.83170.qm@web38804.mail.mud.yahoo.com> <484FC689.7090807@gmail.com> Message-ID: On Jun 11, 2008, at 5:35 AM, Mats Cronqvist wrote: > Thomas Lindgren wrote: >> >> ...I thought the "wide finder" work of >> last fall was pretty interesting. (Likewise for the >> WF2 getting started now.) >> >> > i tuned out of that discussion, but i was under the impression that > erlang pretty much... sucked. > so it was interesting to see Jeff Atwood's summary(*); I think it's more accurate to say that early, "idiomatic" Erlang implementations sucked. Once you give up on using line-oriented I/O and write a few hundred lines of code using raw I/O, it gets fast. For the record, though, the winning Perl implementation was a couple dozen lines and, while clever, not obfuscated. -k From ulf@REDACTED Wed Jun 11 18:01:08 2008 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 11 Jun 2008 18:01:08 +0200 Subject: [erlang-questions] US DOD Certification of Erlang? In-Reply-To: References: <00d501c8cb86$15eba4c0$6501a8c0@moneymaker2> <8209f740806110154of9cfeeci56283917b06e4855@mail.gmail.com> <00f901c8cba3$66b69a50$6501a8c0@moneymaker2> Message-ID: <8209f740806110901j6dc93e24x12233c589b5cebb8@mail.gmail.com> It's always good if you can focus on the "what" rather than the "how". BTW, I just caught this interesting proposal from SIGPLAN on how to upgrade the importance of functional programming in the Harvard Computer Science curriculum: http://wiki.acm.org/cs2001/index.php?title=SIGPLAN_Proposal Knowledge Unit Current Proposed PF4 Recursion 5 2 PF5 Event-driven programming 4 2 PL1 Overview of PL 2 0 PL2 Virtual Machines 1 0 PL3 Language Translation 2 0 PL6 Object-oriented programming 10 10 PL7 Functional Programming 0 10 Total 24 24 Here are some of the arguments why: " # A functional programming language illustrates a way of programming that departs dramatically from the procedural and OO languages typical of CS1 and CS2 courses, and so illustrates alternate ways of expressing computation, alternate models of computation, alternate ways of thinking about problems, etc. leading to a better educated graduate. Other languages ? say, logic languages ? could also serve this purpose; FP seems to have the highest leverage. # Studying a functional programming language introduces important computer science concepts ? recursion, higher order functions, statelessness, sideeffects, lazy evaluation, etc. ? that arise in many contexts and situations. It's simply a good place acquire these ideas. # The substantial differences between the typical CS1/2 languages and FP illustrate a diversity among languages that shows students how selecting a programming language can affect the ease/difficulty of finding a solution to a programming problem, and implies that learning and applying new languages can be very advantageous. " BR, Ulf W 2008/6/11 Adam Wagner : > Thanks for the feedback Valintin and Ulf. I think you're correct that > in my case it's more of an excuse than a legitimate concern. I know > that there are costs/risks in getting new (uncertified) and > open-source technologies accepted for use on certain > platforms/environments. However, in this case I think it's reluctance > by management to stray too far from something familiar to their pool > of Java/C++ developers. I was curious if someone had explicitly dealt > with the certification issue for Erlang, but it's interesting to know > the DoD has (unknowingly?) been using products implemented in Erlang. > I'm planning to do a proof of concept with Erlang and see what I can > get away with. > > Thanks again! > - Adam > > On Wed, Jun 11, 2008 at 2:13 AM, Valentin Micic wrote: >> Wow! >> Two birds with one stone ("Made in U.S.A" tag and open-source! ((-:||). >> It seems that "those higher up" are just using "certification" as an excuse. >> >> V. >> ----- Original Message ----- From: "Ulf Wiger" >> To: "Valentin Micic" ; "Adam Wagner" >> ; >> Sent: Wednesday, June 11, 2008 10:54 AM >> Subject: Re: [erlang-questions] US DOD Certification of Erlang? >> >> >>> The DOD was AFAIK a major user of the Nortel SSL offload accelerator, >>> which was written in Erlang. >>> >>> BR, >>> Ulf W >>> >>> 2008/6/11, Valentin Micic : >>>> >>>> IMHO, open source may introduce security risks that might be very >>>> dificult >>>> to quantify. Maybe a sound approach would be to see if they ever >>>> certified >>>> any open-source environment (how's Java doin', for example) and see if it >>>> is >>>> even possible. But I have a strong feeling that DoD would not certify >>>> something that does not have a "Made in U.S.A." tag, woulld they? ;-) I >>>> mean, just think about it -- if they can run it, so can Russia, Iran... >>>> where's the competitive advantage? >>>> >>>> V. >>>> >>>> ----- Original Message ----- >>>> From: "Adam Wagner" >>>> To: >>>> Sent: Wednesday, June 11, 2008 3:07 AM >>>> Subject: [erlang-questions] US DOD Certification of Erlang? >>>> >>>> >>>>> I currently work for a US defense contractor, but am relatively new to >>>>> government work. From what I've learned so far about Erlang I can see >>>>> programs on the horizon where it would be a great fit. Whenever I've >>>>> mentioned the idea to those higher up, I'm told that it would never >>>>> happen because the DOD would never "certify" it. Has anybody else >>>>> gone down this road with Erlang that could offer advice? >>>>> >>>>> Thanks, >>>>> Adam >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>> >> >> > From mats.cronqvist@REDACTED Wed Jun 11 18:20:22 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Wed, 11 Jun 2008 18:20:22 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: References: <194477.83170.qm@web38804.mail.mud.yahoo.com> <484FC689.7090807@gmail.com> Message-ID: <484FFB46.40500@gmail.com> Kevin Scaldeferri wrote: > > On Jun 11, 2008, at 5:35 AM, Mats Cronqvist wrote: > >> Thomas Lindgren wrote: >>> >>> ...I thought the "wide finder" work of >>> last fall was pretty interesting. (Likewise for the >>> WF2 getting started now.) >>> >>> >> i tuned out of that discussion, but i was under the impression that >> erlang pretty much... sucked. >> so it was interesting to see Jeff Atwood's summary(*); > > I think it's more accurate to say that early, "idiomatic" Erlang > implementations sucked. Once you give up on using line-oriented I/O > and write a few hundred lines of code using raw I/O, it gets fast. > For the record, though, the winning Perl implementation was a couple > dozen lines and, while clever, not obfuscated. i was surprised that the erlang implementation was able to get within a factor of 10 for what's pretty much the canonical perl problem. kind of like implementing a phone switch with 10,000 concurrent tasks and get nine nines reliability in perl. mats From vychodil.hynek@REDACTED Wed Jun 11 18:21:03 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Wed, 11 Jun 2008 18:21:03 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: References: <194477.83170.qm@web38804.mail.mud.yahoo.com> <484FC689.7090807@gmail.com> Message-ID: <4d08db370806110921k49b98a2drfdbf5bdadd4565f4@mail.gmail.com> What I can write about WF1 is, this was a little bit ... well I write it clearly: "stupid" because WF1 was measured with data file cached. It is why this Perl implementation won. This implementation mmap file and than forked processes around this file and then rush. Each process reads it's mmap forked copy which in normal cause random access to disk and must end with far slowest solution. On Wed, Jun 11, 2008 at 5:41 PM, Kevin Scaldeferri wrote: > > On Jun 11, 2008, at 5:35 AM, Mats Cronqvist wrote: > > > Thomas Lindgren wrote: > >> > >> ...I thought the "wide finder" work of > >> last fall was pretty interesting. (Likewise for the > >> WF2 getting started now.) > >> > >> > > i tuned out of that discussion, but i was under the impression that > > erlang pretty much... sucked. > > so it was interesting to see Jeff Atwood's summary(*); > > I think it's more accurate to say that early, "idiomatic" Erlang > implementations sucked. Once you give up on using line-oriented I/O > and write a few hundred lines of code using raw I/O, it gets fast. > For the record, though, the winning Perl implementation was a couple > dozen lines and, while clever, not obfuscated. > > -k > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Wed Jun 11 18:28:21 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Wed, 11 Jun 2008 18:28:21 +0200 Subject: [erlang-questions] Mnesia transactions and error handling good practices In-Reply-To: References: <8268eea30806110657y3cbfa16el4963336c1fba0c02@mail.gmail.com> Message-ID: <4d08db370806110928p48c50aecs6015a290cc06694@mail.gmail.com> On Wed, Jun 11, 2008 at 4:38 PM, Daniel Luna wrote: > On Wed, 11 Jun 2008, Andreas Hillqvist wrote: > > By the way, what is the cost of nested transaction in mnesia? > > Would it be "better practice" to keep this archive and set operation > > within only on transaction? > > The cost of a nested transaction is proportional to the size of both of > the transactions. > > We rewrote some code at Kreditor into using nested transactions (which is > more readable), but had to throw it away. A test which usually takes > about 20-30 minutes on a desktop machine wasn't finished after 10 hours > (when the test was aborted). > > /Luna > -- > Daniel Luna | Top reasons that I have a beard: > luna@REDACTED | a) Laziness. > http://www.update.uu.se/~luna/ | > b) I can. > Don't look at my homepage (it stinks).| c) I can get away with it. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > Just check, It means that if outer transaction is loose coupled to inner (not make intensive data load from inner to outer), there is no problem with transaction? -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf.wiger@REDACTED Wed Jun 11 18:35:05 2008 From: ulf.wiger@REDACTED (Ulf Wiger (TN/EAB)) Date: Wed, 11 Jun 2008 18:35:05 +0200 Subject: [erlang-questions] Mnesia transactions and error handling good practices In-Reply-To: <4d08db370806110928p48c50aecs6015a290cc06694@mail.gmail.com> References: <8268eea30806110657y3cbfa16el4963336c1fba0c02@mail.gmail.com> <4d08db370806110928p48c50aecs6015a290cc06694@mail.gmail.com> Message-ID: <484FFEB9.2020300@ericsson.com> Hynek Vychodil skrev: > On Wed, Jun 11, 2008 at 4:38 PM, Daniel Luna > wrote: > > On Wed, 11 Jun 2008, Andreas Hillqvist wrote: > > By the way, what is the cost of nested transaction in mnesia? > > Would it be "better practice" to keep this archive and set operation > > within only on transaction? > > The cost of a nested transaction is proportional to the size of both of > the transactions. ... > Just check, It means that if outer transaction is loose coupled to inner > (not make intensive data load from inner to outer), there is no problem > with transaction? No, all the data from the outer transaction is copied into the inner transaction, and all the committed data from the inner transaction is copied back into the outer. BR, Ulf W From kevin@REDACTED Wed Jun 11 18:42:04 2008 From: kevin@REDACTED (Kevin Scaldeferri) Date: Wed, 11 Jun 2008 09:42:04 -0700 Subject: [erlang-questions] clueless performance question In-Reply-To: <4d08db370806110921k49b98a2drfdbf5bdadd4565f4@mail.gmail.com> References: <194477.83170.qm@web38804.mail.mud.yahoo.com> <484FC689.7090807@gmail.com> <4d08db370806110921k49b98a2drfdbf5bdadd4565f4@mail.gmail.com> Message-ID: <0AB58BC9-B473-498E-B474-1603F22B908C@scaldeferri.com> On Jun 11, 2008, at 9:21 AM, Hynek Vychodil wrote: > What I can write about WF1 is, this was a little bit ... well I > write it clearly: "stupid" because WF1 was measured with data file > cached. It is why this Perl implementation won. This implementation > mmap file and than forked processes around this file and then rush. > Each process reads it's mmap forked copy which in normal cause > random access to disk and must end with far slowest solution. > > I don't think it can really be considered cheating to take advantage of features provided by the OS. -kevin From dougedmunds@REDACTED Wed Jun 11 19:08:27 2008 From: dougedmunds@REDACTED (DougEdmunds) Date: Wed, 11 Jun 2008 10:08:27 -0700 Subject: [erlang-questions] ets:file2tab/1 question Message-ID: <4850068B.5050001@gmail.com> In the docs for the ets module: file2tab(Filename) -> {ok,Tab} | {error,Reason} Types: Filename = string() | atom() Tab = tid() | atom() Reason = term() Can someone provide an example where Tab returns an atom() and not a tid()? Thanks. =dae From hayeah@REDACTED Wed Jun 11 19:21:48 2008 From: hayeah@REDACTED (Howard Yeh) Date: Wed, 11 Jun 2008 10:21:48 -0700 Subject: [erlang-questions] [erlang-announce] Erlang/OTP R12B-3 has been released In-Reply-To: References: Message-ID: yay for regex! On 6/11/08, Matthew Dempsky wrote: > WHOOO!! > > > On Wed, Jun 11, 2008 at 6:42 AM, Bjorn Gustavsson > wrote: > > Bug fix release : otp_src_R12B-3 > > Build date : 2008-06-11 > > > > This is bug fix release 3 for the R12B release. > > > > You can find the README file for the release at > > > > http://www.erlang.org/download/otp_src_R12B-3.readme (this file) > > > > The source distribution and binary distribution for Windows can be > > downloaded from > > > > http://www.erlang.org/download/otp_src_R12B-3.tar.gz > > http://www.erlang.org/download/otp_win32_R12B-3.exe > > > > The distribution can also be downloaded using the BitTorrent > > protocol. Use the following torrent files to download the source > > distribution and binary distribution for Windows: > > > > http://www.erlang.org/download/otp_src_R12B-3.tar.gz.torrent > > http://www.erlang.org/download/otp_win32_R12B-3.exe.torrent > > > > Note: To unpack the TAR archive you need a GNU TAR compatible program. > > > > For installation instructions please read the README file that is part > > of the distribution. > > > > The on-line documentation can be found at: http://www.erlang.org/doc/ > > You can also download the complete HTML documentation or the Unix manual files > > > > http://www.erlang.org/download/otp_doc_html_R12B-3.tar.gz > > http://www.erlang.org/download/otp_doc_man_R12B-3.tar.gz > > > > We also want to thank those that sent us patches, suggestions and bug > > reports, > > > > The OTP Team > > > > -- > > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB > > _______________________________________________ > > > erlang-announce mailing list > > erlang-announce@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-announce > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- hayeah.wordpress.com --metacircular thinking From adam@REDACTED Wed Jun 11 16:49:27 2008 From: adam@REDACTED (Adam Wagner) Date: Wed, 11 Jun 2008 07:49:27 -0700 Subject: [erlang-questions] US DOD Certification of Erlang? In-Reply-To: <00f901c8cba3$66b69a50$6501a8c0@moneymaker2> References: <00d501c8cb86$15eba4c0$6501a8c0@moneymaker2> <8209f740806110154of9cfeeci56283917b06e4855@mail.gmail.com> <00f901c8cba3$66b69a50$6501a8c0@moneymaker2> Message-ID: Thanks for the feedback Valintin and Ulf. I think you're correct that in my case it's more of an excuse than a legitimate concern. I know that there are costs/risks in getting new (uncertified) and open-source technologies accepted for use on certain platforms/environments. However, in this case I think it's reluctance by management to stray too far from something familiar to their pool of Java/C++ developers. I was curious if someone had explicitly dealt with the certification issue for Erlang, but it's interesting to know the DoD has (unknowingly?) been using products implemented in Erlang. I'm planning to do a proof of concept with Erlang and see what I can get away with. Thanks again! - Adam On Wed, Jun 11, 2008 at 2:13 AM, Valentin Micic wrote: > Wow! > Two birds with one stone ("Made in U.S.A" tag and open-source! ((-:||). > It seems that "those higher up" are just using "certification" as an excuse. > > V. > ----- Original Message ----- From: "Ulf Wiger" > To: "Valentin Micic" ; "Adam Wagner" > ; > Sent: Wednesday, June 11, 2008 10:54 AM > Subject: Re: [erlang-questions] US DOD Certification of Erlang? > > >> The DOD was AFAIK a major user of the Nortel SSL offload accelerator, >> which was written in Erlang. >> >> BR, >> Ulf W >> >> 2008/6/11, Valentin Micic : >>> >>> IMHO, open source may introduce security risks that might be very >>> dificult >>> to quantify. Maybe a sound approach would be to see if they ever >>> certified >>> any open-source environment (how's Java doin', for example) and see if it >>> is >>> even possible. But I have a strong feeling that DoD would not certify >>> something that does not have a "Made in U.S.A." tag, woulld they? ;-) I >>> mean, just think about it -- if they can run it, so can Russia, Iran... >>> where's the competitive advantage? >>> >>> V. >>> >>> ----- Original Message ----- >>> From: "Adam Wagner" >>> To: >>> Sent: Wednesday, June 11, 2008 3:07 AM >>> Subject: [erlang-questions] US DOD Certification of Erlang? >>> >>> >>>> I currently work for a US defense contractor, but am relatively new to >>>> government work. From what I've learned so far about Erlang I can see >>>> programs on the horizon where it would be a great fit. Whenever I've >>>> mentioned the idea to those higher up, I'm told that it would never >>>> happen because the DOD would never "certify" it. Has anybody else >>>> gone down this road with Erlang that could offer advice? >>>> >>>> Thanks, >>>> Adam >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> > > From per.melin@REDACTED Wed Jun 11 19:53:08 2008 From: per.melin@REDACTED (Per Melin) Date: Wed, 11 Jun 2008 19:53:08 +0200 Subject: [erlang-questions] ets:file2tab/1 question In-Reply-To: <4850068B.5050001@gmail.com> References: <4850068B.5050001@gmail.com> Message-ID: 2008/6/11 DougEdmunds : > In the docs for the ets module: > > file2tab(Filename) -> {ok,Tab} | {error,Reason} > > Types: > > Filename = string() | atom() > Tab = tid() | atom() > Reason = term() > > Can someone provide an example where > Tab returns an atom() and not a tid()? 1> ets:new(foo, [named_table]). foo 2> ets:tab2file(foo, "foo.ets"). ok 3> ets:delete(foo). true 4> ets:file2tab("foo.ets"). {ok,foo} From alexander.lamb@REDACTED Wed Jun 11 21:03:56 2008 From: alexander.lamb@REDACTED (Alexander Lamb) Date: Wed, 11 Jun 2008 21:03:56 +0200 Subject: [erlang-questions] Mnesia transactions and error handling good practices In-Reply-To: <484FFEB9.2020300@ericsson.com> References: <8268eea30806110657y3cbfa16el4963336c1fba0c02@mail.gmail.com> <4d08db370806110928p48c50aecs6015a290cc06694@mail.gmail.com> <484FFEB9.2020300@ericsson.com> Message-ID: <27AB1A99-34B0-4174-BAC0-A0D448D236A1@rodanotech.ch> Ok, so on the one hand, the Erlang principles (and nested transactions) encourage you to divide your code in small functions. On the other hand, performance issues might be a problem with nested transactions. Now, there are 3 ways of handling this: 1) use nested transactions, at least for small computations, and ignore performance issues (if possible) 2) write larger functions which cumulate the functionalities normally split among several functions 3) write "no transaction" functions Before I understood the concept of nested transactions, I did a mix of 2 and 3. 3 means I write functions which imply they are going to be called from within a transaction. This works actually rather well, except you end up for several functions by writing to versions of each: one with a transaction and one without a transaction. Solution 2 is not very good because it goes against the principle of atomicity in the code. I don't want code doing the same kind of task duplicated in several functions. Obviously, in the example I gave, it may not be very important, but in some cases it ended up with functions 40 lines long. So I guess I will stick to solution 1 for small transactions and rely on a "bad match" exception for error handling, which seems a rather good compromise. Indeed, systematically testing for {aborted, Reason} even for transaction which should just not fail seems cumbersome in the code. Thanks for the inputs, Alex Le 11 juin 08 ? 18:35, Ulf Wiger (TN/EAB) a ?crit : > Hynek Vychodil skrev: >> On Wed, Jun 11, 2008 at 4:38 PM, Daniel Luna > > wrote: >> >> On Wed, 11 Jun 2008, Andreas Hillqvist wrote: >>> By the way, what is the cost of nested transaction in mnesia? >>> Would it be "better practice" to keep this archive and set operation >>> within only on transaction? >> >> The cost of a nested transaction is proportional to the size of >> both of >> the transactions. > ... >> Just check, It means that if outer transaction is loose coupled to >> inner >> (not make intensive data load from inner to outer), there is no >> problem >> with transaction? > > No, all the data from the outer transaction is copied into the inner > transaction, and all the committed data from the inner transaction > is copied back into the outer. > > BR, > Ulf W > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From alexander.lamb@REDACTED Wed Jun 11 21:26:20 2008 From: alexander.lamb@REDACTED (Alexander Lamb) Date: Wed, 11 Jun 2008 21:26:20 +0200 Subject: [erlang-questions] Mnesia transactions and error handling good practices In-Reply-To: <484FFEB9.2020300@ericsson.com> References: <8268eea30806110657y3cbfa16el4963336c1fba0c02@mail.gmail.com> <4d08db370806110928p48c50aecs6015a290cc06694@mail.gmail.com> <484FFEB9.2020300@ericsson.com> Message-ID: Replying to myself, another option to avoid using several nested transactions would be to test in the functions if there is an ongoing transaction. If so, don't create a transaction, if not, create a transaction. The same kind of logic could be applied by passing a parameter to those functions indicating if we need a transaction or not. Alex Le 11 juin 08 ? 18:35, Ulf Wiger (TN/EAB) a ?crit : > Hynek Vychodil skrev: >> On Wed, Jun 11, 2008 at 4:38 PM, Daniel Luna > > wrote: >> >> On Wed, 11 Jun 2008, Andreas Hillqvist wrote: >>> By the way, what is the cost of nested transaction in mnesia? >>> Would it be "better practice" to keep this archive and set operation >>> within only on transaction? >> >> The cost of a nested transaction is proportional to the size of >> both of >> the transactions. > ... >> Just check, It means that if outer transaction is loose coupled to >> inner >> (not make intensive data load from inner to outer), there is no >> problem >> with transaction? > > No, all the data from the outer transaction is copied into the inner > transaction, and all the committed data from the inner transaction > is copied back into the outer. > > BR, > Ulf W > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From sebastian.bello@REDACTED Wed Jun 11 21:24:12 2008 From: sebastian.bello@REDACTED (Sebastian Bello) Date: Wed, 11 Jun 2008 16:24:12 -0300 Subject: [erlang-questions] SNMP questions In-Reply-To: <484E7AA1.8020605@inswitch.us> References: <484E7AA1.8020605@inswitch.us> Message-ID: <4850265C.6040702@inswitch.us> Any hint? Sebastian- Sebastian Bello escribi?: > Hi list, > > we are currently trying to use the Erlang SNMP toolkit and have some > questions: > 1- we are using /snmpa:send_trap/ to send traps from within the master > agent node. How should this be done from a different node? Maybe using > Erlang communication to send the traps' information to the master > agent node and then issuing a /snmpa:send_trap/ from it? Or is there a > way to specify a remote node in /snmpa:send_trap/? > > 2- in a distributed application, how should the MIB represent data > from applications running on different nodes? > > Thanks, > Sebastian- > > __________ NOD32 3172 (20080610) Information __________ > > This message was checked by NOD32 antivirus system. > http://www.eset.com > ------------------------------------------------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > __________ NOD32 3172 (20080610) Information __________ > > This message was checked by NOD32 antivirus system. > http://www.eset.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbj@REDACTED Wed Jun 11 22:06:30 2008 From: mbj@REDACTED (Martin Bjorklund) Date: Wed, 11 Jun 2008 22:06:30 +0200 (CEST) Subject: [erlang-questions] SNMP questions In-Reply-To: <4850265C.6040702@inswitch.us> References: <484E7AA1.8020605@inswitch.us> <4850265C.6040702@inswitch.us> Message-ID: <20080611.220630.11876259.mbj@tail-f.com> Hi, Sebastian Bello wrote: > > 1- we are using /snmpa:send_trap/ to send traps from within the > > master agent node. How should this be done from a different node? > > Maybe using Erlang communication to send the traps' information to > > the master agent node and then issuing a /snmpa:send_trap/ from > > it? Or is there a way to specify a remote node in > > /snmpa:send_trap/? Use the normal distribution. Either do a rpc to the master node, or use global to register the master agent. > > 2- in a distributed application, how should the MIB represent data > > from applications running on different nodes? This is a MIB design question - the nodes in the cluster can be explicitly modelled in the MIBs or not. /martin From gijsbert_de_haan@REDACTED Thu Jun 12 00:00:16 2008 From: gijsbert_de_haan@REDACTED (Gijsbert de Haan) Date: Wed, 11 Jun 2008 18:00:16 -0400 Subject: [erlang-questions] terminate() not finishing after application:stop() In-Reply-To: <484E858F.9010808@ericsson.com> References: <1213105147.14614.1257699575@webmail.messagingengine.com> <484E858F.9010808@ericsson.com> Message-ID: <1213221616.12215.1257991023@webmail.messagingengine.com> Hi, My terminate() function is not finishing after application:stop(myapp). I am wondering if my terminate() function takes too long and is getting killed before it finishes. I have tried to increase the Shutdown timeout in the child specification but it seems to have no influence. The process is started by supervisor:start_child(session_sup, Opts), session_sup is specified as: {session_sup, {supervisor, start_link, [{local, session_sup},?MODULE,[session]]}, permanent, infinity, supervisor, [] } And the init that returns the spec for the child is this: init([Module]) -> {ok, {_SupFlags = {simple_one_for_one, ?MAX_RESTART, ?MAX_TIME}, [{undefined, {Module, start_link, []}, temporary, 2000, worker, [] }] } }. On Tue, 10 Jun 2008 15:45:51 +0200, "Ulf Wiger (TN/EAB)" said: > > Make sure your process is trapping exits. > If it isn't, the process will die immediately when > the supervisor calls exit(Pid, shutdown). > > BR, > Ulf W > > Gijsbert de Haan skrev: > > Hi, > > > > I have an application that I want to stop. In terminate() of one of the > > processes I save some state to an xml file (with xmerl:export() and > > file:write_file()). When this process terminates by itself all is well, > > and the file is saved. > > But when I call application:stop(myapp) the call to xmerl:export() (in > > terminate) never returns and the state is not saved. There are no error > > reports in the log, just the usual info report about myapp having > > exited. I have put a try/catch (throw/exit/error) around the > > xmerl:export() with some io:format() calls but nothing comes out. > > > > Anybody have a suggestion? > > Thanks, > > Gijsbert > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions From kostis@REDACTED Thu Jun 12 00:16:12 2008 From: kostis@REDACTED (Kostis Sagonas) Date: Thu, 12 Jun 2008 01:16:12 +0300 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: References: Message-ID: <48504EAC.8000500@cs.ntua.gr> Bjorn Gustavsson wrote: > Bug fix release : otp_src_R12B-3 > Build date : 2008-06-11 > > This is bug fix release 3 for the R12B release. > > You can find the README file for the release at > > http://www.erlang.org/download/otp_src_R12B-3.readme (this file) I just noticed that the README file reads as follows as far as the hipe application in R12B-3 is concerned. --- hipe-3.6.7 --------------------------------------------------------- OTP-7388 Minor changes. ------------------------------------------------------------------------ It's of course our fault that the README file reads this way -- we are too busy with other things -- but I just want to point out that the native code produced by the HiPE compiler in R12B-3 reports precise exception information (including stacktraces) similar to the exceptions that the code by the BEAM compiler generates. This is an important change which has been on our TODO list for far too long. In my opinion, the main reason to be reluctant to use native code in applications does not exist any longer. Anyway, the point is that this is not a "minor change"... Kostis (for the HiPE group) From fritchie@REDACTED Thu Jun 12 01:32:19 2008 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Wed, 11 Jun 2008 18:32:19 -0500 Subject: [erlang-questions] Index Overhead In Mnesia In-Reply-To: Message of "Tue, 10 Jun 2008 19:03:52 +0200." <7082733C-07D7-4572-9ADB-02119FCB6C01@gmail.com> Message-ID: <63837.1213227139@snookles.snookles.com> Sean Hinde wrote: >> If so, is this a bug or a feature? sh> It is a long standing "feature" of mnesia secondary indexes. It is sh> one that I also ran into a few weeks into my use of Erlang/mnesia. Yes, "feature" is a good word for it. sh> I guess it is in the same category as repeated queue scanning for sh> selective receive - easy to workaround in most cases, so no major sh> incentive to change the behaviour. At least one workaround has been mentioned elsewhere in this thread, IIRC. Another one is to change the term stored in each secondary index'ed column from: term() ... where term() is too popular, e.g. 5 to something like: {term(), much_less_frequent_filler_term()} ... where the 2nd term is calculated by something like: {_, _, FillerNumber} = erlang:now() Because the entire term is unique (or has very few duplicates), the secondary index bag (or duplicate_bag, I forget) doesn't blow up with O(N^2) behavior. This causes problems such as: * needing to use index_match_object() (e.g. with a pattern of {5, '_'}) instead of index_read() to fetch records matching 5. * needing to use {DesiredTerm, '_'} in other places that you perform queries, such as using QLC. * perhaps others that I've forgotten? -Scott From saleyn@REDACTED Thu Jun 12 02:50:06 2008 From: saleyn@REDACTED (Serge Aleynikov) Date: Wed, 11 Jun 2008 20:50:06 -0400 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: References: Message-ID: <485072BE.602@gmail.com> Thanks for including the OTP-6940 patch! There's a typo in the docs of the code_server - the new environment variable is called "ERL_LIBS" rather than "ERL_LIB". It would be worth mentioning in documentation of erlc that the ERL_LIBS setting has the same effect. This is very convenient when erlc is used in Makefiles. Serge Bjorn Gustavsson wrote: > Bug fix release : otp_src_R12B-3 > Build date : 2008-06-11 > > This is bug fix release 3 for the R12B release. > > You can find the README file for the release at > > http://www.erlang.org/download/otp_src_R12B-3.readme (this file) > > The source distribution and binary distribution for Windows can be > downloaded from > > http://www.erlang.org/download/otp_src_R12B-3.tar.gz > http://www.erlang.org/download/otp_win32_R12B-3.exe > > The distribution can also be downloaded using the BitTorrent > protocol. Use the following torrent files to download the source > distribution and binary distribution for Windows: > > http://www.erlang.org/download/otp_src_R12B-3.tar.gz.torrent > http://www.erlang.org/download/otp_win32_R12B-3.exe.torrent > > Note: To unpack the TAR archive you need a GNU TAR compatible program. > > For installation instructions please read the README file that is part > of the distribution. > > The on-line documentation can be found at: http://www.erlang.org/doc/ > You can also download the complete HTML documentation or the Unix manual files > > http://www.erlang.org/download/otp_doc_html_R12B-3.tar.gz > http://www.erlang.org/download/otp_doc_man_R12B-3.tar.gz > > We also want to thank those that sent us patches, suggestions and bug > reports, > > The OTP Team > From devdoer2@REDACTED Thu Jun 12 05:13:30 2008 From: devdoer2@REDACTED (devdoer bird) Date: Thu, 12 Jun 2008 11:13:30 +0800 Subject: [erlang-questions] is there any logging module for erlang? Message-ID: Hi: I wonder whether any logging module for erlang exists,like python's logging module which support :critical level setting ,filtering ,diffrent handlers (stream handler,socket handler,ram handler,file handler...). Thanks a lot! -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.polvorin@REDACTED Thu Jun 12 05:52:45 2008 From: pablo.polvorin@REDACTED (Pablo Polvorin) Date: Thu, 12 Jun 2008 05:52:45 +0200 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: References: Message-ID: <1ffe809c0806112052x6e183cf6y5389693a96764448@mail.gmail.com> Hi, did you tried to use the the error_logger module, in the standard OTP distribution? http://www.erlang.org/doc/man/error_logger.html also the the report browser is quite useful http://www.erlang.org/doc/man/rb.html cheers. pablo 2008/6/12 devdoer bird : > Hi: > > I wonder whether any logging module for erlang exists,like python's logging > module which support :critical level setting ,filtering ,diffrent handlers > (stream handler,socket handler,ram handler,file handler...). > > Thanks a lot! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- -- pablo http://ppolv.wordpress.com ---- -------------- next part -------------- An HTML attachment was scrubbed... URL: From devdoer2@REDACTED Thu Jun 12 06:15:55 2008 From: devdoer2@REDACTED (devdoer bird) Date: Thu, 12 Jun 2008 12:15:55 +0800 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: <1ffe809c0806112052x6e183cf6y5389693a96764448@mail.gmail.com> References: <1ffe809c0806112052x6e183cf6y5389693a96764448@mail.gmail.com> Message-ID: Thanks. I've seen the error logger,but I need code something to make It support ciritcal levelt setting(info/debug/warming/error) and filterting(which can define conditions control whether one specific log record should be printed ) What I need is more like log4cpp or python's logging module. And I've stolen some logging code from couchdb and ejabbered ,but they only support critical level setting . Is any other logging package you guys know? 2008/6/12, Pablo Polvorin : > > Hi, > did you tried to use the the error_logger module, in the standard OTP > distribution? > http://www.erlang.org/doc/man/error_logger.html > > also the the report browser is quite useful > http://www.erlang.org/doc/man/rb.html > > cheers. > pablo > > 2008/6/12 devdoer bird : > >> Hi: >> >> I wonder whether any logging module for erlang exists,like python's >> logging module which support :critical level setting ,filtering ,diffrent >> handlers (stream handler,socket handler,ram handler,file handler...). >> >> Thanks a lot! >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > -- > pablo > http://ppolv.wordpress.com > ---- > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf.wiger@REDACTED Thu Jun 12 08:03:47 2008 From: ulf.wiger@REDACTED (Ulf Wiger (TN/EAB)) Date: Thu, 12 Jun 2008 08:03:47 +0200 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: References: <1ffe809c0806112052x6e183cf6y5389693a96764448@mail.gmail.com> Message-ID: <4850BC43.3090105@ericsson.com> devdoer bird skrev: > Thanks. > I've seen the error logger,but I need code something to make It support > ciritcal levelt setting(info/debug/warming/error) and filterting(which > can define conditions control whether one specific log record should be > printed ) > > What I need is more like log4cpp or python's logging module. > > And I've stolen some logging code from couchdb and ejabbered ,but they > only support critical level setting . > > Is any other logging package you guys know? You can take a look at how mnesia does it. Basically, it's this code in mnesia_lib.erl. BR, Ulf W report_system_event({'EXIT', Reason}, Event) -> Mod = mnesia_monitor:get_env(event_module), case mnesia_sup:start_event() of {ok, Pid} -> link(Pid), gen_event:call(mnesia_event, Mod, Event, infinity), unlink(Pid), %% We get an exit signal if server dies receive {'EXIT', Pid, _Reason} -> {error, {node_not_running, node()}} after 0 -> gen_event:stop(mnesia_event), ok end; Error -> Msg = "Mnesia(~p): Cannot report event ~p: ~p (~p)~n", error_logger:format(Msg, [node(), Event, Reason, Error]) end; report_system_event(_Res, _Event) -> ignore. %% important messages are reported regardless of debug level important(Format, Args) -> save({Format, Args}), report_system_event({mnesia_info, Format, Args}). %% Warning messages are reported regardless of debug level warning(Format, Args) -> save({Format, Args}), report_system_event({mnesia_warning, Format, Args}). %% error messages are reported regardless of debug level error(Format, Args) -> save({Format, Args}), report_system_event({mnesia_error, Format, Args}). %% verbose messages are reported if debug level == debug or verbose verbose(Format, Args) -> case mnesia_monitor:get_env(debug) of none -> save({Format, Args}); verbose -> important(Format, Args); debug -> important(Format, Args); trace -> important(Format, Args) end. %% debug message are display if debug level == 2 dbg_out(Format, Args) -> case mnesia_monitor:get_env(debug) of none -> ignore; verbose -> save({Format, Args}); _ -> report_system_event({mnesia_info, Format, Args}) end. From bengt.kleberg@REDACTED Thu Jun 12 09:19:45 2008 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Thu, 12 Jun 2008 09:19:45 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: <484FFB46.40500@gmail.com> References: <194477.83170.qm@web38804.mail.mud.yahoo.com> <484FC689.7090807@gmail.com> <484FFB46.40500@gmail.com> Message-ID: <1213255185.32143.5.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> Greetings, The numbers I have seen for Erlang (on AXD301) is nine nines. Would not "factor of 10" mean that the perl phone switch ''only'' have to reach eight nines? bengt On Wed, 2008-06-11 at 18:20 +0200, Mats Cronqvist wrote: > Kevin Scaldeferri wrote: > > > > On Jun 11, 2008, at 5:35 AM, Mats Cronqvist wrote: > > > >> Thomas Lindgren wrote: > >>> > >>> ...I thought the "wide finder" work of > >>> last fall was pretty interesting. (Likewise for the > >>> WF2 getting started now.) > >>> > >>> > >> i tuned out of that discussion, but i was under the impression that > >> erlang pretty much... sucked. > >> so it was interesting to see Jeff Atwood's summary(*); > > > > I think it's more accurate to say that early, "idiomatic" Erlang > > implementations sucked. Once you give up on using line-oriented I/O > > and write a few hundred lines of code using raw I/O, it gets fast. > > For the record, though, the winning Perl implementation was a couple > > dozen lines and, while clever, not obfuscated. > > i was surprised that the erlang implementation was able to get within > a factor of 10 for what's pretty much the canonical perl problem. > kind of like implementing a phone switch with 10,000 concurrent tasks > and get nine nines reliability in perl. > > mats > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From tobbe@REDACTED Thu Jun 12 09:59:06 2008 From: tobbe@REDACTED (Torbjorn Tornkvist) Date: Thu, 12 Jun 2008 09:59:06 +0200 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: References: Message-ID: Great work! A little warning: there is a risk for confusion here because the new regexp module has been named 're' which also is the name of the regexp module that Robert Virding has put on User Contrib at trapexit.org. Cheers, Tobbe Bjorn Gustavsson wrote: > Bug fix release : otp_src_R12B-3 > Build date : 2008-06-11 > > This is bug fix release 3 for the R12B release. > > You can find the README file for the release at > > http://www.erlang.org/download/otp_src_R12B-3.readme (this file) > > The source distribution and binary distribution for Windows can be > downloaded from > > http://www.erlang.org/download/otp_src_R12B-3.tar.gz > http://www.erlang.org/download/otp_win32_R12B-3.exe > > The distribution can also be downloaded using the BitTorrent > protocol. Use the following torrent files to download the source > distribution and binary distribution for Windows: > > http://www.erlang.org/download/otp_src_R12B-3.tar.gz.torrent > http://www.erlang.org/download/otp_win32_R12B-3.exe.torrent > > Note: To unpack the TAR archive you need a GNU TAR compatible program. > > For installation instructions please read the README file that is part > of the distribution. > > The on-line documentation can be found at: http://www.erlang.org/doc/ > You can also download the complete HTML documentation or the Unix manual files > > http://www.erlang.org/download/otp_doc_html_R12B-3.tar.gz > http://www.erlang.org/download/otp_doc_man_R12B-3.tar.gz > > We also want to thank those that sent us patches, suggestions and bug > reports, > > The OTP Team > From sverker@REDACTED Thu Jun 12 10:29:13 2008 From: sverker@REDACTED (Sverker Eriksson) Date: Thu, 12 Jun 2008 10:29:13 +0200 Subject: [erlang-questions] ets:file2tab/1 question In-Reply-To: References: <4850068B.5050001@gmail.com> Message-ID: <4850DE59.5040403@erix.ericsson.se> Per Melin wrote: > 2008/6/11 DougEdmunds : > >> In the docs for the ets module: >> >> file2tab(Filename) -> {ok,Tab} | {error,Reason} >> >> Types: >> >> Filename = string() | atom() >> Tab = tid() | atom() >> Reason = term() >> >> Can someone provide an example where >> Tab returns an atom() and not a tid()? >> > > 1> ets:new(foo, [named_table]). > foo > 2> ets:tab2file(foo, "foo.ets"). > ok > 3> ets:delete(foo). > true > 4> ets:file2tab("foo.ets"). > {ok,foo Every ets table has a an opaque tid(). Named tables also have a name (atom()). In current implementation however, the opaque tid() of a named table happens to be the same as the name. But this is not guaranteed by the interface. So, the statement "Tab = tid() | atom()" would to imply that ets:file2tab returns the tid() for an unnamed table and the name for a named table. /Sverker, Erlang OTP Ericsson From ulf.wiger@REDACTED Thu Jun 12 10:56:01 2008 From: ulf.wiger@REDACTED (Ulf Wiger (TN/EAB)) Date: Thu, 12 Jun 2008 10:56:01 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: <1213255185.32143.5.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> References: <194477.83170.qm@web38804.mail.mud.yahoo.com> <484FC689.7090807@gmail.com> <484FFB46.40500@gmail.com> <1213255185.32143.5.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> Message-ID: <4850E4A1.6080706@ericsson.com> Bengt Kleberg skrev: > Greetings, > > The numbers I have seen for Erlang (on AXD301) is nine nines. Would not > "factor of 10" mean that the perl phone switch ''only'' have to reach > eight nines? I wish people could use "better than 5 nines" rather than "9 nines". Considering that planned downtime is included, it's still very, very good, and we have consistent measurements from the field over several years indicating that it's also true (I won't say how much better, since it's not an officially disclosed figure). The system is designed to reach 5 nines availability or better. The reason why the "9 nines" figure came to be bandied about was that Joe was looking for an officially approved number to use in his LL2 talk at MIT. The only one we found (that was explicitly "approved for external communication") was a quote from British Telecom. So while "9 nines" is true in a sense, it was for a particular network, after a certain amount of time (in fact, the initial trial). As such, it was a formidable success, far beyond the hopes of anyone involved. I'm not quite sure how one would design for 9 nines, but then, one would also have to look closely at how downtime is actually measured. In telecoms, one usually doesn't count disturbances shorter than 30 seconds, which means that you can potentially get away with quite a lot of short outages, as long as you recover quickly enough. Obviously, with a "9 nines" target (3.1 ms/year), one 30 second outage uses up the entire budget for the next ten thousand years, so in this case, it's a useless target. It could only be interesting in cases where failure is a fatal incident, which must never happen. Also, through the life of a telecoms product, high availability is not the only metric that counts. Sometimes, those pesky customers want something entirely new, like migrating from ATM to IP. This overrides requirements of availability, since even the peskiest customers realize that the cost of ensuring no downtime when replacing the entire network would be impossibly high (impossible?) BTW, the 9 nines case of the AXD 301 referred to on single case of local overload - due to an extreme peak of callers to the "Pop Idol" show, causing one control processor to reboot unnecessarily. The standby took over, and for 15 seconds, no calls were let through on the affected lines - the rest of the system was unaffected. According to today's way of measuring downtime, this actually wouldn't have counted as downtime, and the official availability would have been 100%... Finally, let's remember that we cannot credit Erlang for all the good availability of the AXD 301. For one thing, we could replace the switch core without service interruption (initially, it was only theoretically possible, but didn't work in practice, since the rewiring required the small hands of a 9 year-old, and various legal issues prevented us from deploying such in the field for upgrades). BR, Ulf W From btolputt@REDACTED Thu Jun 12 01:09:41 2008 From: btolputt@REDACTED (Benjamin Tolputt) Date: Thu, 12 Jun 2008 09:09:41 +1000 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: <48504EAC.8000500@cs.ntua.gr> References: <48504EAC.8000500@cs.ntua.gr> Message-ID: <48505B35.6070006@bigpond.net.au> Whilst this is a fantastic (and I am in no way belittling it - I am impressed & grateful for said changes!), I must state that there is a very good reason to be reluctant about native code for Windows developers. Unless something has changed while I wasn't paying attention, Windows is still not supported for HiPE compilation/deployment. This is the biggest hurdle I have to cross whenever suggesting Erlang for custom development I undertake... especially now that I can show the performance improvements possible with SMP on our 4-core servers. --Ben Kostis Sagonas wrote: > I just noticed that the README file reads as follows as far as the hipe > application in R12B-3 is concerned. > > --- hipe-3.6.7 --------------------------------------------------------- > > OTP-7388 Minor changes. > > ------------------------------------------------------------------------ > > It's of course our fault that the README file reads this way -- we are > too busy with other things -- but I just want to point out that the > native code produced by the HiPE compiler in R12B-3 reports precise > exception information (including stacktraces) similar to the exceptions > that the code by the BEAM compiler generates. > > This is an important change which has been on our TODO list for far too > long. In my opinion, the main reason to be reluctant to use native code > in applications does not exist any longer. Anyway, the point is that > this is not a "minor change".. From 0x6e6562@REDACTED Thu Jun 12 13:03:20 2008 From: 0x6e6562@REDACTED (Ben Hood) Date: Thu, 12 Jun 2008 12:03:20 +0100 Subject: [erlang-questions] Index Overhead In Mnesia In-Reply-To: <269388e30806120402ice6d053s2eed292046bf67a4@mail.gmail.com> References: <7082733C-07D7-4572-9ADB-02119FCB6C01@gmail.com> <63837.1213227139@snookles.snookles.com> <269388e30806120402ice6d053s2eed292046bf67a4@mail.gmail.com> Message-ID: <269388e30806120403j4ef6c324o65344cc027c41815@mail.gmail.com> Scott, On Thu, Jun 12, 2008 at 12:32 AM, Scott Lystig Fritchie wrote: > > term() ... where term() is too popular, e.g. 5 > > to something like: > > {term(), much_less_frequent_filler_term()} > > ... where the 2nd term is calculated by something like: > > {_, _, FillerNumber} = erlang:now() > > Because the entire term is unique (or has very few duplicates), the > secondary index bag (or duplicate_bag, I forget) doesn't blow up with > O(N^2) behavior. Doesn't this render the index useless though? Is this not just creating more index entries so that reading the index is the same as doing a full table scan? Ben From chsu79@REDACTED Thu Jun 12 13:18:47 2008 From: chsu79@REDACTED (Christian S) Date: Thu, 12 Jun 2008 13:18:47 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: <4850E4A1.6080706@ericsson.com> References: <194477.83170.qm@web38804.mail.mud.yahoo.com> <484FC689.7090807@gmail.com> <484FFB46.40500@gmail.com> <1213255185.32143.5.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> <4850E4A1.6080706@ericsson.com> Message-ID: On Thu, Jun 12, 2008 at 10:56 AM, Ulf Wiger (TN/EAB) wrote: > recover quickly enough. Obviously, with a "9 nines" target > (3.1 ms/year), one 30 second outage uses up the entire > budget for the next ten thousand years, so in this case, it's When I first got interested in Erlang, it was from reading texts that focused more on the mean-time-to-recover-from-failure. It is an interesting number that has more impact on the system design. I'm not sure if this is Joe Armstrong's contribution, or if he is channeling what the telecom industry as a whole already knew. Erlang/OTP certainly focuses on recovering from errors, looking at them as inevitable. If you want a very low mean-time-to-recover, then humans cant be involved. The system needs to fail over automatically, it needs to monitor itself, it needs to restart itself. If you need a human to recover from an error, your system is not likely a system that makes the system owner happy. I can see many applications where it is MUCH better if the mean-time-between failure is 24hs with a mean-time-to-recover around a minute, in contrast to a system where the mean-time-between failure is 1 year and the mean-time-to-recover is around 365 minutes. From tobbe@REDACTED Thu Jun 12 13:43:10 2008 From: tobbe@REDACTED (Torbjorn Tornkvist) Date: Thu, 12 Jun 2008 13:43:10 +0200 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: <48505B35.6070006@bigpond.net.au> References: <48504EAC.8000500@cs.ntua.gr> <48505B35.6070006@bigpond.net.au> Message-ID: Benjamin Tolputt wrote: > Whilst this is a fantastic (and I am in no way belittling it - I am > impressed & grateful for said changes!), I must state that there is a > very good reason to be reluctant about native code for Windows > developers. Unless something has changed while I wasn't paying > attention, Windows is still not supported for HiPE compilation/deployment. > > This is the biggest hurdle I have to cross whenever suggesting Erlang > for custom development I undertake... especially now that I can show the > performance improvements possible with SMP on our 4-core servers. This is an interesting question, i.e how much do you really gain from using hipe (native) compiled code? I suspect that in (Erlang written) production systems, I/O is where most of the time is spent. So using Hipe compiled code will not have any real impact. If would love to be proven wrong here, so if someone has got any figures to present... --Tobbe > > --Ben > > Kostis Sagonas wrote: >> I just noticed that the README file reads as follows as far as the hipe >> application in R12B-3 is concerned. >> >> --- hipe-3.6.7 --------------------------------------------------------- >> >> OTP-7388 Minor changes. >> >> ------------------------------------------------------------------------ >> >> It's of course our fault that the README file reads this way -- we are >> too busy with other things -- but I just want to point out that the >> native code produced by the HiPE compiler in R12B-3 reports precise >> exception information (including stacktraces) similar to the exceptions >> that the code by the BEAM compiler generates. >> >> This is an important change which has been on our TODO list for far too >> long. In my opinion, the main reason to be reluctant to use native code >> in applications does not exist any longer. Anyway, the point is that >> this is not a "minor change".. From greg.burri@REDACTED Thu Jun 12 13:56:12 2008 From: greg.burri@REDACTED (Greg Burri) Date: Thu, 12 Jun 2008 13:56:12 +0200 Subject: [erlang-questions] Socket keepalive option Message-ID: <60ed8a460806120456r295e07d6gc5ee6ecfea75ebd2@mail.gmail.com> Hi, Is it possible to set the option {keepalive, Boolean} to a socket by the server side after the socket has been created ? I try to do that with an application under Yaws (http://yaws.hyber.org/) : out(A) -> inet:setopts(A#arg.clisock, [{active, true}, {keepalive, true}]), [..] When the server waits and it doesn't send data I capture the TCP stream with Wireshark (http://www.wireshark.org/) and see if there is some 'Ack' from the server... but I don't see anything.. Is anyone have an idea about that ? /Greg From ulf.wiger@REDACTED Thu Jun 12 13:59:15 2008 From: ulf.wiger@REDACTED (Ulf Wiger (TN/EAB)) Date: Thu, 12 Jun 2008 13:59:15 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: References: <194477.83170.qm@web38804.mail.mud.yahoo.com> <484FC689.7090807@gmail.com> <484FFB46.40500@gmail.com> <1213255185.32143.5.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> <4850E4A1.6080706@ericsson.com> Message-ID: <48510F93.1010109@ericsson.com> Christian S skrev: > > If you want a very low mean-time-to-recover, then humans cant be > involved. The system needs to fail over automatically, it needs to > monitor itself, it needs to restart itself. If you need a human to > recover from an error, your system is not likely a system that makes > the system owner happy. Indeed. There are also different types of system: - central-office applications, which are located in a place which is (perhaps) manned 24/7. - remote systems, which are installed in places that are not manned (one operator in Iceland used to hang AXD 301 switches on the inside of the door to power converter stations; unfortunately, they weren't even heated...) - systems that are impossible to get to (the Mars space probe comes to mind) This obviously affects Mean Time To Repair, and thus, strategies for recovery in the system - even from serious errors. Since the AXD 301 is a remote system, we designed it to even recover from partitioned networks and database inconsistency automatically (at the price of potential loss of persistent data), and we could also handle file system inconsistency (even though that would have to be triggered through the network management interface). One really beautiful aspect of Erlang, though is the way software can easily be made to "heal itself" from spurious errors. This happens so quickly and unobtrusively that it can be easy to miss in an unmanned system. We ended up adding logic to catch such spurious restarts, and report if some named process kept restarting seldom enough not to trigger the restart escalation in the supervisor. Otherwise, we might miss the pattern even when studying the wrap logs. BR, Ulf W From adam@REDACTED Thu Jun 12 14:26:41 2008 From: adam@REDACTED (Adam Lindberg) Date: Thu, 12 Jun 2008 14:26:41 +0200 Subject: [erlang-questions] erlrc, a package manager Erlang integration kit Message-ID: <6344005f0806120526sd4c56dej41882a9fc88332e9@mail.gmail.com> Hi, Just spotted erlrc over at Google Code: http://code.google.com/p/erlrc/wiki/ErlrcHowto It aims to integrate the package manager of the OS with Erlang hot code loading, so that when a package gets upgraded the Erlang modules it contains are automatically loaded on any configured node. Could be a nice alternative to manual upgrades or script hacks. Especially if your live system is running on a Linux distribution with a package manager. :-) Use case: Install Ubuntu Server, install ejabberd, get automatic hot loading updates without downtime, profit! Cheers! Adam -- Adam Lindberg http://www.erlang-consulting.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Thu Jun 12 15:09:42 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 12 Jun 2008 15:09:42 +0200 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: <485072BE.602@gmail.com> References: <485072BE.602@gmail.com> Message-ID: Serge Aleynikov writes: > Thanks for including the OTP-6940 patch! There's a typo in the docs > of the code_server - the new environment variable is called "ERL_LIBS" > rather than "ERL_LIB". Thanks! Corrected in the on-line documentation and in the development branch for R12B-4. /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From devdoer2@REDACTED Thu Jun 12 15:36:48 2008 From: devdoer2@REDACTED (devdoer bird) Date: Thu, 12 Jun 2008 21:36:48 +0800 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: <4850BC43.3090105@ericsson.com> References: <1ffe809c0806112052x6e183cf6y5389693a96764448@mail.gmail.com> <4850BC43.3090105@ericsson.com> Message-ID: thanks but it also doesn't support log filtering. Here is the explanation of filtering from python manual. " Filters can be used by Handlers and Loggers for more sophisticated filtering than is provided by levels. The base filter class only allows events which are below a certain point in the logger hierarchy. For example, a filter initialized with "A.B" will allow events logged by loggers "A.B", "A.B.C", "A.B.C.D", "A.B.D" etc. but not "A.BB", "B.A.B" etc. If initialized with the empty string, all events are passed. *class Filter*( [name]) Returns an instance of the Filter class. If nameis specified, it names a logger which, together with its children, will have its events allowed through the filter. If no name is specified, allows every event. *filter*( record) Is the specified record to be logged? Returns zero for no, nonzero for yes. If deemed appropriate, the record may be modified in-place by this method. " I think I need write one by myself. Thank you . 2008/6/12, Ulf Wiger (TN/EAB) : > > devdoer bird skrev: > >> Thanks. >> I've seen the error logger,but I need code something to make It support >> ciritcal levelt setting(info/debug/warming/error) and filterting(which can >> define conditions control whether one specific log record should be printed >> ) >> What I need is more like log4cpp or python's logging module. >> And I've stolen some logging code from couchdb and ejabbered ,but they >> only support critical level setting . >> Is any other logging package you guys know? >> > > You can take a look at how mnesia does it. > Basically, it's this code in mnesia_lib.erl. > > BR, > Ulf W > > > report_system_event({'EXIT', Reason}, Event) -> > Mod = mnesia_monitor:get_env(event_module), > case mnesia_sup:start_event() of > {ok, Pid} -> > link(Pid), > gen_event:call(mnesia_event, Mod, Event, infinity), > unlink(Pid), > > %% We get an exit signal if server dies > receive > {'EXIT', Pid, _Reason} -> > {error, {node_not_running, node()}} > after 0 -> > gen_event:stop(mnesia_event), > ok > end; > > Error -> > Msg = "Mnesia(~p): Cannot report event ~p: ~p (~p)~n", > error_logger:format(Msg, [node(), Event, Reason, Error]) > end; > report_system_event(_Res, _Event) -> > ignore. > > %% important messages are reported regardless of debug level > important(Format, Args) -> > save({Format, Args}), > report_system_event({mnesia_info, Format, Args}). > > %% Warning messages are reported regardless of debug level > warning(Format, Args) -> > save({Format, Args}), > report_system_event({mnesia_warning, Format, Args}). > > %% error messages are reported regardless of debug level > error(Format, Args) -> > save({Format, Args}), > report_system_event({mnesia_error, Format, Args}). > > %% verbose messages are reported if debug level == debug or verbose > verbose(Format, Args) -> > case mnesia_monitor:get_env(debug) of > none -> save({Format, Args}); > verbose -> important(Format, Args); > debug -> important(Format, Args); > trace -> important(Format, Args) > end. > > %% debug message are display if debug level == 2 > dbg_out(Format, Args) -> > case mnesia_monitor:get_env(debug) of > none -> ignore; > verbose -> save({Format, Args}); > _ -> report_system_event({mnesia_info, Format, Args}) > end. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dizzyd@REDACTED Thu Jun 12 16:39:07 2008 From: dizzyd@REDACTED (Dave Smith) Date: Thu, 12 Jun 2008 08:39:07 -0600 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: References: <1ffe809c0806112052x6e183cf6y5389693a96764448@mail.gmail.com> <4850BC43.3090105@ericsson.com> Message-ID: I think it might be worth noting that Erlang provides some powerful runtime tracing tools. Where in python/java/c++ I would litter my code with logging statements to keep track of what's happening where, I find myself not needing that as much, since I can simply throw a trace on the things I'm interested in, when/if there are problem. That's why (I think) erlang is so light on logging subsystems. Just a thought.. :) D. 2008/6/12 devdoer bird : > thanks but it also doesn't support log filtering. > Here is the explanation of filtering from python manual. > " > > Filters can be used by Handlers and Loggers for more sophisticated filtering > than is provided by levels. The base filter class only allows events which > are below a certain point in the logger hierarchy. For example, a filter > initialized with "A.B" will allow events logged by loggers "A.B", "A.B.C", > "A.B.C.D", "A.B.D" etc. but not "A.BB", "B.A.B" etc. If initialized with the > empty string, all events are passed. > > class Filter( [name]) > Returns an instance of the Filter class. If name is specified, it names a > logger which, together with its children, will have its events allowed > through the filter. If no name is specified, allows every event. > > filter( record) > Is the specified record to be logged? Returns zero for no, nonzero for yes. > If deemed appropriate, the record may be modified in-place by this method. > > " > I think I need write one by myself. > Thank you . > > > 2008/6/12, Ulf Wiger (TN/EAB) : >> >> devdoer bird skrev: >>> >>> Thanks. >>> I've seen the error logger,but I need code something to make It support >>> ciritcal levelt setting(info/debug/warming/error) and filterting(which can >>> define conditions control whether one specific log record should be printed >>> ) >>> What I need is more like log4cpp or python's logging module. >>> And I've stolen some logging code from couchdb and ejabbered ,but they >>> only support critical level setting . >>> Is any other logging package you guys know? >> >> You can take a look at how mnesia does it. >> Basically, it's this code in mnesia_lib.erl. >> >> BR, >> Ulf W >> >> >> report_system_event({'EXIT', Reason}, Event) -> >> Mod = mnesia_monitor:get_env(event_module), >> case mnesia_sup:start_event() of >> {ok, Pid} -> >> link(Pid), >> gen_event:call(mnesia_event, Mod, Event, infinity), >> unlink(Pid), >> >> %% We get an exit signal if server dies >> receive >> {'EXIT', Pid, _Reason} -> >> {error, {node_not_running, node()}} >> after 0 -> >> gen_event:stop(mnesia_event), >> ok >> end; >> >> Error -> >> Msg = "Mnesia(~p): Cannot report event ~p: ~p (~p)~n", >> error_logger:format(Msg, [node(), Event, Reason, Error]) >> end; >> report_system_event(_Res, _Event) -> >> ignore. >> >> %% important messages are reported regardless of debug level >> important(Format, Args) -> >> save({Format, Args}), >> report_system_event({mnesia_info, Format, Args}). >> >> %% Warning messages are reported regardless of debug level >> warning(Format, Args) -> >> save({Format, Args}), >> report_system_event({mnesia_warning, Format, Args}). >> >> %% error messages are reported regardless of debug level >> error(Format, Args) -> >> save({Format, Args}), >> report_system_event({mnesia_error, Format, Args}). >> >> %% verbose messages are reported if debug level == debug or verbose >> verbose(Format, Args) -> >> case mnesia_monitor:get_env(debug) of >> none -> save({Format, Args}); >> verbose -> important(Format, Args); >> debug -> important(Format, Args); >> trace -> important(Format, Args) >> end. >> >> %% debug message are display if debug level == 2 >> dbg_out(Format, Args) -> >> case mnesia_monitor:get_env(debug) of >> none -> ignore; >> verbose -> save({Format, Args}); >> _ -> report_system_event({mnesia_info, Format, Args}) >> end. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From vances@REDACTED Thu Jun 12 16:46:11 2008 From: vances@REDACTED (Vance Shipley) Date: Thu, 12 Jun 2008 10:46:11 -0400 Subject: [erlang-questions] clueless performance question In-Reply-To: <48510F93.1010109@ericsson.com> References: <194477.83170.qm@web38804.mail.mud.yahoo.com> <484FC689.7090807@gmail.com> <484FFB46.40500@gmail.com> <1213255185.32143.5.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> <4850E4A1.6080706@ericsson.com> <48510F93.1010109@ericsson.com> Message-ID: <20080612144608.GA329@h216-235-12-173.host.egate.net> On Thu, Jun 12, 2008 at 01:59:15PM +0200, Ulf Wiger (TN/EAB) wrote: } One really beautiful aspect of Erlang, though is the way software } can easily be made to "heal itself" from spurious errors. This } happens so quickly and unobtrusively that it can be easy to miss Indeed. My very first production erlang code went into service as an embedded system. It was a half call model system where there was a call state finite state machine for each logical channel. When I got aroud to looking at the logs a few days later I found that every call through the system had result in a process crash. The crash was happening during call teardown so it wasn't noticed by anyone. The supervisor simply started a new gen_fsm in the idle state and the next call was handled as normal. I was quite impressed at the time because although I understood the theory seeing it work in a real sense was something else. -Vance From devdoer2@REDACTED Thu Jun 12 17:02:47 2008 From: devdoer2@REDACTED (devdoer bird) Date: Thu, 12 Jun 2008 23:02:47 +0800 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: References: <1ffe809c0806112052x6e183cf6y5389693a96764448@mail.gmail.com> <4850BC43.3090105@ericsson.com> Message-ID: you mean "erlang:trace(PidSpec, How, FlagList)" ,"erlang:trace_pattern(MFA, MatchSpec, FlagList)" ? 2008/6/12, Dave Smith : > > I think it might be worth noting that Erlang provides some powerful > runtime tracing tools. Where in python/java/c++ I would litter my code > with logging statements to keep track of what's happening where, I > find myself not needing that as much, since I can simply throw a trace > on the things I'm interested in, when/if there are problem. That's why > (I think) erlang is so light on logging subsystems. > > Just a thought.. :) > > D. > > 2008/6/12 devdoer bird : > > thanks but it also doesn't support log filtering. > > Here is the explanation of filtering from python manual. > > " > > > > Filters can be used by Handlers and Loggers for more sophisticated > filtering > > than is provided by levels. The base filter class only allows events > which > > are below a certain point in the logger hierarchy. For example, a filter > > initialized with "A.B" will allow events logged by loggers "A.B", > "A.B.C", > > "A.B.C.D", "A.B.D" etc. but not "A.BB", "B.A.B" etc. If initialized with > the > > empty string, all events are passed. > > > > class Filter( [name]) > > Returns an instance of the Filter class. If name is specified, it names a > > logger which, together with its children, will have its events allowed > > through the filter. If no name is specified, allows every event. > > > > filter( record) > > Is the specified record to be logged? Returns zero for no, nonzero for > yes. > > If deemed appropriate, the record may be modified in-place by this > method. > > > > " > > I think I need write one by myself. > > Thank you . > > > > > > 2008/6/12, Ulf Wiger (TN/EAB) : > >> > >> devdoer bird skrev: > >>> > >>> Thanks. > >>> I've seen the error logger,but I need code something to make It > support > >>> ciritcal levelt setting(info/debug/warming/error) and filterting(which > can > >>> define conditions control whether one specific log record should be > printed > >>> ) > >>> What I need is more like log4cpp or python's logging module. > >>> And I've stolen some logging code from couchdb and ejabbered ,but they > >>> only support critical level setting . > >>> Is any other logging package you guys know? > >> > >> You can take a look at how mnesia does it. > >> Basically, it's this code in mnesia_lib.erl. > >> > >> BR, > >> Ulf W > >> > >> > >> report_system_event({'EXIT', Reason}, Event) -> > >> Mod = mnesia_monitor:get_env(event_module), > >> case mnesia_sup:start_event() of > >> {ok, Pid} -> > >> link(Pid), > >> gen_event:call(mnesia_event, Mod, Event, infinity), > >> unlink(Pid), > >> > >> %% We get an exit signal if server dies > >> receive > >> {'EXIT', Pid, _Reason} -> > >> {error, {node_not_running, node()}} > >> after 0 -> > >> gen_event:stop(mnesia_event), > >> ok > >> end; > >> > >> Error -> > >> Msg = "Mnesia(~p): Cannot report event ~p: ~p (~p)~n", > >> error_logger:format(Msg, [node(), Event, Reason, Error]) > >> end; > >> report_system_event(_Res, _Event) -> > >> ignore. > >> > >> %% important messages are reported regardless of debug level > >> important(Format, Args) -> > >> save({Format, Args}), > >> report_system_event({mnesia_info, Format, Args}). > >> > >> %% Warning messages are reported regardless of debug level > >> warning(Format, Args) -> > >> save({Format, Args}), > >> report_system_event({mnesia_warning, Format, Args}). > >> > >> %% error messages are reported regardless of debug level > >> error(Format, Args) -> > >> save({Format, Args}), > >> report_system_event({mnesia_error, Format, Args}). > >> > >> %% verbose messages are reported if debug level == debug or verbose > >> verbose(Format, Args) -> > >> case mnesia_monitor:get_env(debug) of > >> none -> save({Format, Args}); > >> verbose -> important(Format, Args); > >> debug -> important(Format, Args); > >> trace -> important(Format, Args) > >> end. > >> > >> %% debug message are display if debug level == 2 > >> dbg_out(Format, Args) -> > >> case mnesia_monitor:get_env(debug) of > >> none -> ignore; > >> verbose -> save({Format, Args}); > >> _ -> report_system_event({mnesia_info, Format, Args}) > >> end. > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dizzyd@REDACTED Thu Jun 12 17:11:42 2008 From: dizzyd@REDACTED (Dave Smith) Date: Thu, 12 Jun 2008 09:11:42 -0600 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: References: <1ffe809c0806112052x6e183cf6y5389693a96764448@mail.gmail.com> <4850BC43.3090105@ericsson.com> Message-ID: Actually, I was referring to the dbg module in runtime_tools. Also the sys module in stdlib can be helpful when poking around OTP processes. >From my perspective, Erlang has a very different take on logging due to the fact that you can easily hop on a live system via a shell and introspect without stopping. Other languages/platforms don't really have the idea of being able to hook into the live system with a shell and poke around. Obviously, this can be dangerous, but at the same time it's something you simply MUST be able to do if you seriously want your system to stay up. There's always a place for error and/or transaction logging, of course. But for the vast majority of "debug" logs, I find that tracing can do the trick and help me work out exactly what's happening with my system without requiring me to litter the code with io:format/2. Perhaps I'm in the minority, but that's been my experience with erlang. :) D. On Thu, Jun 12, 2008 at 9:02 AM, devdoer bird wrote: > you mean "erlang:trace(PidSpec, How, FlagList)" ,"erlang:trace_pattern(MFA, > MatchSpec, FlagList)" ? > > 2008/6/12, Dave Smith : >> >> I think it might be worth noting that Erlang provides some powerful >> runtime tracing tools. Where in python/java/c++ I would litter my code >> with logging statements to keep track of what's happening where, I >> find myself not needing that as much, since I can simply throw a trace >> on the things I'm interested in, when/if there are problem. That's why >> (I think) erlang is so light on logging subsystems. >> >> Just a thought.. :) >> >> D. >> >> 2008/6/12 devdoer bird : >> > thanks but it also doesn't support log filtering. >> > Here is the explanation of filtering from python manual. >> > " >> > >> > Filters can be used by Handlers and Loggers for more sophisticated >> > filtering >> > than is provided by levels. The base filter class only allows events >> > which >> > are below a certain point in the logger hierarchy. For example, a filter >> > initialized with "A.B" will allow events logged by loggers "A.B", >> > "A.B.C", >> > "A.B.C.D", "A.B.D" etc. but not "A.BB", "B.A.B" etc. If initialized with >> > the >> > empty string, all events are passed. >> > >> > class Filter( [name]) >> > Returns an instance of the Filter class. If name is specified, it names >> > a >> > logger which, together with its children, will have its events allowed >> > through the filter. If no name is specified, allows every event. >> > >> > filter( record) >> > Is the specified record to be logged? Returns zero for no, nonzero for >> > yes. >> > If deemed appropriate, the record may be modified in-place by this >> > method. >> > >> > " >> > I think I need write one by myself. >> > Thank you . >> > >> > >> > 2008/6/12, Ulf Wiger (TN/EAB) : >> >> >> >> devdoer bird skrev: >> >>> >> >>> Thanks. >> >>> I've seen the error logger,but I need code something to make It >> >>> support >> >>> ciritcal levelt setting(info/debug/warming/error) and >> >>> filterting(which can >> >>> define conditions control whether one specific log record should be >> >>> printed >> >>> ) >> >>> What I need is more like log4cpp or python's logging module. >> >>> And I've stolen some logging code from couchdb and ejabbered ,but >> >>> they >> >>> only support critical level setting . >> >>> Is any other logging package you guys know? >> >> >> >> You can take a look at how mnesia does it. >> >> Basically, it's this code in mnesia_lib.erl. >> >> >> >> BR, >> >> Ulf W >> >> >> >> >> >> report_system_event({'EXIT', Reason}, Event) -> >> >> Mod = mnesia_monitor:get_env(event_module), >> >> case mnesia_sup:start_event() of >> >> {ok, Pid} -> >> >> link(Pid), >> >> gen_event:call(mnesia_event, Mod, Event, infinity), >> >> unlink(Pid), >> >> >> >> %% We get an exit signal if server dies >> >> receive >> >> {'EXIT', Pid, _Reason} -> >> >> {error, {node_not_running, node()}} >> >> after 0 -> >> >> gen_event:stop(mnesia_event), >> >> ok >> >> end; >> >> >> >> Error -> >> >> Msg = "Mnesia(~p): Cannot report event ~p: ~p (~p)~n", >> >> error_logger:format(Msg, [node(), Event, Reason, Error]) >> >> end; >> >> report_system_event(_Res, _Event) -> >> >> ignore. >> >> >> >> %% important messages are reported regardless of debug level >> >> important(Format, Args) -> >> >> save({Format, Args}), >> >> report_system_event({mnesia_info, Format, Args}). >> >> >> >> %% Warning messages are reported regardless of debug level >> >> warning(Format, Args) -> >> >> save({Format, Args}), >> >> report_system_event({mnesia_warning, Format, Args}). >> >> >> >> %% error messages are reported regardless of debug level >> >> error(Format, Args) -> >> >> save({Format, Args}), >> >> report_system_event({mnesia_error, Format, Args}). >> >> >> >> %% verbose messages are reported if debug level == debug or verbose >> >> verbose(Format, Args) -> >> >> case mnesia_monitor:get_env(debug) of >> >> none -> save({Format, Args}); >> >> verbose -> important(Format, Args); >> >> debug -> important(Format, Args); >> >> trace -> important(Format, Args) >> >> end. >> >> >> >> %% debug message are display if debug level == 2 >> >> dbg_out(Format, Args) -> >> >> case mnesia_monitor:get_env(debug) of >> >> none -> ignore; >> >> verbose -> save({Format, Args}); >> >> _ -> report_system_event({mnesia_info, Format, Args}) >> >> end. >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://www.erlang.org/mailman/listinfo/erlang-questions >> > > > From attila.rajmund.nohl@REDACTED Thu Jun 12 17:31:17 2008 From: attila.rajmund.nohl@REDACTED (attila.rajmund.nohl@REDACTED) Date: Thu, 12 Jun 2008 17:31:17 +0200 (CEST) Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: References: <1ffe809c0806112052x6e183cf6y5389693a96764448@mail.gmail.com> <4850BC43.3090105@ericsson.com> Message-ID: On Thu, 12 Jun 2008, Dave Smith wrote: > I think it might be worth noting that Erlang provides some powerful > runtime tracing tools. Where in python/java/c++ I would litter my code > with logging statements to keep track of what's happening where, I > find myself not needing that as much, since I can simply throw a trace > on the things I'm interested in, when/if there are problem. That's why > (I think) erlang is so light on logging subsystems. Logging is useful to detect errors in the first place (see the other thread where the bug wasn't noticed until the logfiles were read). tracing is useful if it's known that there's an error, but it's not clear that where that error actually is. Bye,NAR -- "Beware of bugs in the above code; I have only proved it correct, not tried it." From sebastian.bello@REDACTED Thu Jun 12 18:37:38 2008 From: sebastian.bello@REDACTED (Sebastian Bello) Date: Thu, 12 Jun 2008 13:37:38 -0300 Subject: [erlang-questions] SNMP questions In-Reply-To: <20080611.220630.11876259.mbj@tail-f.com> References: <484E7AA1.8020605@inswitch.us> <4850265C.6040702@inswitch.us> <20080611.220630.11876259.mbj@tail-f.com> Message-ID: <485150D2.3040709@inswitch.us> Martin, 1- what do you mean with "use global to register the master agent"? 2- I know it is a MIB design question, may you share your point of view with me? Thanks, Sebastian- Martin Bjorklund escribi?: > Hi, > > Sebastian Bello wrote: > >>> 1- we are using /snmpa:send_trap/ to send traps from within the >>> master agent node. How should this be done from a different node? >>> Maybe using Erlang communication to send the traps' information to >>> the master agent node and then issuing a /snmpa:send_trap/ from >>> it? Or is there a way to specify a remote node in >>> /snmpa:send_trap/? >>> > > Use the normal distribution. Either do a rpc to the master node, or > use global to register the master agent. > > >>> 2- in a distributed application, how should the MIB represent data >>> from applications running on different nodes? >>> > > This is a MIB design question - the nodes in the cluster can be > explicitly modelled in the MIBs or not. > > > /martin > > __________ NOD32 3178 (20080611) Information __________ > > This message was checked by NOD32 antivirus system. > http://www.eset.com > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Thu Jun 12 21:59:41 2008 From: rvirding@REDACTED (Robert Virding) Date: Thu, 12 Jun 2008 21:59:41 +0200 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: References: Message-ID: <3dbc6d1c0806121259g2877fb21s96c26730ba09abf8@mail.gmail.com> 2008/6/12 Torbjorn Tornkvist : > > Great work! > > A little warning: there is a risk for confusion here > because the new regexp module has been named 're' which > also is the name of the regexp module that Robert Virding > has put on User Contrib at trapexit.org. > > Cheers, Tobbe Yes, I know. I will have to start a name war to get back my name. Infect R12 with a virus which crashes all using the new built in module re. Or change the name of my module. :-) Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From raould@REDACTED Fri Jun 13 01:03:59 2008 From: raould@REDACTED (Raoul Duke) Date: Thu, 12 Jun 2008 16:03:59 -0700 Subject: [erlang-questions] refactoring + lfe? Message-ID: <91a2ba3e0806121603m42cbec62i8d2ca3c371b25a8e@mail.gmail.com> ok, just to be crazy, any thoughts on the chances of LFE working with Wrangler? http://www.cs.kent.ac.uk/projects/forse/ :) From ok@REDACTED Fri Jun 13 01:43:52 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 13 Jun 2008 11:43:52 +1200 Subject: [erlang-questions] clueless performance question In-Reply-To: References: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> <490D297E-0151-489D-BF26-355E9AB486B2@cs.otago.ac.nz> Message-ID: On 11 Jun 2008, at 11:57 pm, Mike Shaver wrote: > It's probably[*] going to be less efficient than early binding on > first call, but there's no reason that it can't be equivalent to early > binding on every subsequent call, until a module replacement triggers > the flushing of a JIT cache or what have you. The startup costs can be high. Comparing C code and Java code on my Sun UltraSPARC II, my C programs have usually *finished* before the Java code completes startup. > > > Work on tracing JITs indicates that the overhead of late binding can > be made to disappear completely in the loops and functions that > comprise virtually all of a program's work -- even in languages that > don't have Erlang's functional goodness to assist the compilers, and > permit mutation of objects in more difficult-to-handle ways than > module reload. I keep hearing this. Please provide some references. Best of all, point me to something that I can try on my own machines, because all of the jam tomorrow that I've ever heard about how C++ was going to be great one day and Java was going to be as fast as C have *never* panned out into anything I could measure for myself. (For what it's worth, I am an admirer of Self and have a copy of Self on my Sun. Too bad the characters are too tiny for me to read and nobody ever responded to my questions about how to make them bigger.) Remember: I am not complaining about Erlang. I think late binding is a great thing. I think Objective C is all the better for being such a suitable host for F-Script. From devdoer2@REDACTED Fri Jun 13 04:54:14 2008 From: devdoer2@REDACTED (devdoer bird) Date: Fri, 13 Jun 2008 10:54:14 +0800 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: References: <1ffe809c0806112052x6e183cf6y5389693a96764448@mail.gmail.com> <4850BC43.3090105@ericsson.com> Message-ID: Thanks for your info. 2008/6/12, attila.rajmund.nohl@REDACTED < attila.rajmund.nohl@REDACTED>: > > On Thu, 12 Jun 2008, Dave Smith wrote: > > > I think it might be worth noting that Erlang provides some powerful > > runtime tracing tools. Where in python/java/c++ I would litter my code > > with logging statements to keep track of what's happening where, I > > find myself not needing that as much, since I can simply throw a trace > > on the things I'm interested in, when/if there are problem. That's why > > (I think) erlang is so light on logging subsystems. > > Logging is useful to detect errors in the first place (see the other > thread where the bug wasn't noticed until the logfiles were read). > tracing is useful if it's known that there's an error, but it's not > clear that where that error actually is. > > Bye,NAR > -- > "Beware of bugs in the above code; I have only proved it correct, not > tried it." > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuad@REDACTED Fri Jun 13 06:06:25 2008 From: fuad@REDACTED (Fuad Tabba) Date: Fri, 13 Jun 2008 16:06:25 +1200 Subject: [erlang-questions] (newbie) Using Functions as Guards in Erlang Message-ID: <909c265f0806122106v88a09b5l1a2554c9c7a553fb@mail.gmail.com> Hi, I'm new to Erlang, so I apologize in advance for asking what might be obvious to some. I've run into problem trying to use functions as guards in Erlang. I looked through the manual, and discovered that this isn't allowed by the language in case those functions have side-effects. The thing is, my functions do not have side effects. Is there a way to convey that message to the compiler and allow my functions to be used as guards? (Something along the lines of #pragma i_swear_this_function_has_no_side_effects). I know that code can be rewritten to avoid this problem, however, and probably due to my limited experience with Erlang, the particular piece of code I have in mind would look really ugly and really complicated without the ability to do this. Below is a snippet of my code that I'm referring to, if anyone has any suggestions on how to make it work in Erlang and look elegant I'd appreciate it. This code deals with fixing up a red black tree after a node has been deleted, and here's what I'm trying to do:- % Algorithm from http://sage.mc.yu.edu/kbeen/teaching/algorithms/resources/red-black-tree.html % Case B1 delfix({Ky, Vy, Cy, {Kx, Vx, bb, A, B}, {Kz, Vz, b, C, D}}) when (isBlack(C) and isBlack(D)) -> blackToken({Ky, Vy, Cy, {Kx, Vx, b, A, B}, {Kz, Vz, r, C, D}}); % Above is one of many clauses for the delfix function. Below is the full isBlack function:- % An empty node (or subtree) or a black node (or subtree) are black (true), everything else is red (false). isBlack({}) -> true; isBlack({_, b, _, _, _}) -> true; isBlack(_) -> false. Now, rewriting the code for delfix as an if-statement wouldn't work without modifying a bunch of other things, because if C or D aren't black, there are other different patterns (not shown above) that might match. Expanding the tuples C and the D into something along the lines of {Kc, Vc, b, Lc, Rc} to match looks ugly, and would ignore the case where either might be empty subtrees {}. I hope that my question actually makes sense :) If you've come this far in the email then thanks for bearing with me and my long, newbie question. On a side node, I tried to create an account at the trapexit forum; but once I reach the page after agreeing to the conditions of the forums I get a huge SQL error message - something is wrong with the whole registration mechanism.... Cheers, /Fuad -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay@REDACTED Fri Jun 13 06:57:57 2008 From: jay@REDACTED (Jay Nelson) Date: Thu, 12 Jun 2008 21:57:57 -0700 Subject: [erlang-questions] lists documentation error Message-ID: The documentation for lists:sublist/3 gives an example to show that there is no error for start+length > the length of the list: > lists:sublist([1,2,3,4], 5, 2). [] However, in R12B-2 I try the following: > lists:sublist([1,2,3,4], 6, 2). ** exception error: no function clause matching lists:nthtail(1,[]) in function lists:sublist/3 jay From dougedmunds@REDACTED Fri Jun 13 08:25:29 2008 From: dougedmunds@REDACTED (Doug Edmunds) Date: Thu, 12 Jun 2008 23:25:29 -0700 Subject: [erlang-questions] (newbie) Using Functions as Guards in Erlang In-Reply-To: <909c265f0806122106v88a09b5l1a2554c9c7a553fb@mail.gmail.com> References: <909c265f0806122106v88a09b5l1a2554c9c7a553fb@mail.gmail.com> Message-ID: check out the discussion on http://wrongnotes.blogspot.com/2007/09/little-erlang.html "The solution is to call the function and store it in a variable before we use it in the guard conditions." -dae 2008/6/12 Fuad Tabba : > Hi, > > I've run into problem trying to use functions as guards in Erlang. I looked > through the manual, and > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dougedmunds@REDACTED Fri Jun 13 08:37:51 2008 From: dougedmunds@REDACTED (Doug Edmunds) Date: Thu, 12 Jun 2008 23:37:51 -0700 Subject: [erlang-questions] lists documentation error In-Reply-To: References: Message-ID: The Type information for lists:sublist/3 places this restriction on Start: Start = 1..(length(List1)+1) As long as Start does not exceed length(List1)+1, Start + Length can be > than the length of the list. For example: lists:sublist([1,2,3,4], 4, 2000). is ok. -dae On Thu, Jun 12, 2008 at 9:57 PM, Jay Nelson wrote: > The documentation for lists:sublist/3 gives an example to show that > there is no error for start+length > the length of the list: > > > lists:sublist([1,2,3,4], 5, 2). > [] > > However, in R12B-2 I try the following: > > > lists:sublist([1,2,3,4], 6, 2). > ** exception error: no function clause matching lists:nthtail(1,[]) > in function lists:sublist/3 > > jay > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From btolputt@REDACTED Fri Jun 13 02:02:01 2008 From: btolputt@REDACTED (Benjamin Tolputt) Date: Fri, 13 Jun 2008 10:02:01 +1000 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: References: <48504EAC.8000500@cs.ntua.gr> <48505B35.6070006@bigpond.net.au> Message-ID: <4851B8F9.3070104@bigpond.net.au> Please see comments embedded below. Torbjorn Tornkvist wrote: > This is an interesting question, i.e how much do you really gain > from using hipe (native) compiled code? > This is a question I sadly cannot answer for Windows without access to HiPE on the platform *laugh* That said, for Linux platforms & the like - the performance would depend on the code being executed (as your next comment referred to). I would be interested in seeing a benchmark on the performance gains over a variety of different tasks. > I suspect that in (Erlang written) production systems, I/O is where most > of the time is spent. So using Hipe compiled code will not have any real > impact. If would love to be proven wrong here, so if someone has got any > figures to present... My thoughts on this are "Yes & no". Yes, I believe most production systems written in Erlang that are *currently deployed* would be highly IO bound. However, this I believe is an artefact of Erlang's origins more than a good excuse to not bother with HiPE on other platforms. Take for example, Wings 3D, my favourite modelling application. Some functions are highly CPU bound (smoothing & rendering for instance). Or Vendetta Online's Kourier back-end system controlling & simulating NPC agents throughout their virtual space game. Or my AIML interpreter prototype (i.e. Alice chatbot for Erlang), which is bound on IO only for the initial input & the resulting output. Or my experiments with Erlang NEAT (neural network agent evolution). And so on. My point is, Erlang is not only good for a variety of non-IO bound application, it is already being used for such applications and their number is growing as the news of Erlang spreads. Erlang is already past IO bound applications and as such we should look beyond this performance limitation when considering Erlang features. --Ben From mats.cronqvist@REDACTED Fri Jun 13 09:38:23 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Fri, 13 Jun 2008 09:38:23 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: <1213255185.32143.5.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> References: <194477.83170.qm@web38804.mail.mud.yahoo.com> <484FC689.7090807@gmail.com> <484FFB46.40500@gmail.com> <1213255185.32143.5.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> Message-ID: <485223EF.6010007@gmail.com> Bengt Kleberg wrote: > Greetings, > > The numbers I have seen for Erlang (on AXD301) is nine nines. Would not > "factor of 10" mean that the perl phone switch ''only'' have to reach > eight nines? > > > bengt > > On Wed, 2008-06-11 at 18:20 +0200, Mats Cronqvist wrote: > >> kind of like implementing a phone switch with 10,000 concurrent tasks >> and get nine nines reliability in perl. >> no, any old Erlang system can handle 100,000 concurrent tasks. mats From ulf.wiger@REDACTED Fri Jun 13 10:20:25 2008 From: ulf.wiger@REDACTED (Ulf Wiger (TN/EAB)) Date: Fri, 13 Jun 2008 10:20:25 +0200 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: <3dbc6d1c0806121259g2877fb21s96c26730ba09abf8@mail.gmail.com> References: <3dbc6d1c0806121259g2877fb21s96c26730ba09abf8@mail.gmail.com> Message-ID: <48522DC9.5020503@ericsson.com> Robert Virding skrev: > > Yes, I know. I will have to start a name war to get back my name. > Infect R12 with a virus which crashes all using the new built in > module re. Or change the name of my module. :-) > > Robert OTP owns the un-prefixed namespace, you know. ;-) BR, Ulf W From jachym.holecek@REDACTED Fri Jun 13 10:35:34 2008 From: jachym.holecek@REDACTED (Jachym Holecek) Date: Fri, 13 Jun 2008 10:35:34 +0200 Subject: [erlang-questions] (newbie) Using Functions as Guards in Erlang In-Reply-To: References: <909c265f0806122106v88a09b5l1a2554c9c7a553fb@mail.gmail.com> Message-ID: On Fri, 13 Jun 2008 08:25:29 +0200, Doug Edmunds wrote: > check out the discussion on > http://wrongnotes.blogspot.com/2007/09/little-erlang.html > > "The solution is to call the function and store it in a > variable before we use it in the guard conditions." Yes, but it's a solution to the particular case you hit working with if/case/fun; not to the general inability to use custom functions in guards. Clearly, you can't "store it in a variable" when you're at toplevel context (which is what Fuad has been asking about) or when the object you'd like to inspect comes from the pattern immediately preceding the guard. -- Jachym From bengt.kleberg@REDACTED Fri Jun 13 10:37:16 2008 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Fri, 13 Jun 2008 10:37:16 +0200 Subject: [erlang-questions] clueless performance question In-Reply-To: <485223EF.6010007@gmail.com> References: <194477.83170.qm@web38804.mail.mud.yahoo.com> <484FC689.7090807@gmail.com> <484FFB46.40500@gmail.com> <1213255185.32143.5.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> <485223EF.6010007@gmail.com> Message-ID: <1213346236.32143.28.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> So the factor was on the number of processes used, not the resulting reliability. Interesting point of view. That makes me wonder where you applied the factor on for the Wide Finder problem. Lines of code? bengt On Fri, 2008-06-13 at 09:38 +0200, Mats Cronqvist wrote: > Bengt Kleberg wrote: > > Greetings, > > > > The numbers I have seen for Erlang (on AXD301) is nine nines. Would not > > "factor of 10" mean that the perl phone switch ''only'' have to reach > > eight nines? > > > > > > bengt > > > > On Wed, 2008-06-11 at 18:20 +0200, Mats Cronqvist wrote: > > > >> kind of like implementing a phone switch with 10,000 concurrent tasks > >> and get nine nines reliability in perl. > >> > no, any old Erlang system can handle 100,000 concurrent tasks. > > mats > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From fuad@REDACTED Fri Jun 13 11:43:54 2008 From: fuad@REDACTED (Fuad Tabba) Date: Fri, 13 Jun 2008 21:43:54 +1200 Subject: [erlang-questions] (newbie) Using Functions as Guards in Erlang In-Reply-To: References: <909c265f0806122106v88a09b5l1a2554c9c7a553fb@mail.gmail.com> Message-ID: <909c265f0806130243k22bc308cr1f33f67492aa139b@mail.gmail.com> Thanks Doug and Jachym. Jachym is right, Doug's suggestion would work if the guard was inside my function rather than at a top level context. I know that I could rewrite my code and convert it to a bunch of if statements, but I can't think of a way to write such code without it looking ugly. Being able to use a function with a guard seems like it would be the most elegant solution. Any suggestions? Cheers, /Fuad On Fri, Jun 13, 2008 at 8:35 PM, Jachym Holecek wrote: > On Fri, 13 Jun 2008 08:25:29 +0200, Doug Edmunds > wrote: > >> check out the discussion on >> http://wrongnotes.blogspot.com/2007/09/little-erlang.html >> >> "The solution is to call the function and store it in a >> variable before we use it in the guard conditions." >> > > Yes, but it's a solution to the particular case you hit > working with if/case/fun; not to the general inability > to use custom functions in guards. > > Clearly, you can't "store it in a variable" when you're > at toplevel context (which is what Fuad has been asking > about) or when the object you'd like to inspect comes > from the pattern immediately preceding the guard. > > -- Jachym > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rapsey@REDACTED Fri Jun 13 11:53:16 2008 From: rapsey@REDACTED (Rapsey) Date: Fri, 13 Jun 2008 11:53:16 +0200 Subject: [erlang-questions] inet_reply message Message-ID: <97619b170806130253r3e209ec5ofb6169813260c567@mail.gmail.com> Can someone tell me what is a {inet_reply, Socket, ok} message? Sergej -------------- next part -------------- An HTML attachment was scrubbed... URL: From gleber.p@REDACTED Fri Jun 13 12:34:14 2008 From: gleber.p@REDACTED (Gleb Peregud) Date: Fri, 13 Jun 2008 12:34:14 +0200 Subject: [erlang-questions] (newbie) Using Functions as Guards in Erlang In-Reply-To: <909c265f0806122106v88a09b5l1a2554c9c7a553fb@mail.gmail.com> References: <909c265f0806122106v88a09b5l1a2554c9c7a553fb@mail.gmail.com> Message-ID: <14f0e3620806130334o7766980dk5c0fda052beb0fb4@mail.gmail.com> 2008/6/13 Fuad Tabba : > I looked > through the manual, and discovered that this isn't allowed by the language > in case those functions have side-effects. The thing is, my functions do not > have side effects. Is there a way to convey that message to the compiler and > allow my functions to be used as guards? > > ... > > % Algorithm from > http://sage.mc.yu.edu/kbeen/teaching/algorithms/resources/red-black-tree.html > > % Case B1 > delfix({Ky, Vy, Cy, {Kx, Vx, bb, A, B}, {Kz, Vz, b, C, D}}) when (isBlack(C) > and isBlack(D)) -> > blackToken({Ky, Vy, Cy, {Kx, Vx, b, A, B}, {Kz, Vz, r, C, D}}); > > % Above is one of many clauses for the delfix function. Below is the full > isBlack function:- > > % An empty node (or subtree) or a black node (or subtree) are black (true), > everything else is red (false). > isBlack({}) -> true; > isBlack({_, b, _, _, _}) -> true; > isBlack(_) -> false. You may use macros: -define(is_black(X), ((X =:= {}) orelse (element(2, X) =:= b))). and delfix({Ky, Vy, Cy, {Kx, Vx, bb, A, B}, {Kz, Vz, b, C, D}}) when (?is_black(C) and ?is_black(D)) -> blackToken({Ky, Vy, Cy, {Kx, Vx, b, A, B}, {Kz, Vz, r, C, D}}); I didn't checked this, but it may work Best regards, -- Gleb Peregud http://gleber.pl/ Every minute is to be grasped. Time waits for nobody. -- Inscription on a Zen Gong From jachym.holecek@REDACTED Fri Jun 13 13:19:42 2008 From: jachym.holecek@REDACTED (Jachym Holecek) Date: Fri, 13 Jun 2008 13:19:42 +0200 Subject: [erlang-questions] (newbie) Using Functions as Guards in Erlang In-Reply-To: <909c265f0806130243k22bc308cr1f33f67492aa139b@mail.gmail.com> References: <909c265f0806122106v88a09b5l1a2554c9c7a553fb@mail.gmail.com> <909c265f0806130243k22bc308cr1f33f67492aa139b@mail.gmail.com> Message-ID: On Fri, 13 Jun 2008 11:43:54 +0200, Fuad Tabba wrote: > Being able to use a function with a guard seems like it would be the most > elegant solution. It could be useful and I admit I don't understand why is that prohibited (besides "side effects and infinite loops in guards are Evil" to which one can reply "the compiler can refuse a guard function if it can't prove it to be safe"). Anyway, I'm not a compiler expert, so maybe there's a good reason I'm missing. > Any suggestions? Maybe choose different data representation that allows you to pattern-match node colour? Something along the lines of: -record(rb_node, { left, %% Left subtree. :: rb_node() right, %% Right subtree. :: rb_node() is_black, %% Red or black? :: bool() data}). %% Contents. :: term() Either that, or give up doing all the dispatch in function clauses; ie. live with having explicit 'case' statements (there's nothing wrong about them after all). Or restructure the code in different way, but it's hard to suggest how w/o seeing the rest of the module... HTH, -- Jachym From klphanikar@REDACTED Fri Jun 13 14:32:44 2008 From: klphanikar@REDACTED (Phanikar.K) Date: Fri, 13 Jun 2008 18:02:44 +0530 Subject: [erlang-questions] Target System in erlang Message-ID: Hi, I am trying to make an distributed application through which I will be able to run the same application using remote disk-less nodes. Regarding this issue I am facing some problems . I have created a target system and am able to run it on two nodes with distributed option in the .config file. But Now I wanted to add some more nodes where I don't want even to have any binaries of the modules in the application but should be able to work with the application running on the first two nodes. 1.Is there any way to push the application from any one of the first two nodes on to all the connected nodes as we do with the module using nl(module). Can i do the same for the application as nl(module) works only with the modules but not applications? 2. And if the first two nodes are running on two different machines then do I need to do the hot code loading on both the machines or is there any option that if I update the code on one machine that should be updated on the other machine as these two machines are using an distributed application? Please suggest me if there is any round about method also. This is the second time i am postiong this question can anyone please help me in overcoming this problem Reply me as soon as possible Thanks, With Regards Phanikar -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay@REDACTED Fri Jun 13 15:54:07 2008 From: jay@REDACTED (Jay Nelson) Date: Fri, 13 Jun 2008 06:54:07 -0700 Subject: [erlang-questions] lists documentation error In-Reply-To: References: Message-ID: <37F46628-5B44-4049-AC31-52B024554489@duomark.com> That seems like a good addition to the text of the documentation. The way it is written now implies that any combination of start + length is OK. On Jun 12, 2008, at 11:37 PM, Doug Edmunds wrote: > > The Type information for lists:sublist/3 > places this restriction on Start: > > Start = 1..(length(List1)+1) > > As long as Start does not exceed length(List1)+1, > Start + Length can be > than the length of the list. > > For example: > lists:sublist([1,2,3,4], 4, 2000). > is ok. > > -dae > > On Thu, Jun 12, 2008 at 9:57 PM, Jay Nelson wrote: > The documentation for lists:sublist/3 gives an example to show that > there is no error for start+length > the length of the list: > > > lists:sublist([1,2,3,4], 5, 2). > [] > > However, in R12B-2 I try the following: > > > lists:sublist([1,2,3,4], 6, 2). > ** exception error: no function clause matching lists:nthtail(1,[]) > in function lists:sublist/3 > > jay > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From thomasl_erlang@REDACTED Fri Jun 13 15:00:56 2008 From: thomasl_erlang@REDACTED (Thomas Lindgren) Date: Fri, 13 Jun 2008 06:00:56 -0700 (PDT) Subject: [erlang-questions] (newbie) Using Functions as Guards in Erlang In-Reply-To: Message-ID: <15116.68105.qm@web38801.mail.mud.yahoo.com> --- Jachym Holecek wrote: > It could be useful and I admit I don't understand > why is that prohibited > (besides > "side effects and infinite loops in guards are Evil" > to which one can > reply "the > compiler can refuse a guard function if it can't > prove it to be safe"). Arguably, that's already the case :-) > Anyway, > I'm not a compiler expert, so maybe there's a good > reason I'm missing. There are some potential compiler gains from having easy-to-analyze guards (e.g., you can see whether clauses are mutually exclusive more easily/often and can easily reorder tests in guards), but I think this restriction is mostly for historical reasons. Consider that you can write arbitrary "guards" using standard Erlang expressions, e.g.: case catch (E1 andalso E2 andalso ... E3) of true -> Body; _ -> end or even the useful case catch abstype:is_abstype(X) of true -> Body; _ -> end So, as far as I'm concerned, if you in practice end up writing your complex tests using case instead of guards, the guard syntax becomes less and less relevant. Why hang on to such a restriction when it just leads to more convoluted programs? Nowadays I'm thus in favour of allowing arbitrary guards (possibly disallowing side-effects by dynamic checking, but perhaps I'm just reluctant to let go entirely :-). Best, Thomas PS. The concurrent logic languages contemporary with early Erlang in the 1980s also required flat guards, but in those cases there was good reason to do so: it was simply too hard to implement efficient arbitrary guards. From rvirding@REDACTED Fri Jun 13 16:41:27 2008 From: rvirding@REDACTED (Robert Virding) Date: Fri, 13 Jun 2008 16:41:27 +0200 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: <48522DC9.5020503@ericsson.com> References: <3dbc6d1c0806121259g2877fb21s96c26730ba09abf8@mail.gmail.com> <48522DC9.5020503@ericsson.com> Message-ID: <3dbc6d1c0806130741o5f12d150n73680579f4ecc376@mail.gmail.com> 2008/6/13 Ulf Wiger (TN/EAB) : > Robert Virding skrev: > >> >> Yes, I know. I will have to start a name war to get back my name. >> Infect R12 with a virus which crashes all using the new built in >> module re. Or change the name of my module. :-) >> >> Robert >> > > OTP owns the un-prefixed namespace, you know. ;-) > > BR, > Ulf W Well, in that case they can't use 're' as it is prefixed, it's just not postfixed. Or rather the postfix is very short. :-) Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From hayeah@REDACTED Fri Jun 13 16:57:36 2008 From: hayeah@REDACTED (Howard Yeh) Date: Fri, 13 Jun 2008 07:57:36 -0700 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: <3dbc6d1c0806130741o5f12d150n73680579f4ecc376@mail.gmail.com> References: <3dbc6d1c0806121259g2877fb21s96c26730ba09abf8@mail.gmail.com> <48522DC9.5020503@ericsson.com> <3dbc6d1c0806130741o5f12d150n73680579f4ecc376@mail.gmail.com> Message-ID: why not use rv for *R*obert *V*irding's re? they rhyme. On 6/13/08, Robert Virding wrote: > 2008/6/13 Ulf Wiger (TN/EAB) : > > Robert Virding skrev: > > > > > > > > Yes, I know. I will have to start a name war to get back my name. > > > Infect R12 with a virus which crashes all using the new built in > > > module re. Or change the name of my module. :-) > > > > > > Robert > > > > > > > OTP owns the un-prefixed namespace, you know. ;-) > > > > BR, > > Ulf W > > Well, in that case they can't use 're' as it is prefixed, it's just not > postfixed. Or rather the postfix is very short. :-) > > Robert > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- hayeah.wordpress.com --metacircular thinking From paul-trapexit@REDACTED Fri Jun 13 20:18:12 2008 From: paul-trapexit@REDACTED (Paul Mineiro) Date: Fri, 13 Jun 2008 11:18:12 -0700 (PDT) Subject: [erlang-questions] (newbie) Using Functions as Guards in Erlang In-Reply-To: <15116.68105.qm@web38801.mail.mud.yahoo.com> References: <15116.68105.qm@web38801.mail.mud.yahoo.com> Message-ID: Couldn't one implement the "evaluate into in a variable, then match" strategy with a parse transform? The resulting error messages might confuse (a call to foo/3 leading to a bad match in foo/7, but there's no foo/7 explicit in the code). However the code would look nice. -- p On Fri, 13 Jun 2008, Thomas Lindgren wrote: > > --- Jachym Holecek > wrote: > > > It could be useful and I admit I don't understand > > why is that prohibited > > (besides > > "side effects and infinite loops in guards are Evil" > > to which one can > > reply "the > > compiler can refuse a guard function if it can't > > prove it to be safe"). > > Arguably, that's already the case :-) > > > Anyway, > > I'm not a compiler expert, so maybe there's a good > > reason I'm missing. > > There are some potential compiler gains from having > easy-to-analyze guards (e.g., you can see whether > clauses are mutually exclusive more easily/often and > can easily reorder tests in guards), but I think this > restriction is mostly for historical reasons. Consider > that you can write arbitrary "guards" using standard > Erlang expressions, e.g.: > > case catch (E1 andalso E2 andalso ... E3) of > true -> Body; > _ -> > end > > or even the useful > > case catch abstype:is_abstype(X) of > true -> Body; > _ -> > end > > So, as far as I'm concerned, if you in practice end up > writing your complex tests using case instead of > guards, the guard syntax becomes less and less > relevant. Why hang on to such a restriction when it > just leads to more convoluted programs? Nowadays I'm > thus in favour of allowing arbitrary guards (possibly > disallowing side-effects by dynamic checking, but > perhaps I'm just reluctant to let go entirely :-). > > Best, > Thomas > > PS. The concurrent logic languages contemporary with > early Erlang in the 1980s also required flat guards, > but in those cases there was good reason to do so: it > was simply too hard to implement efficient arbitrary guards. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > In an artificial world, only extremists live naturally. -- Paul Graham From matthew.reilly@REDACTED Fri Jun 13 19:38:36 2008 From: matthew.reilly@REDACTED (Matthew Reilly) Date: Fri, 13 Jun 2008 10:38:36 -0700 Subject: [erlang-questions] (newbie) Using Functions as Guards in Erlang In-Reply-To: References: <909c265f0806122106v88a09b5l1a2554c9c7a553fb@mail.gmail.com> Message-ID: <4852B09C.708@sipphone.com> Jachym Holecek wrote: > On Fri, 13 Jun 2008 08:25:29 +0200, Doug Edmunds > wrote: > >> check out the discussion on >> http://wrongnotes.blogspot.com/2007/09/little-erlang.html >> >> "The solution is to call the function and store it in a >> variable before we use it in the guard conditions." >> > > Yes, but it's a solution to the particular case you hit > working with if/case/fun; not to the general inability > to use custom functions in guards. > > Clearly, you can't "store it in a variable" when you're > at toplevel context (which is what Fuad has been asking > about) or when the object you'd like to inspect comes > from the pattern immediately preceding the guard. > > As similar approach that works for functions is to have two functions. The first calculates the non-guard conditions, then passes them to the second function which can pattern match. e.g.: delfix({Ky, Vy, Cy, {Kx, Vx, bb, A, B}, {Kz, Vz, b, C, D}}) -> delfix1({Ky, Vy, Cy, {Kx, Vx, b, A, B}, {Kz, Vz, r, C, D}},isBlack(A),isBlack(B),isBlack(C),isBlack(D)); delfix1({Ky, Vy, Cy, {Kx, Vx, bb, A, B}, {Kz, Vz, b, C, D}},_,_,true,true) -> blackToken({Ky, Vy, Cy, {Kx, Vx, b, A, B}, {Kz, Vz, r, C, D}}); delfix1(...) -> ... > -- Jachym > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > From per@REDACTED Sat Jun 14 12:17:15 2008 From: per@REDACTED (Per Hedeland) Date: Sat, 14 Jun 2008 12:17:15 +0200 (CEST) Subject: [erlang-questions] Socket keepalive option In-Reply-To: <60ed8a460806120456r295e07d6gc5ee6ecfea75ebd2@mail.gmail.com> Message-ID: <200806141017.m5EAHFp4022034@pluto.hedeland.org> "Greg Burri" wrote: > >Is it possible to set the option {keepalive, Boolean} to a socket by >the server side after the socket has been created ? I certainly think it should be, but haven't actually verified it. >I try to do that with an application under Yaws (http://yaws.hyber.org/) : > >out(A) -> > inet:setopts(A#arg.clisock, [{active, true}, {keepalive, true}]), > [..] > >When the server waits and it doesn't send data I capture the TCP >stream with Wireshark (http://www.wireshark.org/) and see if there is >some 'Ack' from the server... but I don't see anything.. Depending on your OS, the timeout for starting keepalive probing is on the order of a couple of hours, so you need to be patient.:-) E.g. FreeBSD 7.0: $ sysctl -a | grep tcp.keep net.inet.tcp.keepidle: 7200000 net.inet.tcp.keepintvl: 75000 net.inet.tcp.keepinit: 75000 tcp(4): keepinit Timeout, in milliseconds, for new, non-established TCP connections. keepidle Amount of time, in milliseconds, that the connection must be idle before keepalive probes (if enabled) are sent. keepintvl The interval, in milliseconds, between keepalive probes sent to remote machines. After TCPTV_KEEPCNT (default 8) probes are sent, with no response, the connection is dropped. --Per Hedeland From per@REDACTED Sat Jun 14 12:21:57 2008 From: per@REDACTED (Per Hedeland) Date: Sat, 14 Jun 2008 12:21:57 +0200 (CEST) Subject: [erlang-questions] clueless performance question In-Reply-To: <0AB58BC9-B473-498E-B474-1603F22B908C@scaldeferri.com> Message-ID: <200806141021.m5EALvww022067@pluto.hedeland.org> Kevin Scaldeferri wrote: > >On Jun 11, 2008, at 9:21 AM, Hynek Vychodil wrote: > >> What I can write about WF1 is, this was a little bit ... well I >> write it clearly: "stupid" because WF1 was measured with data file >> cached. It is why this Perl implementation won. This implementation >> mmap file and than forked processes around this file and then rush. >> Each process reads it's mmap forked copy which in normal cause >> random access to disk and must end with far slowest solution. >> >> > >I don't think it can really be considered cheating to take advantage >of features provided by the OS. Hm, I think you misunderstand - it's not the mmap that is cheating, it's the caching. Or do you know of an OS that has the feature of always keeping all files cached in memory? (Let me know so I can avoid it.:-) --Per Hedeland From thomasl_erlang@REDACTED Sat Jun 14 12:56:46 2008 From: thomasl_erlang@REDACTED (Thomas Lindgren) Date: Sat, 14 Jun 2008 03:56:46 -0700 (PDT) Subject: [erlang-questions] (newbie) Using Functions as Guards in Erlang In-Reply-To: Message-ID: <752843.94282.qm@web38801.mail.mud.yahoo.com> --- Paul Mineiro wrote: > Couldn't one implement the "evaluate into in a > variable, then match" > strategy with a parse transform? I realize I might have been a bit unclear in the example. The Ei in a pseudo-guard catch (E1 andalso E2... andalso En) would be metavariables standing for expressions. So a clause fun({byte, X},Y) when X >= 0, X =< 16#ff, integer(Y), X < Y -> ...; ... would become something like fun(T1, T2) -> case catch (begin {byte, X} = T1, Y = T2, true end andalso X >= 0 andalso X =< 16#ff andalso integer(Y) andalso X < Y) of true -> ... _ -> end To answer your question: In principle, yes, but you'd have to be careful to ensure that the transformation was efficient. First, there is possible code duplication. Second, the transform has to play nicely with the compiler. The example above would probably confuse the pattern match compiler, for instance, which might slow things down considerably. Best, Thomas From ahmed.nawras@REDACTED Sat Jun 14 15:34:08 2008 From: ahmed.nawras@REDACTED (Ahmed Ali) Date: Sat, 14 Jun 2008 17:34:08 +0400 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: References: Message-ID: Hi, I understand what you want. Having used log4j myself, I've tried to find something similar for my own. Eventually, I had to write it myself. This module only supports file logs but it also have support for multiple log levels, including critical (fatal) level, different loggers for different modules, size-based rotation...etc. I haven't shared the project yet, as it has some shortcomings I need to address before releasing it, however, I'll announce it soon. I hope this is what you're look for. I'll keep you updated. Best regards, Ahmed Al-Issaei 2008/6/12 devdoer bird : > Hi: > > I wonder whether any logging module for erlang exists,like python's logging > module which support :critical level setting ,filtering ,diffrent handlers > (stream handler,socket handler,ram handler,file handler...). > > Thanks a lot! > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From rvirding@REDACTED Sat Jun 14 16:26:18 2008 From: rvirding@REDACTED (Robert Virding) Date: Sat, 14 Jun 2008 16:26:18 +0200 Subject: [erlang-questions] (newbie) Using Functions as Guards in Erlang In-Reply-To: <909c265f0806130243k22bc308cr1f33f67492aa139b@mail.gmail.com> References: <909c265f0806122106v88a09b5l1a2554c9c7a553fb@mail.gmail.com> <909c265f0806130243k22bc308cr1f33f67492aa139b@mail.gmail.com> Message-ID: <3dbc6d1c0806140726l7dedd840r49fc83e7153d66b5@mail.gmail.com> 2008/6/13 Fuad Tabba : > Thanks Doug and Jachym. > > Jachym is right, Doug's suggestion would work if the guard was inside my > function rather than at a top level context. I know that I could rewrite my > code and convert it to a bunch of if statements, but I can't think of a way > to write such code without it looking ugly. > > Being able to use a function with a guard seems like it would be the most > elegant solution. > It would open a can of worms that I would rather see remain closed, specifically side effects and message passing. Just being able to declare something as "guard safe" wouldn't really help as the system would still have to be able to do something sensible in all cases. Otherswise you would be surprised what users can dream up. "It's hard to make a program foolproof because fools are so ingenious". Any suggestions? I used the following macros for just that case: %% {r,Left,Key,Val,Right} %% {b,Left,Key,Val,Right} %% empty -define(IS_RED(N), (is_tuple(N) andalso element(1, N) == r)). %% -define(IS_BLACK(N), not (is_tuple(N) andalso element(1, N) == r)). -define(IS_BLACK(N), ((is_tuple(N) andalso (element(1, N) == b)) orelse (N == empty))). These are guard safe and usable on the rhs as well. If they are the most efficient I don't know. I have also found another algorithm for deleting if you are interested which is a little faster. At least in my implementation. Deleting in RB-trees is not trivial. Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Sat Jun 14 16:31:03 2008 From: rvirding@REDACTED (Robert Virding) Date: Sat, 14 Jun 2008 16:31:03 +0200 Subject: [erlang-questions] refactoring + lfe? In-Reply-To: <91a2ba3e0806121603m42cbec62i8d2ca3c371b25a8e@mail.gmail.com> References: <91a2ba3e0806121603m42cbec62i8d2ca3c371b25a8e@mail.gmail.com> Message-ID: <3dbc6d1c0806140731h373d1713ke60c4ad10619718f@mail.gmail.com> 2008/6/13 Raoul Duke : > ok, just to be crazy, any thoughts on the chances of LFE working with > Wrangler? > http://www.cs.kent.ac.uk/projects/forse/ > > :) Well, of course LFE works with anything. This goes without saying. Seriously it would be easier to generate LFE code than Erlang or even Core, but this is probably not where the main problems lie. I have a new patch/upgrade with list/binary comprehensions and extended 'cond'. Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Sat Jun 14 16:33:04 2008 From: rvirding@REDACTED (Robert Virding) Date: Sat, 14 Jun 2008 16:33:04 +0200 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: References: <3dbc6d1c0806121259g2877fb21s96c26730ba09abf8@mail.gmail.com> <48522DC9.5020503@ericsson.com> <3dbc6d1c0806130741o5f12d150n73680579f4ecc376@mail.gmail.com> Message-ID: <3dbc6d1c0806140733i6a52c32fjc5a26ea8d399ee39@mail.gmail.com> 2008/6/13 Howard Yeh : > why not use rv for *R*obert *V*irding's re? > > they rhyme. > Yes, could do that of course. Unless I save rv for something really big and important. :-) Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Sat Jun 14 16:35:30 2008 From: rvirding@REDACTED (Robert Virding) Date: Sat, 14 Jun 2008 16:35:30 +0200 Subject: [erlang-questions] Erlang London / Stockholm UG meetings this week In-Reply-To: <484E6F9F.7030703@freiheit.com> References: <4831A6FB.6090007@erlang-consulting.com> <290b3ba10805191807m18f7e730ica029a96cf28e06c@mail.gmail.com> <48327F7A.1020303@erlang-consulting.com> <484E6F9F.7030703@freiheit.com> Message-ID: <3dbc6d1c0806140735x265f77a9x5078e4c057fd3664@mail.gmail.com> I have no idea what has happened to the recordings. Robert 2008/6/10 Christoph Mewes : > Hi. > > Did the recording happen? I can't find it on google yet, so i thought i'd > ask back if there is a recording to wait for. > > Greets, > Fossi > > Francesco Cesarini schrieb: > > Yes, both events will be taped. >> >> Francesco >> -- >> http://www.erlang-consulting.com >> >> t ty wrote: >> >>> Hello, >>> >>> Will both events be taped for those of us unfortunately enough to be >>> on the wrong side of the Atlantic ? >>> >>> Thanks, >>> >>> Tee >>> >>> On Mon, May 19, 2008 at 12:12 PM, Francesco Cesarini >>> wrote: >>> >>> >>>> Hi All, >>>> >>>> a note to remind you that if you are planing to attend Robert Virding's >>>> presentation on Lisp Flavored Erlang on the 22nd of May in Stockholm, >>>> you need to register. There are plenty of spaces available, so spread >>>> the word and bring your friends, but it is good if we know in advance >>>> how many you are just in case we need to move to an even larger lecture >>>> hall. You can sign up following the register link: >>>> http://www.erlang-consulting.com/erlang/events.html#58 >>>> Stockholms >>>> Universitet is kindly providing the lecture room. Fredrik Thulin will >>>> introduce the event and give a brief presentation on the usage of Erlang >>>> within the University. To subscribe to the Stockholm UG mailing list, >>>> send a blank email to erlangstockholm-subscribe@REDACTED >>>> >>>> For those in London, the UG meeting will be on the 21st of May. Hunter >>>> Morris, CTO and co-founder of Smarkets, a London-based betting exchange >>>> startup with a primarily Erlang code base will be presenting their >>>> Erlang Web Framework. For more informaiton on this event and to >>>> register, visit http://www.erlang-consulting.com/erlang/events.html#56 >>>> To subscribe to the London UG mailing list, send a blank email to >>>> erlang-london-subscribe@REDACTED >>>> >>>> See you there, >>>> >>>> Francesco >>>> -- >>>> http://www.erlang-consulting.com >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>>> >>> >>> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> >> > > -- > Christoph Mewes > Dipl.-Wi.-Inform. > > freiheit.com technologies gmbh > Stra?enbahnring 22 / 20251 Hamburg, Germany > fon +49 (0)40 / 890584-0 > fax +49 (0)40 / 890584-20 > HRB Hamburg 70814 > > FE97 1705 44D3 ED72 1E59 7F3E 2D93 C469 042A 43CD > Gesch?ftsf?hrer: Claudia Dietze, Stefan Richter, J?rg Kirchhof > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From saleyn@REDACTED Sat Jun 14 17:30:06 2008 From: saleyn@REDACTED (Serge Aleynikov) Date: Sat, 14 Jun 2008 11:30:06 -0400 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: References: Message-ID: <4853E3FE.4060707@gmail.com> In jungerl you can find a lama contribution, which installs an event handler that supports various severity levels (just like syslog) and sends reports to syslog. For a while now I've been planning to update that contrib with the newer version of lama that supports pluggable logging destinations of two types: 1. Single-line logger (e.g. syslog) that formats the error reports in one line 2. Multi-line logger (e.g. console) that writes the report to a destination similar to how SASL does it. Will try to push this version out when I find some time. Maybe within a month or so. Serge devdoer bird wrote: > Hi: > > I wonder whether any logging module for erlang exists,like python's logging > module which support :critical level setting ,filtering ,diffrent handlers > (stream handler,socket handler,ram handler,file handler...). > > Thanks a lot! > > > > ------------------------------------------------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From yarivsadan@REDACTED Sun Jun 15 09:27:19 2008 From: yarivsadan@REDACTED (Yariv Sadan) Date: Sun, 15 Jun 2008 00:27:19 -0700 Subject: [erlang-questions] cost of operations in ordered_set tables Message-ID: <17244f480806150027p6666bfa3jcd79a840b6f55efc@mail.gmail.com> Hi, What are the costs of insert() and delete() in ordered_set ets tables? Are they O(log n)? Also, what kind of tree is used to store the keys? Is it a balanced tree (e.g. red-black tree)? Thanks, Yariv From rvirding@REDACTED Sun Jun 15 12:44:46 2008 From: rvirding@REDACTED (Robert Virding) Date: Sun, 15 Jun 2008 12:44:46 +0200 Subject: [erlang-questions] rbdict - A dictionary as a Red-Black tree Message-ID: <3dbc6d1c0806150344g4af7320eq8171e970466ff6da@mail.gmail.com> I have just released rbdict, a dictionary as a Red-Black tree. This is a dict compatible dictionary based on Red-Black trees. It supports the full dict interface and is drop-in compatible with both dict and orddict. Documentation is included. There is also an alternate implementation, in rbdict1, which is a little slower but is included as another example of working on red-black trees. These are tricky little bastards so it is helpful to see other implementations. It is down-loadable from trapext.org: http://forum.trapexit.org/viewtopic.php?p=43948#43948 It would be good to see how it fares under the same unit tests as dict, and orddict. Hint, hint! -------------- next part -------------- An HTML attachment was scrubbed... URL: From litaocheng@REDACTED Sun Jun 15 16:59:25 2008 From: litaocheng@REDACTED (=?UTF-8?B?5oiQ56uL5rab?=) Date: Sun, 15 Jun 2008 22:59:25 +0800 Subject: [erlang-questions] the module attribute in erl file Message-ID: hi, everyone! I'm a newbie to erlang. I'm reading Bob's mochiweb source recently. In the mochiweb_request.erl file, the moudule attribute declarations is strage, "-module(mochiweb_request, [Socket, Method, RawPath, Version, Headers]).", I see the erlang reference mannul and search in the google, but find nothing about this fature. Can anyone give me some explain? thank you very much! -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Sun Jun 15 17:32:59 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Sun, 15 Jun 2008 17:32:59 +0200 Subject: [erlang-questions] the module attribute in erl file In-Reply-To: References: Message-ID: <4d08db370806150832u46d967e8ra69809d05fb2c816@mail.gmail.com> Something about this feature is for example here: http://www.lshift.net/blog/2008/05/18/late-binding-with-erlang 2008/6/15 ??? : > hi, everyone! > I'm a newbie to erlang. I'm reading Bob's mochiweb source recently. In the > mochiweb_request.erl file, the moudule attribute declarations is strage, > "-module(mochiweb_request, [Socket, Method, RawPath, Version, Headers]).", I > see the erlang reference mannul and search in the google, but find nothing > about this fature. Can anyone give me some explain? thank you very much! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Sun Jun 15 22:22:00 2008 From: rvirding@REDACTED (Robert Virding) Date: Sun, 15 Jun 2008 22:22:00 +0200 Subject: [erlang-questions] rbsets - A set as a Red-Black tree Message-ID: <3dbc6d1c0806151322l2dbbd8cenb8c3008bfc367b20@mail.gmail.com> I have just released rbsets, a set as a Red-Black tree. This is a sets compatible set implementation based on Red-Black trees. It supports the full setsinterface and is drop-in compatible with both sets and ordsets. Documentation is included. http://forum.trapexit.org/viewtopic.php?p=43949#43949 It would be good to see how it fares under the same unit tests as sets, and ordsets. Hint, hint! -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuad@REDACTED Mon Jun 16 00:33:40 2008 From: fuad@REDACTED (Fuad Tabba) Date: Mon, 16 Jun 2008 10:33:40 +1200 Subject: [erlang-questions] (newbie) Using Functions as Guards in Erlang In-Reply-To: <3dbc6d1c0806140726l7dedd840r49fc83e7153d66b5@mail.gmail.com> References: <909c265f0806122106v88a09b5l1a2554c9c7a553fb@mail.gmail.com> <909c265f0806130243k22bc308cr1f33f67492aa139b@mail.gmail.com> <3dbc6d1c0806140726l7dedd840r49fc83e7153d66b5@mail.gmail.com> Message-ID: <909c265f0806151533v674341d0k5a0aa9a9aacbbea3@mail.gmail.com> Thanks for everyone for your responses. They were definitely helpful, I now understand Erlang and some of these issues a bit better. I ended up using macros (as Gleb and Robert suggested; specifically the macro Gleb posted). I think this is a more elegant solution than doing any other transformations of the code, but I could be wrong... Anyway, I uploaded my code to:- http://www.cs.auckland.ac.nz/~fuad/rbtree.erl http://www.altabba.org/2008/06/full-red-black-tree-implementation-in.html(for some background) If anyone has any comments about my code I would appreciate it. Robert: I've based my delete algorithm on the one at http://sage.mc.yu.edu/kbeen/teaching/algorithms/resources/red-black-tree.html, would definitely be interested in a faster algorithm. Is it the one you're using for the rbdict you posted? Thanks again. Cheers, /Fuad On Sun, Jun 15, 2008 at 2:26 AM, Robert Virding wrote: > 2008/6/13 Fuad Tabba : > >> Thanks Doug and Jachym. >> >> Jachym is right, Doug's suggestion would work if the guard was inside my >> function rather than at a top level context. I know that I could rewrite my >> code and convert it to a bunch of if statements, but I can't think of a way >> to write such code without it looking ugly. >> >> Being able to use a function with a guard seems like it would be the most >> elegant solution. >> > > It would open a can of worms that I would rather see remain closed, > specifically side effects and message passing. Just being able to declare > something as "guard safe" wouldn't really help as the system would still > have to be able to do something sensible in all cases. Otherswise you would > be surprised what users can dream up. "It's hard to make a program > foolproof because fools are so ingenious". > > Any suggestions? > > > I used the following macros for just that case: > > %% {r,Left,Key,Val,Right} > %% {b,Left,Key,Val,Right} > %% empty > > -define(IS_RED(N), (is_tuple(N) andalso element(1, N) == r)). > %% -define(IS_BLACK(N), not (is_tuple(N) andalso element(1, N) == r)). > -define(IS_BLACK(N), > ((is_tuple(N) andalso (element(1, N) == b)) orelse (N == empty))). > > These are guard safe and usable on the rhs as well. If they are the most > efficient I don't know. > > I have also found another algorithm for deleting if you are interested > which is a little faster. At least in my implementation. Deleting in > RB-trees is not trivial. > > Robert > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Jun 16 02:03:15 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 16 Jun 2008 12:03:15 +1200 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: References: Message-ID: <655BA3A6-4F6F-43C2-BFD1-DBC4C12528B3@cs.otago.ac.nz> The README for OTP R12B-3 has says, amongst good stuff: OTP-7181 An experimental module "re" is added to the emulator which interfacesa publicly available regular expression library for Perl-like regular expressions (PCRE). The interface is purely experimental and *will* be subject to change. One way I hope that it will change is that it won't be PCRE. There are two major problems with Perl regular expressions: (A) the design is so bloated that it is extraordinarily hard for people to master them. This is in marked contrast to the wise restraint of POSIX "Extended Regular Expressions". (B) the design is so bloated that they completely miss the whole point of using regular expressions in the first place, which is LINEAR-TIME MATCHING. True, there are some cases where Perl (and PCRE) take exponential time when they could take linear time, but I'm not talking about accidents of the implementation. The design-by-elephantiasis approach has resulted in a matching language which is provably intractable. One of the reasons for (A) is that it is very hard for anyone not intimately familiar with the Perl or PCRE implementation to tell which patterns will be matched efficiently and which will not. There are free implementations of Posix EREs out there that do guarantee matching linear in the amount of data matched, so any RE library for Erlang built on top of C should use such a library whenever possible, falling back on PCRE only when absolutely necessary. This has been discussed at length in this mailing list before. From morph@REDACTED Mon Jun 16 02:46:34 2008 From: morph@REDACTED (John Chandler) Date: Sun, 15 Jun 2008 20:46:34 -0400 Subject: [erlang-questions] clueless newbie can't compile Message-ID: I can't interpret the error message, either. Two different tutorials advise creating a file and compiling it, which seems like a reasonable way to start. To take one, it advises me to create a file, "tut.erl", containing the following: ==== -module(tut). -export([double/1]). double(X) -> 2 * X. ==== And then, within erl, to type "c(tut).". Which also seems fine. What happens is: ==== 1> c(tut). ** exited: {undef,[{compile,file,[tut,[report_errors,report_warnings]]}, {c,c,2}, {erl_eval,do_apply,5}, {shell,exprs,6}, {shell,eval_loop,3}]} ** =ERROR REPORT==== 15-Jun-2008::20:07:51 === Error in process <0.29.0> with exit value: {undef,[{compile,file,[tut, [report_errors,report_warnings]]},{c,c,2},{erl_eval,do_apply,5}, {shell,exprs,6},{shell,eval_loop,3}]} 2> ==== OK, so what am I being told here? Something is undefined, but it's not clear to me what that thing is, or where it's supposed to be, much less what I can do about it. Can someone toss me a clue? I'm using the CEAN package, and while I believe I followed the instructions in the FAQ, I could have made a mistake somewhere. I launched erl from the directory where tut.erl sits. I had, per the FAQ, set the BASEDIR to /usr/local/cean, so I thought maybe for some unpleasant reason the file had to reside there, and copied it over, but the same symptom resulted. Thanks. -jmc From Anthony.Kong@REDACTED Mon Jun 16 03:14:54 2008 From: Anthony.Kong@REDACTED (Anthony Kong) Date: Mon, 16 Jun 2008 11:14:54 +1000 Subject: [erlang-questions] clueless newbie can't compile In-Reply-To: References: Message-ID: Hi, I am wondering which platfrom you are using? Linux? If so, which distribution? The "c(xxx)." is basically a shorthand to some underlying erlang modules. If you have a look at c.erl, c/1 is actually passing the parameters to compile:file/2. So, what is likely to have happen is you have a corrupted installation of erlang, so erl runtime, for some reason cannot find the compile module. Hence the "undef". Try the basic first. Install erlang/otp via the standard package management system of your OS and see if it improves. Then move on to try CEAN. Cheers, Anthony -----Original Message----- From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of John Chandler Sent: Monday, 16 June 2008 10:47 AM To: erlang-questions@REDACTED Subject: [erlang-questions] clueless newbie can't compile I can't interpret the error message, either. Two different tutorials advise creating a file and compiling it, which seems like a reasonable way to start. To take one, it advises me to create a file, "tut.erl", containing the following: ==== -module(tut). -export([double/1]). double(X) -> 2 * X. ==== And then, within erl, to type "c(tut).". Which also seems fine. What happens is: ==== 1> c(tut). ** exited: {undef,[{compile,file,[tut,[report_errors,report_warnings]]}, {c,c,2}, {erl_eval,do_apply,5}, {shell,exprs,6}, {shell,eval_loop,3}]} ** =ERROR REPORT==== 15-Jun-2008::20:07:51 === Error in process <0.29.0> with exit value: {undef,[{compile,file,[tut, [report_errors,report_warnings]]},{c,c,2},{erl_eval,do_apply,5}, {shell,exprs,6},{shell,eval_loop,3}]} 2> ==== OK, so what am I being told here? Something is undefined, but it's not clear to me what that thing is, or where it's supposed to be, much less what I can do about it. Can someone toss me a clue? I'm using the CEAN package, and while I believe I followed the instructions in the FAQ, I could have made a mistake somewhere. I launched erl from the directory where tut.erl sits. I had, per the FAQ, set the BASEDIR to /usr/local/cean, so I thought maybe for some unpleasant reason the file had to reside there, and copied it over, but the same symptom resulted. Thanks. -jmc _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://www.erlang.org/mailman/listinfo/erlang-questions NOTICE This e-mail and any attachments are confidential and may contain copyright material of Macquarie Group Limited or third parties. If you are not the intended recipient of this email you should not read, print, re-transmit, store or act in reliance on this e-mail or any attachments, and should destroy all copies of them. Macquarie Group Limited does not guarantee the integrity of any emails or any attached files. The views or opinions expressed are the author's own and may not reflect the views or opinions of Macquarie Group Limited. From rvirding@REDACTED Mon Jun 16 03:15:45 2008 From: rvirding@REDACTED (Robert Virding) Date: Mon, 16 Jun 2008 03:15:45 +0200 Subject: [erlang-questions] (newbie) Using Functions as Guards in Erlang In-Reply-To: <909c265f0806151533v674341d0k5a0aa9a9aacbbea3@mail.gmail.com> References: <909c265f0806122106v88a09b5l1a2554c9c7a553fb@mail.gmail.com> <909c265f0806130243k22bc308cr1f33f67492aa139b@mail.gmail.com> <3dbc6d1c0806140726l7dedd840r49fc83e7153d66b5@mail.gmail.com> <909c265f0806151533v674341d0k5a0aa9a9aacbbea3@mail.gmail.com> Message-ID: <3dbc6d1c0806151815h52dc7931nd2636e38c95404ec@mail.gmail.com> 2008/6/16 Fuad Tabba : > > Robert: I've based my delete algorithm on the one at > http://sage.mc.yu.edu/kbeen/teaching/algorithms/resources/red-black-tree.html, would definitely be interested in a faster algorithm. Is it the one you're > using for the rbdict you posted? Looking over my code again, it was a while back I wrote it, it looks like the algorithm in the link you mentioned is the one used in the extra file rbdict1.erl I included in the package. I have a macro DBLACK for the doubly black token so it seems like it. I also see that I don't have any implementation of the deletion algorithm in the first haskell version from http://www.cs.kent.ac.uk/people/staff/smk/redblack/rb.html. I will have to try this and see how it matches up with the others. I have a little test program I use for testing various dictionaries which I include here. It is uncommented but generates lots of test cases, compile it with the 'E' option to see what all the macros generate. Macros are really useful but they are much easier in lisp. Should really redo the file in LFE. :-) Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: db_test.erl Type: application/octet-stream Size: 4615 bytes Desc: not available URL: From tony@REDACTED Mon Jun 16 03:21:59 2008 From: tony@REDACTED (Tony Arcieri) Date: Sun, 15 Jun 2008 19:21:59 -0600 Subject: [erlang-questions] Reia - Python/Ruby mix targeting Beam/HiPE In-Reply-To: <20080604125710.A6153@hal.cs.camosun.bc.ca> References: <14f0e3620806041242n20dd0bp5453e6c160694ac7@mail.gmail.com> <20080604125710.A6153@hal.cs.camosun.bc.ca> Message-ID: On Wed, Jun 4, 2008 at 1:57 PM, Deryk Barker wrote: > I was quite interested until I saw that it supports multiple > assignments to a variable. > Hi everyone, I'm Reia's creator. I wasn't quite ready for Reia to wind up here yet, but since it has, here's my responses to the concerns that have been raised. I'll also be doing an official announcement some time in the next few weeks, when I've finished implementing the majority of the features found in sequential Erlang. -- On multiple assignment / modifying receivers: I'm still torn on this. Multiple assignment is not actually supported in Reia yet. I'm not necessarily sure if I really like the idea, but I'm probably still planning on implementing it soon. The basic idea is to use alpha conversion which maps to new identifiers whenever a variable is potentially rebound. This is what Mikael was describing in the first example above. The bodies of imperative loops are decomposed into named, tail recursive functions which return the new values of any potentially rebound variables, and the binding in the scope of the original function is updated when the looping function returns. Quite a few hoops to go through just to remove what's arguably one of Erlang's most important features. Furthermore state in Reia remains immutable which can cause some confusion, since multiple assignment with immutable state doesn't give the same semantics that mutable state does (for example, old references to the same structure are not updated when the structure is "mutated"). This may be particularly confusing with functions that modify their receiver, since a new copy of the receiver is made with the modifications and the original binding updated to point to the new version. All that said, you may see all references to multiple assignment removed from the Wiki soon. I'm still mulling it over. -- On Core Erlang: I'd originally considered having Reia target Core Erlang. However, Reia makes extensive use of erl_eval. Reia's interpreter allows code outside the scope of modules, / functions, which is evaluated using erl_eval. Reia also has an interactive interpreter / REPL which uses erl_eval. Perhaps I'm just ignorant but I know of no way to evaluate Core Erlang. I've noticed many projects which target Core Erlang do not have interactive interpreters and it seems to be for this very reason. I consider an interactive interpreter an essential feature, and erl_eval is also the source of Reia's scripting abilities (i.e. arbitrary code defined outside modules). I'd like to eventually target Core Erlang, but it seems that in order to do so I'd have to write my own eval for Reia's forms or implement conversion of Reia to both Erlang forms and Core Erlang forms, and neither of those sound like particularly desirable options, at least for the time being. If someone knows of a way to eval Core Erlang, it'd certainly resolve all these problems. -- Tony Arcieri medioh.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Mon Jun 16 03:23:24 2008 From: rvirding@REDACTED (Robert Virding) Date: Mon, 16 Jun 2008 03:23:24 +0200 Subject: [erlang-questions] clueless newbie can't compile In-Reply-To: References: Message-ID: <3dbc6d1c0806151823j4966b63ch50b6c2f118286d02@mail.gmail.com> 2008/6/16 John Chandler : > I can't interpret the error message, either. > > And then, within erl, to type "c(tut).". Which also seems fine. What > happens is: > > ==== > 1> c(tut). > ** exited: {undef,[{compile,file,[tut,[report_errors,report_warnings]]}, > {c,c,2}, > {erl_eval,do_apply,5}, > {shell,exprs,6}, > {shell,eval_loop,3}]} ** > > =ERROR REPORT==== 15-Jun-2008::20:07:51 === > Error in process <0.29.0> with exit value: {undef,[{compile,file,[tut, > [report_errors,report_warnings]]},{c,c,2},{erl_eval,do_apply,5}, > {shell,exprs,6},{shell,eval_loop,3}]} Not being familiar with CEAN I don't really know why, but the error report indicates that the function compile:file/2 is undefined, the 'undef ' error. In this case this is caused by Erlang not being able to find the compile module as we know the file/2 function exists in the module. As I said I am not familiar with CEAN so I can't help you with *why* Erlang can't find the module. Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.kleberg@REDACTED Mon Jun 16 08:04:00 2008 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Mon, 16 Jun 2008 08:04:00 +0200 Subject: [erlang-questions] clueless newbie can't compile In-Reply-To: <3dbc6d1c0806151823j4966b63ch50b6c2f118286d02@mail.gmail.com> References: <3dbc6d1c0806151823j4966b63ch50b6c2f118286d02@mail.gmail.com> Message-ID: <1213596240.20837.2.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> Greetings, CEAN does not include all of Erlang/OTP in the initial install. The compiler is not present. You should be able to do: cean:install(compile). bengt On Mon, 2008-06-16 at 03:23 +0200, Robert Virding wrote: > 2008/6/16 John Chandler : > I can't interpret the error message, either. > > And then, within erl, to type "c(tut).". Which also seems > fine. What > happens is: > > ==== > 1> c(tut). > ** exited: > {undef,[{compile,file,[tut,[report_errors,report_warnings]]}, > {c,c,2}, > {erl_eval,do_apply,5}, > {shell,exprs,6}, > {shell,eval_loop,3}]} ** > > =ERROR REPORT==== 15-Jun-2008::20:07:51 === > Error in process <0.29.0> with exit value: > {undef,[{compile,file,[tut, > [report_errors,report_warnings]]},{c,c,2},{erl_eval,do_apply,5}, > {shell,exprs,6},{shell,eval_loop,3}]} > > Not being familiar with CEAN I don't really know why, but the error > report indicates that the function compile:file/2 is undefined, the > 'undef ' error. In this case this is caused by Erlang not being able > to find the compile module as we know the file/2 function exists in > the module. > > As I said I am not familiar with CEAN so I can't help you with *why* > Erlang can't find the module. > > Robert > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From greg.burri@REDACTED Mon Jun 16 08:57:39 2008 From: greg.burri@REDACTED (Greg Burri) Date: Mon, 16 Jun 2008 08:57:39 +0200 Subject: [erlang-questions] Socket keepalive option In-Reply-To: <200806141017.m5EAHFp4022034@pluto.hedeland.org> References: <60ed8a460806120456r295e07d6gc5ee6ecfea75ebd2@mail.gmail.com> <200806141017.m5EAHFp4022034@pluto.hedeland.org> Message-ID: <60ed8a460806152357h136895d9l3b6865a37f51eda4@mail.gmail.com> Thanks for your reply and explanation. I have theses values for a Debian distribution (Linux 2.6.25-2-amd64) : #> sysctl -a | grep keepalive net.ipv4.tcp_keepalive_time = 7200 net.ipv4.tcp_keepalive_probes = 9 net.ipv4.tcp_keepalive_intvl = 75 Theses values are in second, so the tcp_keepalive_timetime is 2 hours. Is it possible to set this value directly from erlang code without changing the global system value ? /Greg On Sat, Jun 14, 2008 at 12:17 PM, Per Hedeland wrote: > "Greg Burri" wrote: >> >>Is it possible to set the option {keepalive, Boolean} to a socket by >>the server side after the socket has been created ? > > I certainly think it should be, but haven't actually verified it. > >>I try to do that with an application under Yaws (http://yaws.hyber.org/) : >> >>out(A) -> >> inet:setopts(A#arg.clisock, [{active, true}, {keepalive, true}]), >> [..] >> >>When the server waits and it doesn't send data I capture the TCP >>stream with Wireshark (http://www.wireshark.org/) and see if there is >>some 'Ack' from the server... but I don't see anything.. > > Depending on your OS, the timeout for starting keepalive probing is on > the order of a couple of hours, so you need to be patient.:-) E.g. > FreeBSD 7.0: > > $ sysctl -a | grep tcp.keep > net.inet.tcp.keepidle: 7200000 > net.inet.tcp.keepintvl: 75000 > net.inet.tcp.keepinit: 75000 > > tcp(4): > > keepinit Timeout, in milliseconds, for new, non-established TCP > connections. > > keepidle Amount of time, in milliseconds, that the connection > must be idle before keepalive probes (if enabled) are > sent. > > keepintvl The interval, in milliseconds, between keepalive > probes sent to remote machines. After TCPTV_KEEPCNT > (default 8) probes are sent, with no response, the > connection is dropped. > > --Per Hedeland > From kenneth.lundin@REDACTED Mon Jun 16 10:18:05 2008 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Mon, 16 Jun 2008 10:18:05 +0200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 Message-ID: Hi, Regarding the new "experimental" builtin regexp functions in the re module our intention is to provide a fully working implementation where we want feedback on the API functions and also other feedback. With "experimental" we don't mean very buggy or low quality but rather that the API's are not stabilized yet and that there are a few optimizations more to do. Our intention with releasing this regexp implementation before the API's are 100% settled is that we believe we can get more and better comments if people actuall can try themselves to use the functions. We would appreciate if we get the comments in relation to the EEP 11 Built in regular expressions in Erlang http://www.erlang.org/eeps/eep-0011.html which is especially written for this purpose. /Regards Kenneth Lundin Erlang/OTP team at Ericsson On Mon, Jun 16, 2008 at 2:03 AM, Richard A. O'Keefe wrote: > The README for OTP R12B-3 has says, amongst good stuff: > OTP-7181 An experimental module "re" is added to the emulator > which interfacesa publicly available regular expression library > for Perl-like regular expressions (PCRE). The interface is purely > experimental and *will* be subject to change. > One way I hope that it will change is that it won't be PCRE. > There are two major problems with Perl regular expressions: > (A) the design is so bloated that it is extraordinarily hard for > people to master them. This is in marked contrast to the > wise restraint of POSIX "Extended Regular Expressions". > (B) the design is so bloated that they completely miss the whole > point of using regular expressions in the first place, which > is LINEAR-TIME MATCHING. True, there are some cases where > Perl (and PCRE) take exponential time when they could take > linear time, but I'm not talking about accidents of the > implementation. The design-by-elephantiasis approach has > resulted in a matching language which is provably intractable. > > One of the reasons for (A) is that it is very hard for anyone not > intimately familiar with the Perl or PCRE implementation to tell > which patterns will be matched efficiently and which will not. > There are free implementations of Posix EREs out there that do > guarantee matching linear in the amount of data matched, so any > RE library for Erlang built on top of C should use such a library > whenever possible, falling back on PCRE only when absolutely > necessary. > This has been discussed at length in this mailing list before. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From H.Li@REDACTED Mon Jun 16 10:47:47 2008 From: H.Li@REDACTED (H.Li@REDACTED) Date: Mon, 16 Jun 2008 09:47:47 +0100 (BST) Subject: [erlang-questions] refactoring + lfe? In-Reply-To: <91a2ba3e0806121603m42cbec62i8d2ca3c371b25a8e@mail.gmail.com> References: <91a2ba3e0806121603m42cbec62i8d2ca3c371b25a8e@mail.gmail.com> Message-ID: <63280.86.153.131.208.1213606067.squirrel@webmail.cs.kent.ac.uk> That may not be easy. Since Wrangler is based on Erlang's syntax and binding rules, it will need substantial modifications, maybe a new frontend, to handle source code written in LFE. Regards, Huiqing > ok, just to be crazy, any thoughts on the chances of LFE working with > Wrangler? > http://www.cs.kent.ac.uk/projects/forse/ > > :) > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From sverker@REDACTED Mon Jun 16 13:59:07 2008 From: sverker@REDACTED (Sverker Eriksson) Date: Mon, 16 Jun 2008 13:59:07 +0200 Subject: [erlang-questions] cost of operations in ordered_set tables In-Reply-To: <17244f480806150027p6666bfa3jcd79a840b6f55efc@mail.gmail.com> References: <17244f480806150027p6666bfa3jcd79a840b6f55efc@mail.gmail.com> Message-ID: <4856558B.3070107@erix.ericsson.se> Hi Ets ordered_set's are AVL-trees. The cost for insert() and delete() are thus O(log n). /Sverker, Erlang/OTP Ericsson Yariv Sadan wrote: > Hi, > > What are the costs of insert() and delete() in ordered_set ets tables? > Are they O(log n)? Also, what kind of tree is used to store the keys? > Is it a balanced tree (e.g. red-black tree)? > > Thanks, > Yariv > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From rvirding@REDACTED Mon Jun 16 16:15:59 2008 From: rvirding@REDACTED (Robert Virding) Date: Mon, 16 Jun 2008 16:15:59 +0200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: References: Message-ID: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> All I can say is that I agree with Richard but expand more on it the EEP. One quick comment is that I wonder if not part of the problem with PCRE and Perl regular expressions is that in one respect they are integrated into Perl and contain features to interact with the rest of Perl. If I remember correctly, for example, you can set Perl variables directly from the RE. Is this what became the named subexprs in PCRE? Robert 2008/6/16 Kenneth Lundin : > Hi, > > Regarding the new "experimental" builtin regexp functions in the re > module our intention is > to provide a fully working implementation where we want feedback on > the API functions and also other feedback. > With "experimental" we don't mean very buggy or low quality but rather > that the API's are not stabilized yet and that there > are a few optimizations more to do. > > Our intention with releasing this regexp implementation before the > API's are 100% settled is that we believe > we can get more and better comments if people actuall can try > themselves to use the functions. > > We would appreciate if we get the comments in relation to the > EEP 11 Built in regular expressions in Erlang > http://www.erlang.org/eeps/eep-0011.html which is especially written > for this purpose. > > /Regards Kenneth Lundin Erlang/OTP team at Ericsson > > > On Mon, Jun 16, 2008 at 2:03 AM, Richard A. O'Keefe > wrote: > > The README for OTP R12B-3 has says, amongst good stuff: > > OTP-7181 An experimental module "re" is added to the emulator > > which interfacesa publicly available regular expression library > > for Perl-like regular expressions (PCRE). The interface is purely > > experimental and *will* be subject to change. > > One way I hope that it will change is that it won't be PCRE. > > There are two major problems with Perl regular expressions: > > (A) the design is so bloated that it is extraordinarily hard for > > people to master them. This is in marked contrast to the > > wise restraint of POSIX "Extended Regular Expressions". > > (B) the design is so bloated that they completely miss the whole > > point of using regular expressions in the first place, which > > is LINEAR-TIME MATCHING. True, there are some cases where > > Perl (and PCRE) take exponential time when they could take > > linear time, but I'm not talking about accidents of the > > implementation. The design-by-elephantiasis approach has > > resulted in a matching language which is provably intractable. > > > > One of the reasons for (A) is that it is very hard for anyone not > > intimately familiar with the Perl or PCRE implementation to tell > > which patterns will be matched efficiently and which will not. > > There are free implementations of Posix EREs out there that do > > guarantee matching linear in the amount of data matched, so any > > RE library for Erlang built on top of C should use such a library > > whenever possible, falling back on PCRE only when absolutely > > necessary. > > This has been discussed at length in this mailing list before. > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Mon Jun 16 16:42:15 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Mon, 16 Jun 2008 16:42:15 +0200 Subject: [erlang-questions] cost of operations in ordered_set tables In-Reply-To: <4856558B.3070107@erix.ericsson.se> References: <17244f480806150027p6666bfa3jcd79a840b6f55efc@mail.gmail.com> <4856558B.3070107@erix.ericsson.se> Message-ID: <4d08db370806160742j48ab71e7h8707e02c327add3@mail.gmail.com> I have tried primitive benchmark: -module(ets_bench). -export([start/2, insert/2]). start(N, Type) -> S = self(), Sub = spawn_link( fun() -> ok = io:format("| ~15w ", [Type]), Tab = ets:new(a, [Type, private]), insert(Tab, N), ets:delete_all_objects(Tab), {InsertTime, ok} = timer:tc(?MODULE, insert, [Tab, N]), ok = io:format("| ~10g ", [InsertTime/N]), ok = io:format("| ~10g ", [ets:info(Tab, memory)/N]), Counter = fun(_,Acc) -> Acc+1 end, {CountTime, N} = timer:tc(ets, foldl, [Counter, 0, Tab]), ok = io:format("| ~10g |~n", [CountTime/N]), S ! {self(), ok} end ), receive {Sub, ok} -> ok after (N div 10) + 1 -> exit(timeout) end. insert(_Tab, 0) -> ok; insert(Tab, N) -> ets:insert(Tab, {N}), insert(Tab, N-1). And I can see little bit unexpected result: > lists:foreach(fun(N) -> io:format("N: ~10w~n| ~15s | ~10s | ~10s | ~10s |~n", [N, "Type", "Insert[us]", "Size[B]", "Select[us]"]), lists:foreach(fun(T) -> ets_bench:start(N, T) end, [set, ordered_set, bag, duplicate_bag]) end, [100, 1000, 10000, 100000, 1000000, 10000000]). N: 100 | Type | Insert[us] | Size[B] | Select[us] | | set | 1.56000 | 11.8800 | 2.37000 | | ordered_set | 1.70000 | 10.8100 | 2.30000 | | bag | 1.54000 | 11.8800 | 2.28000 | | duplicate_bag | 1.55000 | 11.8800 | 2.29000 | N: 1000 | Type | Insert[us] | Size[B] | Select[us] | | set | 1.53600 | 9.28800 | 2.12400 | | ordered_set | 1.88800 | 10.0810 | 2.08700 | | bag | 1.53700 | 9.28800 | 2.12100 | | duplicate_bag | 1.53600 | 9.28800 | 2.11500 | N: 10000 | Type | Insert[us] | Size[B] | Select[us] | | set | 1.75870 | 9.18230 | 2.12930 | | ordered_set | 1.95290 | 10.0081 | 2.11430 | | bag | 1.79200 | 9.18230 | 2.15800 | | duplicate_bag | 1.81070 | 9.18230 | 2.17460 | N: 100000 | Type | Insert[us] | Size[B] | Select[us] | | set | 2.01674 | 9.14623 | 2.26671 | | ordered_set | 1.97474 | 10.0008 | 2.03010 | | bag | 1.99262 | 9.14623 | 2.19376 | | duplicate_bag | 1.98019 | 9.14623 | 2.19390 | N: 1000000 | Type | Insert[us] | Size[B] | Select[us] | | set | 2.89089 | 9.14377 | 2.25434 | | ordered_set | 2.14992 | 10.0001 | 2.06282 | | bag | 2.88939 | 9.14377 | 2.26617 | | duplicate_bag | 2.90703 | 9.14377 | 2.27656 | N: 1000000 | Type | Insert[us] | Size[B] | Select[us] | | set | 2.89800 | 9.14377 | 2.26243 | | ordered_set | 2.17359 | 10.0001 | 2.04451 | | bag | 2.89416 | 9.14377 | 2.26667 | | duplicate_bag | 2.89497 | 9.14377 | 2.28294 | ok I have expected that ordered_set goes worse and worse in compare to other types. Is it caused by ordered insert and using foldl instead lookup? On Mon, Jun 16, 2008 at 1:59 PM, Sverker Eriksson wrote: > Hi > > Ets ordered_set's are AVL-trees. The cost for insert() and delete() are > thus O(log n). > > /Sverker, Erlang/OTP Ericsson > > > Yariv Sadan wrote: > > Hi, > > > > What are the costs of insert() and delete() in ordered_set ets tables? > > Are they O(log n)? Also, what kind of tree is used to store the keys? > > Is it a balanced tree (e.g. red-black tree)? > > > > Thanks, > > Yariv > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From dougedmunds@REDACTED Mon Jun 16 16:46:17 2008 From: dougedmunds@REDACTED (Doug Edmunds) Date: Mon, 16 Jun 2008 07:46:17 -0700 Subject: [erlang-questions] The re module documentation Message-ID: Random notes on R12B-3 (June 11, 2008) documentation of the re module http://www.erlang.org/doc/man/re.html 1. misspelling of 'be' in the sentence "There should bee no need to go directly to the PCRE library documentation" 2. Need to define 'opaque' as used in mp() = Opaque datatype containing a compiled regular expression. Hopefully it means something other than as defined by http://www.merriam-webster.com/dictionary/opaque "2 a*:* hard to understand or explain <*opaque* prose> b*:* obtuse , thickheaded " A link to where opaque is defined would be helpful. -dae -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Mon Jun 16 16:55:22 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Mon, 16 Jun 2008 16:55:22 +0200 Subject: [erlang-questions] cost of operations in ordered_set tables In-Reply-To: <4d08db370806160742j48ab71e7h8707e02c327add3@mail.gmail.com> References: <17244f480806150027p6666bfa3jcd79a840b6f55efc@mail.gmail.com> <4856558B.3070107@erix.ericsson.se> <4d08db370806160742j48ab71e7h8707e02c327add3@mail.gmail.com> Message-ID: <4d08db370806160755x3715fd57ne931e8c2cff71509@mail.gmail.com> On Mon, Jun 16, 2008 at 4:42 PM, Hynek Vychodil wrote: > I have tried primitive benchmark: > > -module(ets_bench). > > -export([start/2, insert/2]). > > start(N, Type) -> > S = self(), > Sub = spawn_link( fun() -> > ok = io:format("| ~15w ", [Type]), > Tab = ets:new(a, [Type, private]), > insert(Tab, N), > ets:delete_all_objects(Tab), > {InsertTime, ok} = timer:tc(?MODULE, insert, [Tab, N]), > ok = io:format("| ~10g ", [InsertTime/N]), > ok = io:format("| ~10g ", [ets:info(Tab, memory)/N]), > Counter = fun(_,Acc) -> Acc+1 end, > {CountTime, N} = timer:tc(ets, foldl, [Counter, 0, Tab]), > ok = io:format("| ~10g |~n", [CountTime/N]), > S ! {self(), ok} > end ), > receive > {Sub, ok} -> ok > after (N div 10) + 1 -> exit(timeout) > end. > > insert(_Tab, 0) -> ok; > insert(Tab, N) -> ets:insert(Tab, {N}), insert(Tab, N-1). > > And I can see little bit unexpected result: > > > lists:foreach(fun(N) -> io:format("N: ~10w~n| ~15s | ~10s | ~10s | ~10s > |~n", [N, "Type", "Insert[us]", "Size[B]", "Select[us]"]), > lists:foreach(fun(T) -> ets_bench:start(N, T) end, [set, ordered_set, bag, > duplicate_bag]) end, [100, 1000, 10000, 100000, 1000000, 10000000]). > N: 100 > | Type | Insert[us] | Size[B] | Select[us] | > | set | 1.56000 | 11.8800 | 2.37000 | > | ordered_set | 1.70000 | 10.8100 | 2.30000 | > | bag | 1.54000 | 11.8800 | 2.28000 | > | duplicate_bag | 1.55000 | 11.8800 | 2.29000 | > N: 1000 > | Type | Insert[us] | Size[B] | Select[us] | > | set | 1.53600 | 9.28800 | 2.12400 | > | ordered_set | 1.88800 | 10.0810 | 2.08700 | > | bag | 1.53700 | 9.28800 | 2.12100 | > | duplicate_bag | 1.53600 | 9.28800 | 2.11500 | > N: 10000 > | Type | Insert[us] | Size[B] | Select[us] | > | set | 1.75870 | 9.18230 | 2.12930 | > | ordered_set | 1.95290 | 10.0081 | 2.11430 | > | bag | 1.79200 | 9.18230 | 2.15800 | > | duplicate_bag | 1.81070 | 9.18230 | 2.17460 | > N: 100000 > | Type | Insert[us] | Size[B] | Select[us] | > | set | 2.01674 | 9.14623 | 2.26671 | > | ordered_set | 1.97474 | 10.0008 | 2.03010 | > | bag | 1.99262 | 9.14623 | 2.19376 | > | duplicate_bag | 1.98019 | 9.14623 | 2.19390 | > N: 1000000 > | Type | Insert[us] | Size[B] | Select[us] | > | set | 2.89089 | 9.14377 | 2.25434 | > | ordered_set | 2.14992 | 10.0001 | 2.06282 | > | bag | 2.88939 | 9.14377 | 2.26617 | > | duplicate_bag | 2.90703 | 9.14377 | 2.27656 | Sorry, I mashed result for 10 millions, result is: N: 10000000 | Type | Insert[us] | Size[B] | Select[us] | | set | 3.36146 | 9.14344 | 2.35820 | | ordered_set | 2.35257 | 10.0000 | 2.10989 | | bag | 3.36846 | 9.14344 | 2.40425 | | duplicate_bag | 3.36811 | 9.14344 | 2.40532 | Another result with different CPU performance settings: N: 100 | Type | Insert[us] | Size[B] | Select[us] | | set | 0.520000 | 11.8800 | 0.820000 | | ordered_set | 0.570000 | 10.8100 | 0.790000 | | bag | 0.510000 | 11.8800 | 0.770000 | | duplicate_bag | 0.520000 | 11.8800 | 0.780000 | N: 1000 | Type | Insert[us] | Size[B] | Select[us] | | set | 0.514000 | 9.28800 | 0.698000 | | ordered_set | 0.578000 | 10.0810 | 0.692000 | | bag | 0.513000 | 9.28800 | 0.734000 | | duplicate_bag | 0.515000 | 9.28800 | 0.705000 | N: 10000 | Type | Insert[us] | Size[B] | Select[us] | | set | 0.590200 | 9.18230 | 0.710100 | | ordered_set | 0.645100 | 10.0081 | 0.758000 | | bag | 0.638600 | 9.18230 | 0.753200 | | duplicate_bag | 0.960300 | 9.18230 | 0.839500 | N: 100000 | Type | Insert[us] | Size[B] | Select[us] | | set | 0.702480 | 9.14623 | 0.763710 | | ordered_set | 0.657220 | 10.0008 | 0.736930 | | bag | 0.696980 | 9.14623 | 0.764570 | | duplicate_bag | 0.733030 | 9.14623 | 0.762530 | N: 1000000 | Type | Insert[us] | Size[B] | Select[us] | | set | 1.30601 | 9.14377 | 0.805855 | | ordered_set | 0.745067 | 10.0001 | 0.699019 | | bag | 1.32297 | 9.14377 | 0.827663 | | duplicate_bag | 1.30111 | 9.14377 | 0.818840 | N: 10000000 | Type | Insert[us] | Size[B] | Select[us] | | set | 1.59831 | 9.14344 | 0.833202 | | ordered_set | 0.796910 | 10.0000 | 0.698836 | | bag | 1.60898 | 9.14344 | 0.844491 | | duplicate_bag | 1.59895 | 9.14344 | 0.847049 | Original purpose of those tests was compare ets to MySQL engine and ets is less than 10 times slower than MySQL core engine. This is not as bad as I expect. > ok > > I have expected that ordered_set goes worse and worse in compare to other > types. Is it caused by ordered insert and using foldl instead lookup? > > > On Mon, Jun 16, 2008 at 1:59 PM, Sverker Eriksson < > sverker@REDACTED> wrote: > >> Hi >> >> Ets ordered_set's are AVL-trees. The cost for insert() and delete() are >> thus O(log n). >> >> /Sverker, Erlang/OTP Ericsson >> >> >> Yariv Sadan wrote: >> > Hi, >> > >> > What are the costs of insert() and delete() in ordered_set ets tables? >> > Are they O(log n)? Also, what kind of tree is used to store the keys? >> > Is it a balanced tree (e.g. red-black tree)? >> > >> > Thanks, >> > Yariv >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://www.erlang.org/mailman/listinfo/erlang-questions >> > >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > --Hynek (Pichi) Vychodil -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Mon Jun 16 17:03:54 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Mon, 16 Jun 2008 17:03:54 +0200 Subject: [erlang-questions] The re module documentation In-Reply-To: References: Message-ID: <4d08db370806160803i1d318301tda736dff973baee4@mail.gmail.com> 2008/6/16 Doug Edmunds : > Random notes on R12B-3 (June 11, 2008) documentation > of the re module > http://www.erlang.org/doc/man/re.html > > 1. misspelling of 'be' in the sentence > "There should bee no need to go directly to the PCRE library documentation" > > 2. Need to define 'opaque' as used in > > mp() = Opaque datatype containing a compiled regular expression. > > Hopefully it means something other than as defined by > http://www.merriam-webster.com/dictionary/opaque > > "2 a*:* hard to understand or explain <*opaque* prose> > b*:* obtuse , thickheaded " > > A link to where opaque is defined would be helpful. http://www.askoxford.com/concise_oed/opaque?view=uk *1* not able to be seen through; not transparent. I hope this meaning :-) > > -dae > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From mats.cronqvist@REDACTED Mon Jun 16 17:08:20 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Mon, 16 Jun 2008 17:08:20 +0200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> References: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> Message-ID: <485681E4.5080402@gmail.com> Robert Virding wrote: > All I can say is that I agree with Richard but expand more on it the EEP. > > One quick comment is that I wonder if not part of the problem with > PCRE and Perl regular expressions is that in one respect they are > integrated into Perl and contain features to interact with the rest of > Perl. If I remember correctly, for example, you can set Perl variables > directly from the RE. Is this what became the named subexprs in PCRE? AFAIK, perl does not use PCRE as its regexp engine. the PCRE wikipedia entry says; "The name is misleading, because PCRE is Perl-compatible only if you consider a subset of PCRE's settings and a subset of Perl's regular expression facilities." IMO, the main problem with PCRE is that it allows backreferences. as so eminently described by Russ Cox (*), this is an NP-complete problem. Cox says; "The simplest, most effective strategy for backreferences, taken by the original awk and egrep, is not to implement them." then he goes on to argue the opposite point; "This strategy is no longer practical: users have come to rely on backreferences for at least occasional use, and backreferences are part of the POSIX standard for regular expressions ." i think not allowing backreferences is the correct choice for erlang. after all, erlang is about reliability. and it seems not being POSIX-compliant has worked for 20 years. OTOH, as is pointed out in EEP-11, PCRE has many nice features. but i vaguely remember that there was a flag that turned off backreferences.... mats p.s. there was a mail thread a couple of years ago, where i argued that a linking in PCRE was the way to go. we even implemented it, as a BIF (2). (1) http://swtch.com/~rsc/regexp/regexp1.html (2) http://www.erlang.org/pipermail/erlang-questions/2006-July/021556.html From jachym.holecek@REDACTED Mon Jun 16 17:10:46 2008 From: jachym.holecek@REDACTED (Jachym Holecek) Date: Mon, 16 Jun 2008 17:10:46 +0200 Subject: [erlang-questions] The re module documentation In-Reply-To: References: Message-ID: On Mon, 16 Jun 2008 16:46:17 +0200, Doug Edmunds wrote: > 2. Need to define 'opaque' as used in > > mp() = Opaque datatype containing a compiled regular expression. > > Hopefully it means something other than as defined by > http://www.merriam-webster.com/dictionary/opaque Meaning 1) actually makes good sense in this context. "Opaque datatype" is commonly used to denote "datatype whose shape is none of user's business". So the only way you're allowed to work with the "mp()" type is 1. pass it around unaltered 2. invoke 're' functions on it. In other words, the type is "opaque" because as hard as you look, you're never (supposed to be) able to see its internals. > "2 a*:* hard to understand or explain <*opaque* prose> > b*:* obtuse , > thickheaded " Heh :-) -- Jachym From sverker@REDACTED Mon Jun 16 17:11:45 2008 From: sverker@REDACTED (Sverker Eriksson) Date: Mon, 16 Jun 2008 17:11:45 +0200 Subject: [erlang-questions] The re module documentation In-Reply-To: References: Message-ID: <485682B1.5060605@erix.ericsson.se> http://en.wiktionary.org/wiki/opaque: "(computing) Describes a type for which higher-level callers have no knowledge of data values or their representations; all operations are carried out by the type's defined abstract operators." /Sverker, Erlang/OTP Ericsson Doug Edmunds wrote: > Random notes on R12B-3 (June 11, 2008) documentation > of the re module > http://www.erlang.org/doc/man/re.html > > 1. misspelling of 'be' in the sentence > "There should bee no need to go directly to the PCRE library documentation" > > 2. Need to define 'opaque' as used in > > mp() = Opaque datatype containing a compiled regular expression. > > Hopefully it means something other than as defined by > http://www.merriam-webster.com/dictionary/opaque > > "2 a*:* hard to understand or explain <*opaque* prose> > b*:* obtuse , > thickheaded " > > A link to where opaque is defined would be helpful. > > -dae > > > ------------------------------------------------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-quest From dougedmunds@REDACTED Mon Jun 16 17:35:53 2008 From: dougedmunds@REDACTED (Doug Edmunds) Date: Mon, 16 Jun 2008 08:35:53 -0700 Subject: [erlang-questions] The re module documentation In-Reply-To: <485682B1.5060605@erix.ericsson.se> References: <485682B1.5060605@erix.ericsson.se> Message-ID: On Mon, Jun 16, 2008 at 8:11 AM, Sverker Eriksson wrote: > http://en.wiktionary.org/wiki/opaque: > > "(computing) Describes a type for which higher-level callers have no > knowledge of data values or their representations; all operations are > carried out by the type's defined abstract operators." > > /Sverker, Erlang/OTP Ericsson > > Thanks. The only other usage of opaque datatype that I find in the docs is in /Lib/stdlib-1.15.3/Doc/Html/ets.html ets:is_compiled_ms/1 "This function is used to check if a term is a valid compiled match_spec. The compiled match_spec is an opaque datatype which can *not* be sent between Erlang nodes nor be stored on disk. Any attempt to create an external representation of a compiled match_spec will result in an empty binary (<<>>)." -- dae -------------- next part -------------- An HTML attachment was scrubbed... URL: From mats.cronqvist@REDACTED Mon Jun 16 17:53:06 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Mon, 16 Jun 2008 17:53:06 +0200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: References: Message-ID: <48568C62.2040005@gmail.com> Kenneth Lundin wrote: > Hi, > > Regarding the new "experimental" builtin regexp functions in the re > module our intention is > to provide a fully working implementation where we want feedback on > the API functions and also other feedback. > With "experimental" we don't mean very buggy or low quality but rather > that the API's are not stabilized yet and that there > are a few optimizations more to do. > > Our intention with releasing this regexp implementation before the > API's are 100% settled is that we believe > we can get more and better comments if people actuall can try > themselves to use the functions. > > We would appreciate if we get the comments in relation to the > EEP 11 Built in regular expressions in Erlang > http://www.erlang.org/eeps/eep-0011.html which is especially written > for this purpose. i'd like to add this the EEP11; since implementing regular expressions with backreferences is an NP-complete problem, "re" disallows backreferences. this is achieved by calling pcre_fullinfo() and requiring that PCRE_INFO_BACKREFMAX == 0. from http://www.pcre.org/pcre.txt: int pcre_fullinfo(const pcre *code, const pcre_extra *extra, int what, void *where); [...] The possible values for the third argument are defined in pcre.h, and are as follows: PCRE_INFO_BACKREFMAX Return the number of the highest back reference in the pattern. The fourth argument should point to an int variable. Zero is returned if there are no back references. From kevin@REDACTED Mon Jun 16 17:13:17 2008 From: kevin@REDACTED (Kevin A. Smith) Date: Mon, 16 Jun 2008 11:13:17 -0400 Subject: [erlang-questions] clueless newbie can't compile In-Reply-To: <1213596240.20837.2.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> References: <3dbc6d1c0806151823j4966b63ch50b6c2f118286d02@mail.gmail.com> <1213596240.20837.2.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> Message-ID: I think CEAN also has (had?) a developer version of the install which had things like the compiler installed by default. --Kevin On Jun 16, 2008, at 2:04 AM, Bengt Kleberg wrote: > Greetings, > > CEAN does not include all of Erlang/OTP in the initial install. The > compiler is not present. > You should be able to do: > > cean:install(compile). > > > bengt > > On Mon, 2008-06-16 at 03:23 +0200, Robert Virding wrote: >> 2008/6/16 John Chandler : >> I can't interpret the error message, either. >> >> And then, within erl, to type "c(tut).". Which also seems >> fine. What >> happens is: >> >> ==== >> 1> c(tut). >> ** exited: >> {undef,[{compile,file,[tut,[report_errors,report_warnings]]}, >> {c,c,2}, >> {erl_eval,do_apply,5}, >> {shell,exprs,6}, >> {shell,eval_loop,3}]} ** >> >> =ERROR REPORT==== 15-Jun-2008::20:07:51 === >> Error in process <0.29.0> with exit value: >> {undef,[{compile,file,[tut, >> [report_errors,report_warnings]]},{c,c,2},{erl_eval,do_apply, >> 5}, >> {shell,exprs,6},{shell,eval_loop,3}]} >> >> Not being familiar with CEAN I don't really know why, but the error >> report indicates that the function compile:file/2 is undefined, the >> 'undef ' error. In this case this is caused by Erlang not being able >> to find the compile module as we know the file/2 function exists in >> the module. >> >> As I said I am not familiar with CEAN so I can't help you with *why* >> Erlang can't find the module. >> >> Robert >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From japerk@REDACTED Mon Jun 16 18:11:38 2008 From: japerk@REDACTED (Jacob Perkins) Date: Mon, 16 Jun 2008 09:11:38 -0700 Subject: [erlang-questions] rbdict - A dictionary as a Red-Black tree Message-ID: I ran the following code thru fprof using N=1000 for Mod=dict and Mod=rbdict. The basic results were that rbdict was ~10-20ms slower than dict for both store and fetch. store(Mod, N) -> Fold = fun(I, D) -> Mod:store(I, I, D) end, lists:foldl(Fold, Mod:new(), lists:seq(1, N)). fetch(Mod, N, Dict) -> Map = fun(I) -> Mod:fetch(I, Dict) end, lists:map(Map, lists:seq(1, N)). Jacob -------------- next part -------------- An HTML attachment was scrubbed... URL: From pedzsan@REDACTED Mon Jun 16 18:46:34 2008 From: pedzsan@REDACTED (Perry Smith) Date: Mon, 16 Jun 2008 11:46:34 -0500 Subject: [erlang-questions] AIX and R12B-3 Message-ID: I'm new and I'm trying to compile the latest on AIX 5.3. I get errors: sys/common/erl_check_io.c:534: error: 'struct erl_drv_event_data' has no member named 'reqevents' sys/common/erl_check_io.c:535: error: 'struct erl_drv_event_data' has no member named 'reqevents' (there are other similar errors). In sys/poll.h, for 32 bit platforms, AIX defines events to reqevents and revents to rtnevents. Without these defines, I get errors when the poll structure is used in erl_poll.c. Why that is being used for AIX, I don't know since select is the preferred interface on AIX. I'm assuming that someone has compiled the R12B-2 version for AIX and something changed between 2 and 3. I'm poking around trying to figure out how to work around this. Any suggestions would be welcomed. Thanks, Perry Smith From stonecypher@REDACTED Mon Jun 16 21:12:15 2008 From: stonecypher@REDACTED (John Haugeland) Date: Mon, 16 Jun 2008 13:12:15 -0600 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 Message-ID: <8f24f4b10806161212y1efc7956pa94ab0a423e057b1@mail.gmail.com> > after all, erlang is about reliability. and it seems not being > POSIX-compliant has worked for 20 years. By that logic, nothing in Erlang should ever change again. You might want to look up "argumentum ad antiquitatem". -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Mon Jun 16 21:31:39 2008 From: rvirding@REDACTED (Robert Virding) Date: Mon, 16 Jun 2008 21:31:39 +0200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: References: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> Message-ID: <3dbc6d1c0806161231s32220744v88cf2dae83b54307@mail.gmail.com> 2008/6/16 Tony Finch : > On Mon, 16 Jun 2008, Robert Virding wrote: > > > One quick comment is that I wonder if not part of the problem with PCRE > and > > Perl regular expressions is that in one respect they are integrated into > > Perl and contain features to interact with the rest of Perl. If I > remember > > correctly, for example, you can set Perl variables directly from the RE. > Is > > this what became the named subexprs in PCRE? > > Named subpatterns came to PCRE from Python. There isn't a syntax for > setting arbitrary Perl variables from a subpattern (nor is there in > Python). In perl-5.10, which is the first version that supports named > subexpressions, the named captures become values in a hash table. In > Python the MatchObject result has a group() method for accessing captures > by number or name. Then I am at least partially right, not variables but keys in table. Still you can access them by name anyway which means that the name exporting from the regexp into the surrounding programs works. Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Mon Jun 16 21:43:53 2008 From: rvirding@REDACTED (Robert Virding) Date: Mon, 16 Jun 2008 21:43:53 +0200 Subject: [erlang-questions] rbdict - A dictionary as a Red-Black tree In-Reply-To: References: Message-ID: <3dbc6d1c0806161243u322353a8o4bb51609966e57e0@mail.gmail.com> 2008/6/16 Jacob Perkins : > I ran the following code thru fprof using N=1000 for Mod=dict and > Mod=rbdict. The basic results were that rbdict was ~10-20ms slower than dict > for both store and fetch. > > store(Mod, N) -> > Fold = fun(I, D) -> Mod:store(I, I, D) end, > lists:foldl(Fold, Mod:new(), lists:seq(1, N)). > > fetch(Mod, N, Dict) -> > Map = fun(I) -> Mod:fetch(I, Dict) end, > lists:map(Map, lists:seq(1, N)). This is expected, rb-trees are O(n lg n) but dict is based on a hashing algorithm as is almost O(1). Here are some more comparisons done on a 10k element random list: Insert Fetch Update Delete dict 31.8 6.64 22.345 21.09 rbdict 28.36 8.435 27.11 21.795 gb_trees 22.105 9.61 17.735 15.235 30.94 28.825 24.845 Pardon the alignment but it looks ok with constant width. Gb_trees have two values each for inserting/updating/deletion as it has two functions for each depending on whether you know if the element is in the tree or not. It is faster if you know. The dict modules don't have this separation in the interface. The benefit of using rbdict (and gb_trees) is that it is a much lighter structure. It is also ordered but there is no interface for this. Which one you choose depends of course on your application and it is easy to use the different modules and measure. Even testing with gb_trees is easy as the interfaces are close. Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Mon Jun 16 22:50:43 2008 From: rvirding@REDACTED (Robert Virding) Date: Mon, 16 Jun 2008 22:50:43 +0200 Subject: [erlang-questions] rbdict - A dictionary as a Red-Black tree In-Reply-To: <3dbc6d1c0806161243u322353a8o4bb51609966e57e0@mail.gmail.com> References: <3dbc6d1c0806161243u322353a8o4bb51609966e57e0@mail.gmail.com> Message-ID: <3dbc6d1c0806161350g12bc667uba01e5e500e463c8@mail.gmail.com> 2008/6/16 Robert Virding : > > This is expected, rb-trees are O(n lg n) but dict is based on a hashing > algorithm as is almost O(1). > I made a mistake here as Fuad pointed out to me, rb-trees are of course O(lg n). Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From dnew@REDACTED Mon Jun 16 22:10:15 2008 From: dnew@REDACTED (Darren New) Date: Mon, 16 Jun 2008 13:10:15 -0700 Subject: [erlang-questions] efile_drv.c bug? Message-ID: <4856C8A7.5090603@san.rr.com> Not that I'm following the 15-line macro EV_GET_UINT64, but just looking at it, it looks like it has two shifts by 56 bits and no shift by 48 bits. I suspect the second shift by 56 bits should be a 48 instead. (Offhand, I wonder what sort of testing would have caught that?) Any idea why Windows (XP/SP2/32bit) can't handle files >4G but Linux can with the same release? Is this something expected to get fixed eventually? -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From mog-lists@REDACTED Tue Jun 17 01:15:22 2008 From: mog-lists@REDACTED (mog) Date: Mon, 16 Jun 2008 18:15:22 -0500 Subject: [erlang-questions] ansi control codes in the shell. Message-ID: <4856F40A.2040805@rldn.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi in adding clear option to shell, I also wanted to add ANSI color codes, it seems through out Erlang there is escaping of any control character. It was easy to fix i just added 27 (ESC) as a visible char in ttsl_drv.c . This was easy and worked great, however I see that ssh libraries are checking for this too and I can imagine where else it might be as well. I was wondering if anyone else thinks color printing would be nice to add to shell in ssh interface, and should I submit these patches to Erlang-bugs? I only ask as it seems very much the Erlang team's decision to filter out/ escape these escape sequences. Mog -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIVvQIeq+tARrxhnsRAgDVAJ4zxLfUSryNVHUvEvJ1bOfZPStpxACgpNH4 ASiOF62nM11GRDdRYZaGMUY= =L/S4 -----END PGP SIGNATURE----- From per@REDACTED Tue Jun 17 01:19:30 2008 From: per@REDACTED (Per Hedeland) Date: Tue, 17 Jun 2008 01:19:30 +0200 (CEST) Subject: [erlang-questions] Socket keepalive option In-Reply-To: <60ed8a460806152357h136895d9l3b6865a37f51eda4@mail.gmail.com> Message-ID: <200806162319.m5GNJUD6096992@pluto.hedeland.org> "Greg Burri" wrote: > >Thanks for your reply and explanation. > >I have theses values for a Debian distribution (Linux 2.6.25-2-amd64) : >#> sysctl -a | grep keepalive >net.ipv4.tcp_keepalive_time = 7200 >net.ipv4.tcp_keepalive_probes = 9 >net.ipv4.tcp_keepalive_intvl = 75 > >Theses values are in second, so the tcp_keepalive_timetime is 2 hours. >Is it possible to set this value directly from erlang code without >changing the global system value ? On Linux (only, as far as I know), you can set those parameters per socket with setsockopt() calls, see tcp(7). Of course there is no direct support for those in Erlang, but you should be able to do them with the "raw" form of Option for inet:setopts/2. --Per From ok@REDACTED Tue Jun 17 03:26:45 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 17 Jun 2008 13:26:45 +1200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: <485681E4.5080402@gmail.com> References: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> <485681E4.5080402@gmail.com> Message-ID: <4EB3F3AB-CF72-4B2F-906E-DBC71489F5FD@cs.otago.ac.nz> On 17 Jun 2008, at 3:08 am, Mats Cronqvist wrote: > "This strategy is no longer practical: users have come to rely on > backreferences for at least occasional use, and backreferences are > part > of the POSIX standard for regular expressions > >." That is perhaps misleading. POSIX has several stands for regular expressions. The two principal ones are "Basic Regular Expressions", which DO have backreferences, and "Extended Regular Expressions", which DON'T. If you read the Single Unix Specification version 3, Base Definitions, chapter 9 "Regular Expressions", and you will learn (as I did) that if you want to use +, ?, or | in a regex, Extended Regular Expressions are the only game in town, and you DON'T get back references as well. As far as I am aware there is no POSIX specification for any kind of regular expression that includes _both_ alternatives and back- references. > i think not allowing backreferences is the correct choice for erlang. > after all, erlang is about reliability. and it seems not being > POSIX-compliant has worked for 20 years. The only problem with full compliance with POSIX Extended Regular Expressions is that public interfaces to locale data do not provide enough information to support the [=x=] and [.x.] features. From ok@REDACTED Tue Jun 17 03:30:22 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 17 Jun 2008 13:30:22 +1200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: <48568C62.2040005@gmail.com> References: <48568C62.2040005@gmail.com> Message-ID: On 17 Jun 2008, at 3:53 am, Mats Cronqvist wrote: > i'd like to add this the EEP11; > > since implementing regular expressions with backreferences is an NP- > complete problem, "re" disallows backreferences. this is achieved by > calling pcre_fullinfo() and requiring that PCRE_INFO_BACKREFMAX == 0. It's also advisable to disallow everything non-POSIX. That then becomes a reasonable strategy for trying things out, but PCRE's support for all the cruft gets in the way of efficiency, so it's not really the ideal long-term strategy. From circularfunc@REDACTED Tue Jun 17 03:47:02 2008 From: circularfunc@REDACTED (Circular Function) Date: Tue, 17 Jun 2008 01:47:02 +0000 (GMT) Subject: [erlang-questions] erlang on windows, cant run module Message-ID: <327353.13282.qm@web28314.mail.ukl.yahoo.com> I have the werl.exe in: "C:\Program Files\erl5.6.2\bin\werl.exe" where should I then save my file: Sq = fun(X) -> X*X end. I have it now in the bin so the same driectory as werl.exe but when I do: c(square). I get: Erlang (BEAM) emulator version 5.6.2 [async-threads:0] Eshell V5.6.2? (abort with ^G) 1> c(square). ../square.erl:none: no such file or directory error 2> --- Den l?r 2008-05-31 skrev Edwin Fine : Fr?n: Edwin Fine ?mne: Re: [erlang-questions] run program from commandprompt? define function? Till: circularfunc@REDACTED Datum: l?rdag 31 maj 2008 14.26 You can't define a named function directly in the shell like myfunc() -> ok.. You need to do that in a module. You *can* define a fun in the shell, for example Double = fun(X) -> 2 * X end. Example Eshell V5.6.2? (abort with ^G) 1> Double = fun(X) -> 2 * X end. #Fun 2> Double(2).? 4 3> As for your module named tut.erl, you don't copy it to a bin directory. Just work in the current directory. For example: $ cat tut.erl -module(tut). -export([double/1]). double(X) -> ??? 2 * X. $ erl Erlang (BEAM) emulator version 5.6.2 [source] [64-bit] [smp:4] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.6.2? (abort with ^G) 1> c(tut). {ok,tut} 2> tut:double(2). 4 3> Hope this helps. Edwin 2008/5/31 Circular Function : -module(tut). -export([double/1]). double(X) -> ??? 2 * X. saved that in tut.erl, placed it in the bin/ and tried to run it but it asks for a specific program to run. Erlang (BEAM) emulator version 5.6.2 [async-threads:0] Eshell V5.6.2? (abort with ^G) 1> c(tut). ../tut.erl:none: no such file or directory error 2> so which install do i need? how do i define a function? can i do that directlyin the shell? like in python: def sq(x): return x*x how would i dot hat in erlang? S?k efter k?rleken! Hitta din tvillingsj?l p? Yahoo! Dejting: http://se.meetic.yahoo.net _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://www.erlang.org/mailman/listinfo/erlang-questions __________________________________________________________ Ta semester! - s?k efter resor hos Yahoo! Shopping. J?mf?r pris p? flygbiljetter och hotellrum h?r: http://shopping.yahoo.se/c-169901-resor-biljetter.html?partnerId=96914052 -------------- next part -------------- An HTML attachment was scrubbed... URL: From circularfunc@REDACTED Tue Jun 17 04:14:56 2008 From: circularfunc@REDACTED (Circular Function) Date: Tue, 17 Jun 2008 02:14:56 +0000 (GMT) Subject: [erlang-questions] Recursion in the shell=impossible? Message-ID: <426107.72662.qm@web28308.mail.ukl.yahoo.com> 6> Fac = fun(X) -> N * Fac(N-1). * 1: syntax error before: '.' 6> Fac = fun(X) -> N * Fac(N - 1). * 1: syntax error before: '.' 6> Fac = fun(X) -> N * fac(N - 1). * 1: syntax error before: '.' 6> fac = fun(X) -> N * fac(N - 1). * 1: syntax error before: '.' is it not possible to write recursive functions in the shell? i have to do that ina module? can i write functiosn any other way than making an anonymous one and assigning it to a variable? 6> Sq = fun(X) -> X*X end. #Fun 7> Sq(12). 144 8> ___________________________________________________ S?k efter k?rleken! Hitta din tvillingsj?l p? Yahoo! Dejting: http://ad.doubleclick.net/clk;185753627;24584539;x?http://se.meetic.yahoo.net/index.php?mtcmk=148783 -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.gignac@REDACTED Tue Jun 17 04:47:07 2008 From: martin.gignac@REDACTED (Martin Gignac) Date: Mon, 16 Jun 2008 22:47:07 -0400 Subject: [erlang-questions] erlang on windows, cant run module In-Reply-To: <327353.13282.qm@web28314.mail.ukl.yahoo.com> References: <327353.13282.qm@web28314.mail.ukl.yahoo.com> Message-ID: 2008/6/16 Circular Function : > I have the werl.exe in: > "C:\Program Files\erl5.6.2\bin\werl.exe" > > where should I then save my file: Save the file in 'C:\Program Files\erl5.6.2\usr' if you want to be able to compile it from the prompt directly. This is by default the directory where werl starts if you start it from the Programs menu. You can change this start directory by right-clicking on the Erlang shortcut in the Programs menu and changing the 'Start in' parameter in the Shortcut tab. -Martin From drcabana@REDACTED Tue Jun 17 05:07:19 2008 From: drcabana@REDACTED (David Cabana) Date: Mon, 16 Jun 2008 23:07:19 -0400 Subject: [erlang-questions] Recursion in the shell=impossible? In-Reply-To: <426107.72662.qm@web28308.mail.ukl.yahoo.com> References: <426107.72662.qm@web28308.mail.ukl.yahoo.com> Message-ID: <44ed5e0f0806162007p7ef22540r2eae8c2bdb91ec3b@mail.gmail.com> It can be done, but it requires a subtle technique, the use of the y combinator. Here's a link to a post from Bill Clementson which may shed some light: http://bc.tech.coop/blog/070611.html 2008/6/16 Circular Function : > 6> Fac = fun(X) -> N * Fac(N-1). > * 1: syntax error before: '.' > 6> Fac = fun(X) -> N * Fac(N - 1). > * 1: syntax error before: '.' > 6> Fac = fun(X) -> N * fac(N - 1). > * 1: syntax error before: '.' > 6> fac = fun(X) -> N * fac(N - 1). > * 1: syntax error before: '.' > > is it not possible to write recursive functions in the shell? > i have to do that ina module? > can i write functiosn any other way than making an anonymous one and > assigning it to a variable? > 6> Sq = fun(X) -> X*X end. > #Fun > 7> Sq(12). > 144 > 8> > > ------------------------------ > L?na pengar utan s?kerhet. > S?k och j?mf?r hos Yahoo! Shopping. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.reilly@REDACTED Tue Jun 17 05:38:56 2008 From: matthew.reilly@REDACTED (Matthew Reilly) Date: Mon, 16 Jun 2008 20:38:56 -0700 Subject: [erlang-questions] Recursion in the shell=impossible? In-Reply-To: <426107.72662.qm@web28308.mail.ukl.yahoo.com> References: <426107.72662.qm@web28308.mail.ukl.yahoo.com> Message-ID: <485731D0.9060601@sipphone.com> It's possible, but ugly. You need to create an anonymous function that has an additional argument of type Fun. You then call this function, passing the function to itself. Examples speak louder than words: erl> Fact = fun(N) -> F1 = fun (0,_F) -> 1; (M,F) -> M * F(M-1,F) end, F1(N,F1) end. erl> Fact(5). 120 Circular Function wrote: > 6> Fac = fun(X) -> N * Fac(N-1). > * 1: syntax error before: '.' > 6> Fac = fun(X) -> N * Fac(N - 1). > * 1: syntax error before: '.' > 6> Fac = fun(X) -> N * fac(N - 1). > * 1: syntax error before: '.' > 6> fac = fun(X) -> N * fac(N - 1). > * 1: syntax error before: '.' > > is it not possible to write recursive functions in the shell? > i have to do that ina module? > can i write functiosn any other way than making an anonymous one and > assigning it to a variable? > 6> Sq = fun(X) -> X*X end. > #Fun > 7> Sq(12). > 144 > 8> > > > ------------------------------------------------------------------------ > L?na pengar utan s?kerhet. > S?k och j?mf?r hos Yahoo! Shopping. > > > ------------------------------------------------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From morph@REDACTED Tue Jun 17 06:59:45 2008 From: morph@REDACTED (John Chandler) Date: Tue, 17 Jun 2008 00:59:45 -0400 Subject: [erlang-questions] clueless newbie can't compile In-Reply-To: References: <3dbc6d1c0806151823j4966b63ch50b6c2f118286d02@mail.gmail.com> <1213596240.20837.2.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> Message-ID: <709B22EA-562C-4797-8109-CE4AA698F2D4@growler.net> I thank all who responded. I reinstalled CEAN, and got this: ==== Eshell V5.5.5 (abort with ^G) 1> cean:install(compile). compile was not installed. failed to find dependencies: no_pub_file ok 2> ==== Which is better than what I had been getting before. I see that I installed the Production version, so I guess I'll scorch the earth here again and re-do w/the Developer version. Is that "no_pub_file" message interpretable? I notice that the full release is in cean_base.tar.gz, (~3MB) whereas the "cean library" is downloadable from the FAQ page in the file, cean.tar.gz (113KB). This smaller file appears to contain different versions of each of the cean files, and some that do not appear in cean_base.tar.gz. Does anyone know if it is advisable to extract the smaller file over the tree extracted from the larger file? I would default to "no," unless things still don't work. -jmc On Jun 16, 2008, at 11:13 AM, Kevin A. Smith wrote: > I think CEAN also has (had?) a developer version of the install which > had things like the compiler installed by default. > > --Kevin > On Jun 16, 2008, at 2:04 AM, Bengt Kleberg wrote: > >> Greetings, >> >> CEAN does not include all of Erlang/OTP in the initial install. The >> compiler is not present. >> You should be able to do: >> >> cean:install(compile). >> >> >> bengt >> >> On Mon, 2008-06-16 at 03:23 +0200, Robert Virding wrote: >>> 2008/6/16 John Chandler : >>> I can't interpret the error message, either. >>> >>> And then, within erl, to type "c(tut).". Which also seems >>> fine. What >>> happens is: >>> >>> ==== >>> 1> c(tut). >>> ** exited: >>> {undef,[{compile,file,[tut,[report_errors,report_warnings]]}, >>> {c,c,2}, >>> {erl_eval,do_apply,5}, >>> {shell,exprs,6}, >>> {shell,eval_loop,3}]} ** >>> >>> =ERROR REPORT==== 15-Jun-2008::20:07:51 === >>> Error in process <0.29.0> with exit value: >>> {undef,[{compile,file,[tut, >>> [report_errors,report_warnings]]},{c,c,2},{erl_eval,do_apply, >>> 5}, >>> {shell,exprs,6},{shell,eval_loop,3}]} >>> >>> Not being familiar with CEAN I don't really know why, but the error >>> report indicates that the function compile:file/2 is undefined, the >>> 'undef ' error. In this case this is caused by Erlang not being able >>> to find the compile module as we know the file/2 function exists in >>> the module. >>> >>> As I said I am not familiar with CEAN so I can't help you with *why* >>> Erlang can't find the module. >>> >>> Robert >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions NOTICE: Due to Presidential Executive Orders, the National Security Agency may have read this email without warning, warrant, or notice. They may do this without any judicial or legislative oversight. You have little recourse nor protection except to call for the impeachment of the current President. From morph@REDACTED Tue Jun 17 07:12:01 2008 From: morph@REDACTED (John Chandler) Date: Tue, 17 Jun 2008 01:12:01 -0400 Subject: [erlang-questions] clueless newbie can't compile In-Reply-To: <709B22EA-562C-4797-8109-CE4AA698F2D4@growler.net> References: <3dbc6d1c0806151823j4966b63ch50b6c2f118286d02@mail.gmail.com> <1213596240.20837.2.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> <709B22EA-562C-4797-8109-CE4AA698F2D4@growler.net> Message-ID: <3331E3E2-FB37-4428-98F9-AE6E787B1970@growler.net> Installing the Developer version fixes the problem. I still haven't tried a cean:install, though. On Jun 17, 2008, at 12:59 AM, John Chandler wrote: > I thank all who responded. I reinstalled CEAN, and got this: > ==== > Eshell V5.5.5 (abort with ^G) > 1> cean:install(compile). > compile was not installed. failed to find dependencies: no_pub_file > ok > 2> > ==== > > Which is better than what I had been getting before. I see that I > installed > the Production version, so I guess I'll scorch the earth here again > and re-do > w/the Developer version. Is that "no_pub_file" message interpretable? > > I notice that the full release is in cean_base.tar.gz, (~3MB) whereas > the "cean library" is downloadable from the FAQ page in the file, > cean.tar.gz (113KB). This smaller file appears to contain different > versions of each of the cean files, and some that do not appear in > cean_base.tar.gz. Does anyone know if it is advisable to extract > the smaller file over the tree extracted from the larger file? I > would > default to "no," unless things still don't work. > > -jmc > > On Jun 16, 2008, at 11:13 AM, Kevin A. Smith wrote: > >> I think CEAN also has (had?) a developer version of the install which >> had things like the compiler installed by default. >> >> --Kevin >> On Jun 16, 2008, at 2:04 AM, Bengt Kleberg wrote: >> >>> Greetings, >>> >>> CEAN does not include all of Erlang/OTP in the initial install. The >>> compiler is not present. >>> You should be able to do: >>> >>> cean:install(compile). >>> >>> >>> bengt >>> >>> On Mon, 2008-06-16 at 03:23 +0200, Robert Virding wrote: >>>> 2008/6/16 John Chandler : >>>> I can't interpret the error message, either. >>>> >>>> And then, within erl, to type "c(tut).". Which also seems >>>> fine. What >>>> happens is: >>>> >>>> ==== >>>> 1> c(tut). >>>> ** exited: >>>> {undef,[{compile,file,[tut, >>>> [report_errors,report_warnings]]}, >>>> {c,c,2}, >>>> {erl_eval,do_apply,5}, >>>> {shell,exprs,6}, >>>> {shell,eval_loop,3}]} ** >>>> >>>> =ERROR REPORT==== 15-Jun-2008::20:07:51 === >>>> Error in process <0.29.0> with exit value: >>>> {undef,[{compile,file,[tut, >>>> [report_errors,report_warnings]]},{c,c,2}, >>>> {erl_eval,do_apply, >>>> 5}, >>>> {shell,exprs,6},{shell,eval_loop,3}]} >>>> >>>> Not being familiar with CEAN I don't really know why, but the error >>>> report indicates that the function compile:file/2 is undefined, the >>>> 'undef ' error. In this case this is caused by Erlang not being >>>> able >>>> to find the compile module as we know the file/2 function exists in >>>> the module. >>>> >>>> As I said I am not familiar with CEAN so I can't help you with >>>> *why* >>>> Erlang can't find the module. >>>> >>>> Robert >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions > > NOTICE: Due to Presidential Executive Orders, the National Security > Agency may have read this email without warning, warrant, or > notice. They may do this without any judicial or legislative > oversight. You have little recourse nor protection except to call > for the impeachment of the current President. > NOTICE: Due to Presidential Executive Orders, the National Security Agency may have read this email without warning, warrant, or notice. They may do this without any judicial or legislative oversight. You have little recourse nor protection except to call for the impeachment of the current President. From circularfunc@REDACTED Tue Jun 17 06:53:36 2008 From: circularfunc@REDACTED (Circular Function) Date: Tue, 17 Jun 2008 04:53:36 +0000 (GMT) Subject: [erlang-questions] sticky dir, have to restart shell to compile? Message-ID: <475719.88586.qm@web28302.mail.ukl.yahoo.com> after a failed compile i get the response posted below when trying to recompile. it works if i restart the shell and recompile but do i have to restart the shell to do this? 5> c(math). =ERROR REPORT==== 17-Jun-2008::06:50:19 === Can't load module that resides in sticky dir {error,sticky_directory} 6> ___________________________________________________ S?k efter k?rleken! Hitta din tvillingsj?l p? Yahoo! Dejting: http://ad.doubleclick.net/clk;185753627;24584539;x?http://se.meetic.yahoo.net/index.php?mtcmk=148783 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.kleberg@REDACTED Tue Jun 17 07:42:59 2008 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Tue, 17 Jun 2008 07:42:59 +0200 Subject: [erlang-questions] sticky dir, have to restart shell to compile? In-Reply-To: <475719.88586.qm@web28302.mail.ukl.yahoo.com> References: <475719.88586.qm@web28302.mail.ukl.yahoo.com> Message-ID: <1213681379.5157.8.camel@seasc0642.dyn.rnd.as.sw.ericsson.se> Greetings, This is in the FAQ: 9.7 What is a sticky directory? This typically crops up when playing with examples from the old (last century) Erlang book, for instance: 1> c(sets.erl). {error,sticky_directory} The problem is that there already is a standard library module called 'sets'. The Erlang runtime system is protecting you. The easiest solution is to rename your module, e.g. to mysets.erl. It is also possible to 'un-stick' the directory containing the library module. bengt On Tue, 2008-06-17 at 04:53 +0000, Circular Function wrote: > after a failed compile i get the response posted below when trying to > recompile. > it works if i restart the shell and recompile but do i have to restart > the shell to do this? > > 5> c(math). > > =ERROR REPORT==== 17-Jun-2008::06:50:19 === > Can't load module that resides in sticky dir > {error,sticky_directory} > 6> > > > > > > ______________________________________________________________________ > S?k efter k?rleken! > Hitta din tvillingsj?l p? Yahoo! Dejting: http://se.meetic.yahoo..net > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From bjorn@REDACTED Tue Jun 17 07:48:30 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 17 Jun 2008 07:48:30 +0200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: <485681E4.5080402@gmail.com> References: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> <485681E4.5080402@gmail.com> Message-ID: Mats Cronqvist writes: > i think not allowing backreferences is the correct choice for erlang. > after all, erlang is about reliability. and it seems not being > POSIX-compliant has worked for 20 years. Note that Patrik has implemented the re BIFs to be rescheduable, meaning that even a badly written regular expression will not prevent the emulator from running other Erlang processes (even in the non-SMP emulator). As noted in the EEP, some regexp libraries would have been hard or impossible to make rescheduable. As far as we know, all previous interfaces to regexp libaries have ignored the problem of blocking the emulator for the entire duration of the regexp matching. A badly written regexp would in those implementations would indeed block the entire emulator while it was executing (or one scheduler thread in the SMP emulator). /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From circularfunc@REDACTED Tue Jun 17 09:21:41 2008 From: circularfunc@REDACTED (Circular Function) Date: Tue, 17 Jun 2008 07:21:41 +0000 (GMT) Subject: [erlang-questions] power(N,P), cant get it to work Message-ID: <765646.51510.qm@web28308.mail.ukl.yahoo.com> -module(math). -export([fac/1, sq/1,cube/1,power/2]). fac(N) when N>0 -> ??????? N * fac(N-1); fac(0) -> ??????? 1. ??? sq(X) -> X*X. cube(X) -> X*X*X. power(N,P) -> ??????? if P==0 -> ??????????????? 1; ??????? true -> ??????????????? if P>0 -> ??????????????????????? N * power(N,P - 1); ??????????????? true -> ??????????????????????? 1 / power(N,-1 * P). Erlang (BEAM) emulator version 5.6.2 [async-threads:0] Eshell V5.6.2? (abort with ^G) 1> c(math). ../math.erl:20: syntax error before: '.' ../math.erl:2: function power/2 undefined error 2> am i using the "true ->" correctly? it is like else right? __________________________________________________________ G?r det l?ngsamt? Skaffa dig en snabbare bredbandsuppkoppling. S?k och j?mf?r hos Yahoo! Shopping. http://shopping.yahoo.se/c-100015813-bredband.html?partnerId=96914325 -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlm@REDACTED Tue Jun 17 10:11:01 2008 From: vlm@REDACTED (Lev Walkin) Date: Tue, 17 Jun 2008 01:11:01 -0700 Subject: [erlang-questions] power(N,P), cant get it to work In-Reply-To: <765646.51510.qm@web28308.mail.ukl.yahoo.com> References: <765646.51510.qm@web28308.mail.ukl.yahoo.com> Message-ID: <48577195.6030604@lionet.info> Circular Function wrote: > -module(math). > -export([fac/1, sq/1,cube/1,power/2]). > > fac(N) when N>0 -> > N * fac(N-1); > fac(0) -> > 1. > > sq(X) -> X*X. > > cube(X) -> X*X*X. > > power(N,P) -> > if P==0 -> > 1; > true -> > if P>0 -> > N * power(N,P - 1); > true -> > 1 / power(N,-1 * P). place "end end." instead of "." on the last line. From torben.lehoff@REDACTED Tue Jun 17 10:13:12 2008 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Tue, 17 Jun 2008 10:13:12 +0200 Subject: [erlang-questions] power(N,P), cant get it to work In-Reply-To: <765646.51510.qm@web28308.mail.ukl.yahoo.com> References: <765646.51510.qm@web28308.mail.ukl.yahoo.com> Message-ID: 1. You have to end your if's with an end - see http://www.erlang.org/doc/reference_manual/expressions.html#6.7 2. Consider re-writing power using pattern matching and use guard sequences (quite close to the above link in the reference manaul) to make sure that the input data is okay, e.g., math:power(2,0.5) will give you an infinite loop. Cheers, Torben 2008/6/17 Circular Function : > -module(math). > -export([fac/1, sq/1,cube/1,power/2]). > > fac(N) when N>0 -> > N * fac(N-1); > fac(0) -> > 1. > > sq(X) -> X*X. > > cube(X) -> X*X*X. > > power(N,P) -> > if P==0 -> > 1; > true -> > if P>0 -> > N * power(N,P - 1); > true -> > 1 / power(N,-1 * P). > > > Erlang (BEAM) emulator version 5.6.2 [async-threads:0] > > Eshell V5.6.2 (abort with ^G) > 1> c(math). > ./math.erl:20: syntax error before: '.' > ./math.erl:2: function power/2 undefined > error > 2> > > > am i using the "true ->" correctly? it is like else right? > > ------------------------------ > S?k efter k?rleken! > Hitta din tvillingsj?l p? Yahoo! Dejting: http://se.meetic.yahoo.net > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mats.cronqvist@REDACTED Tue Jun 17 10:51:31 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Tue, 17 Jun 2008 10:51:31 +0200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: <8f24f4b10806161212y1efc7956pa94ab0a423e057b1@mail.gmail.com> References: <8f24f4b10806161212y1efc7956pa94ab0a423e057b1@mail.gmail.com> Message-ID: <48577B13.8000809@gmail.com> John Haugeland wrote: > > after all, erlang is about reliability. and it seems not being > > POSIX-compliant has worked for 20 years. > > By that logic, nothing in Erlang should ever change again. You might > want to look up "argumentum ad antiquitatem". By that logic, nothing in Erlang should ever change again just to become POSIX compliant. You might want to look up "Straw man." mats From mats.cronqvist@REDACTED Tue Jun 17 11:08:48 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Tue, 17 Jun 2008 11:08:48 +0200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: References: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> <485681E4.5080402@gmail.com> Message-ID: <48577F20.9030707@gmail.com> Bjorn Gustavsson wrote: > Mats Cronqvist writes: > > >> i think not allowing backreferences is the correct choice for erlang. >> after all, erlang is about reliability. and it seems not being >> POSIX-compliant has worked for 20 years. >> > > Note that Patrik has implemented the re BIFs to be rescheduable, > meaning that even a badly written regular expression will not prevent > the emulator from running other Erlang processes (even in the non-SMP > emulator). > that is of course an example of the sound thinking that makes the emulator the excellent product it is. even so, if i was responsible for, say, a telecom server product, i would be worried about a feature (backreferences) where it's so easy to slip up and peg the CPU at 100%. i would imagine that my application would run into problems if someone wrote a regexp that used up a CPU-second. > As noted in the EEP, some regexp libraries would have been hard or > impossible to make rescheduable. > a crucial point. fwiw, i think using PCRE is a sound decision. the only reasonable alternative i'm aware of is the plan 9 regexp lib. did you look at that? "Libregexp9 is a port of Plan 9's Unicode-capable regular expression library. It is small and simple and provides the traditional extended regular expressions (without modern complications like {} and various \x character classes). It supports Unicode via wide character or UTF-8 encoding." http://swtch.com/plan9port/unix From tuncer.ayaz@REDACTED Tue Jun 17 12:13:08 2008 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Tue, 17 Jun 2008 12:13:08 +0200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: <48577F20.9030707@gmail.com> References: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> <485681E4.5080402@gmail.com> <48577F20.9030707@gmail.com> Message-ID: <4ac8254d0806170313q1444f609p7a7f620776c937ca@mail.gmail.com> On Tue, Jun 17, 2008 at 11:08 AM, Mats Cronqvist wrote: > Bjorn Gustavsson wrote: >> Mats Cronqvist writes: >> >> >>> i think not allowing backreferences is the correct choice for erlang. >>> after all, erlang is about reliability. and it seems not being >>> POSIX-compliant has worked for 20 years. >>> >> >> Note that Patrik has implemented the re BIFs to be rescheduable, >> meaning that even a badly written regular expression will not prevent >> the emulator from running other Erlang processes (even in the non-SMP >> emulator). >> > > that is of course an example of the sound thinking that makes the > emulator the excellent product it is. > even so, if i was responsible for, say, a telecom server product, i > would be worried about a feature (backreferences) where it's so easy to > slip up and peg the CPU at 100%. i would imagine that my application > would run into problems if someone wrote a regexp that used up a CPU-second. > >> As noted in the EEP, some regexp libraries would have been hard or >> impossible to make rescheduable. >> > > a crucial point. fwiw, i think using PCRE is a sound decision. the > only reasonable alternative i'm aware of is the plan 9 regexp lib. did > you look at that? > "Libregexp9 is a port of Plan 9's Unicode-capable regular expression > library. It is small and simple and provides the traditional extended > regular expressions (without modern complications like {} and various \x > character classes). It supports Unicode via wide character or UTF-8 > encoding." > > http://swtch.com/plan9port/unix I hope you guys don't mind me mentioning that if we discuss regex implementations we should at least make sure to use one with an implementation that uses NFAs. That should not be too hard. http://swtch.com/~rsc/regexp/regexp1.html (Thompson NFA info) http://en.wikipedia.org/wiki/Comparison_of_regular_expression_engines From raimo+erlang-questions@REDACTED Tue Jun 17 12:16:52 2008 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 17 Jun 2008 12:16:52 +0200 Subject: [erlang-questions] efile_drv.c bug? In-Reply-To: <4856C8A7.5090603@san.rr.com> References: <4856C8A7.5090603@san.rr.com> Message-ID: <20080617101652.GA9086@erix.ericsson.se> On Mon, Jun 16, 2008 at 01:10:15PM -0700, Darren New wrote: > Not that I'm following the 15-line macro EV_GET_UINT64, but just looking > at it, it looks like it has two shifts by 56 bits and no shift by 48 > bits. I suspect the second shift by 56 bits should be a 48 instead. Oh yes, well spotted! It will be corrected. > > (Offhand, I wonder what sort of testing would have caught that?) I suppose you noticed that macro is not used, so there is even a question if it was a real bug or not... > > Any idea why Windows (XP/SP2/32bit) can't handle files >4G but Linux can > with the same release? Is this something expected to get fixed eventually? If I recall correctly, file pointers (off_t) are signed in Windows but there are extensions in Unix that use unsigned off_t but ((off_t) -1) as an error return value, so it is possible on many 32 bit Unixes to have almost 4GByte files, but 32 bit Windows is stuck with its 2GByte limit. Someone correct me!? > > -- > Darren New / San Diego, CA, USA (PST) > Helpful housekeeping hints: > Check your feather pillows for holes > before putting them in the washing machine. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From rvirding@REDACTED Tue Jun 17 12:22:31 2008 From: rvirding@REDACTED (Robert Virding) Date: Tue, 17 Jun 2008 12:22:31 +0200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: <48577F20.9030707@gmail.com> References: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> <485681E4.5080402@gmail.com> <48577F20.9030707@gmail.com> Message-ID: <3dbc6d1c0806170322sde8ddb8t8bd24ba79fc1c36e@mail.gmail.com> 2008/6/17 Mats Cronqvist : > > As noted in the EEP, some regexp libraries would have been hard or > > impossible to make rescheduable. > > > > a crucial point. fwiw, i think using PCRE is a sound decision. the > only reasonable alternative i'm aware of is the plan 9 regexp lib. did > you look at that? > "Libregexp9 is a port of Plan 9's Unicode-capable regular expression > library. It is small and simple and provides the traditional extended > regular expressions (without modern complications like {} and various \x > character classes). It supports Unicode via wide character or UTF-8 > encoding." > > http://swtch.com/plan9port/unix Hehe, I checked that out and got an *extreme* deja vu feeling. Which is not surprising as as that is exactly the same RE syntax as in the module regexp. Which is also not surprising as regexp is based an AWK which is an old application from the days before REs became so bloated. It is also very similar to POSIX ERE. I would seriously discuss whether we need all the other stuff. Really need it. Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitriid@REDACTED Tue Jun 17 12:35:02 2008 From: dmitriid@REDACTED (Dmitrii Dimandt) Date: Tue, 17 Jun 2008 13:35:02 +0300 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: <4ac8254d0806170313q1444f609p7a7f620776c937ca@mail.gmail.com> References: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> <485681E4.5080402@gmail.com> <48577F20.9030707@gmail.com> <4ac8254d0806170313q1444f609p7a7f620776c937ca@mail.gmail.com> Message-ID: <402CD2A4-D84D-4F8C-93F8-D0D383287270@gmail.com> >> >> >> "Libregexp9 is a port of Plan 9's Unicode-capable regular expression >> library. It is small and simple and provides the traditional extended >> regular expressions (without modern complications like {} and >> various \x >> character classes). It supports Unicode via wide character or UTF-8 >> encoding." >> >> http://swtch.com/plan9port/unix > {} is very useful feature though... From jachym.holecek@REDACTED Tue Jun 17 12:50:57 2008 From: jachym.holecek@REDACTED (Jachym Holecek) Date: Tue, 17 Jun 2008 12:50:57 +0200 Subject: [erlang-questions] efile_drv.c bug? In-Reply-To: <20080617101652.GA9086@erix.ericsson.se> References: <4856C8A7.5090603@san.rr.com> <20080617101652.GA9086@erix.ericsson.se> Message-ID: On Tue, 17 Jun 2008 12:16:52 +0200, Raimo Niskanen wrote: > On Mon, Jun 16, 2008 at 01:10:15PM -0700, Darren New wrote: >> Any idea why Windows (XP/SP2/32bit) can't handle files >4G but Linux can >> with the same release? Is this something expected to get fixed >> eventually? > > If I recall correctly, file pointers (off_t) are signed in Windows > but there are extensions in Unix that use unsigned off_t but > ((off_t) -1) as an error return value, so it is possible on > many 32 bit Unixes to have almost 4GByte files, but 32 bit > Windows is stuck with its 2GByte limit. > > Someone correct me!? Upon quick inspection off_t seems to be int64_t on NetBSD and 'long int' on Linux. So it's signed on at least two Unices. -- Jachym From vlm@REDACTED Tue Jun 17 12:57:06 2008 From: vlm@REDACTED (Lev Walkin) Date: Tue, 17 Jun 2008 03:57:06 -0700 Subject: [erlang-questions] [erlang-bugs] R11B-5 to R12B-3 forks beams eating 100% CPU In-Reply-To: <485758CD.2020801@lionet.info> References: <485489A8.50902@lionet.info> <485758CD.2020801@lionet.info> Message-ID: <48579882.9010304@lionet.info> Update: after all, it seems that yaws:call_cgi() is a culprit. It invokes the beam's fork(), which interacts badly with underlying threading library: 0x0000000800b8b8ca in pthread_sigmask () from /lib/libpthread.so.2 (gdb) bt #0 0x0000000800b8b8ca in pthread_sigmask () from /lib/libpthread.so.2 #1 0x0000000800b8b876 in sigprocmask () from /lib/libpthread.so.2 #2 0x0000000800b95809 in pthread_mutexattr_init () from /lib/libpthread.so.2 #3 0x0000000800b8883a in fork () from /lib/libpthread.so.2 #4 0x000000000052eea6 in fini_getenv_state () #5 0x0000000000482a1a in erts_open_driver () #6 0x00000000004f0e7d in port_get_data_1 () #7 0x00000000004eef88 in open_port_2 () #8 0x0000000000508e9c in process_main () #9 0x0000000000445701 in erl_start () #10 0x000000000042b39b in main () (gdb) c Continuing. no thread to satisfy query 0x0000000800b8b8ca in pthread_sigmask () from /lib/libpthread.so.2 It turned out that FreeBSD port adds --enable-threads to the erts configuration option. I tried two things: 1. Disabled threading in run time, by specifying +A 0 (my default was +A 32). This setting has no effect on the rating of runaway beams' creation. That is, new beams still sporadically pop out eating 100% CPU. 2. Disabling threading by removing the --enable-threads from the Erlang port's configuration. This appears to fix the issue. I firmly believe there should exist a more legitimate fix. Is there anyone willing to perform a consultant's job of fixing this hanging fork() issue under async-threading/FreeBSD/call_cgi, and receive money in exchange for their services? Lev Walkin wrote: > correction: > > Disabling yaws application DOES seem to have effect on > the rate of runaway beam creation. > > > Lev Walkin wrote: >> SYMPTOMS: >> >> A running, unstressed erlang system appears to spontaneously >> fork off BEAM processes (beam) several times an hour, >> each eating 100% CPU. (If there are more than one at a time, >> they split the CPU accordingly). >> >> INVESTIGATION: >> >> The BEAM processes forked appear to be truly separate processes. >> Killing them (-9) does not seem to harm the main VM, which >> continues to execute the necessary set of applications >> (kernel, stdlib, sasl, ssl, yaws, plus two inhouse ones). >> >> Disabling yaws application does not seem to have any effect on >> the rate of runaway beam creation. >> >> ktrace shows a total absence of any system calls in the runaway >> beam process, suggesting an infinite loop. Killing the process >> results in a single ktrace event: >> >> [root@REDACTED ~]# kdump >> 97445 beam PSIG SIGKILL SIG_DFL >> [root@REDACTED ~]# >> >> The R12B-3 has the following fix: >> >> >> OTP-7289 On Mac OS 10.5 (Leopard), sending to socket >> which the other end closes could cause the >> emulator to consume 100% CPU time. >> (Thanks to Matthias Radestock.) >> >> Since Mac OS is sufficiently similar to FreeBSD, I had hoped >> the OTP-7289 has direct relation to my runaway beam problem. >> Alas, R12B-3 behaves exactly the way R11B-5 and the latter >> versions of Erlang VM behave, that is, runaway beams >> are continuing to appear on a regular basis. >> >> WORKAROUND USED: >> >> Created an external process which periodically detects >> and kills the runaway beams. Ugly. >> >> CONFIGURATION: >> >> Erlang (BEAM) emulator version 5.6.3 [source] [64-bit] >> [async-threads:0] [hipe] [kernel-poll:false] >> >> FreeBSD host 6.3-RELEASE FreeBSD 6.3-RELEASE #0: Wed Jan 16 01:43:02 >> UTC 2008 root@REDACTED:/usr/obj/usr/src/sys/SMP amd64 >> >> However, the system was running all versions of erlang VM >> between R11B-5 and R12B-3 with the same results, so >> it is not version specific. >> >> >> Please suggest further steps to debug or eliminate the problem. >> >> > > _______________________________________________ > erlang-bugs mailing list > erlang-bugs@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-bugs From alex.arnon@REDACTED Tue Jun 17 13:06:11 2008 From: alex.arnon@REDACTED (Alex Arnon) Date: Tue, 17 Jun 2008 14:06:11 +0300 Subject: [erlang-questions] AIX and R12B-3 In-Reply-To: References: Message-ID: <944da41d0806170406n182cab69i8cf57f19f4283088@mail.gmail.com> Have you had any luck finding the difference (or resolving the problem)? If so, can you post a patch? On Mon, Jun 16, 2008 at 7:46 PM, Perry Smith wrote: > I'm new and I'm trying to compile the latest on AIX 5.3. > > I get errors: > > sys/common/erl_check_io.c:534: error: 'struct erl_drv_event_data' has > no member named 'reqevents' > sys/common/erl_check_io.c:535: error: 'struct erl_drv_event_data' has > no member named 'reqevents' > > (there are other similar errors). In sys/poll.h, for 32 bit > platforms, AIX defines events to reqevents and revents to rtnevents. > Without these defines, I get errors when the poll structure is used in > erl_poll.c. Why that is being used for AIX, I don't know since select > is the preferred interface on AIX. > > I'm assuming that someone has compiled the R12B-2 version for AIX and > something changed between 2 and 3. > > I'm poking around trying to figure out how to work around this. Any > suggestions would be welcomed. > > Thanks, > Perry Smith > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From o.girondel@REDACTED Tue Jun 17 13:13:11 2008 From: o.girondel@REDACTED (Olivier Girondel) Date: Tue, 17 Jun 2008 13:13:11 +0200 Subject: [erlang-questions] Use of uuidgen with a mnesia application In-Reply-To: <290b3ba10806061055q54727ef2iebf3d8184325b202@mail.gmail.com> References: <4BB1C16B-9F14-47C6-8043-AF2CCCF040A3@rodanotech.ch> <1212764332.7281.105.camel@localhost> <290b3ba10806061055q54727ef2iebf3d8184325b202@mail.gmail.com> Message-ID: <73b6d6c80806170413n755dbd1cw7c334c524e0e27b2@mail.gmail.com> On Fri, Jun 6, 2008 at 7:55 PM, t ty wrote: > There is an Erlang UUID module called erlUUID which uses OSSP uuid. > > http://nixbit.com/cat//utilities/erluuid/ It is now back online at http://dl.biniou.info/erlang/erluuid/ This is quite old btw, but comments/suggestions are always welcomed -- Olivier From vlm@REDACTED Tue Jun 17 13:19:41 2008 From: vlm@REDACTED (Lev Walkin) Date: Tue, 17 Jun 2008 04:19:41 -0700 Subject: [erlang-questions] efile_drv.c bug? In-Reply-To: References: <4856C8A7.5090603@san.rr.com> <20080617101652.GA9086@erix.ericsson.se> Message-ID: <48579DCD.1070008@lionet.info> Jachym Holecek wrote: > On Tue, 17 Jun 2008 12:16:52 +0200, Raimo Niskanen > wrote: >> On Mon, Jun 16, 2008 at 01:10:15PM -0700, Darren New wrote: >>> Any idea why Windows (XP/SP2/32bit) can't handle files >4G but Linux can >>> with the same release? Is this something expected to get fixed >>> eventually? >> If I recall correctly, file pointers (off_t) are signed in Windows >> but there are extensions in Unix that use unsigned off_t but >> ((off_t) -1) as an error return value, so it is possible on >> many 32 bit Unixes to have almost 4GByte files, but 32 bit >> Windows is stuck with its 2GByte limit. >> >> Someone correct me!? > > Upon quick inspection off_t seems to be int64_t on NetBSD and > 'long int' on Linux. So it's signed on at least two Unices. Single Unix Specification version 3, IEEE Std 1003.1 specifies blkcnt_t and off_t shall be signed integer types. http://www.opengroup.org/onlinepubs/000095399/basedefs/sys/types.h.html -- Lev Walkin vlm@REDACTED From devdoer2@REDACTED Tue Jun 17 13:37:59 2008 From: devdoer2@REDACTED (devdoer bird) Date: Tue, 17 Jun 2008 19:37:59 +0800 Subject: [erlang-questions] some questions about cocurrent accessing of mnesia while doing table fragment Message-ID: Hi: I have some questions about mnesia table fragment. While adding a table fragment for the table (say user table for example),what will happen if reading or writing operation the same table is being performed?Will it cause the table in in-consistent status? Another question,while performing table fragment operation,the operation : " Info=fun(Item)->mnesia:table_info(user,Item) end. mnesia:activity(sync_dirty,Info,[frag_size],mnesia_frag) " becomes very slow,but the dirty_read operation is quite quick. Could you someone explain it to me? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pedzsan@REDACTED Tue Jun 17 13:52:32 2008 From: pedzsan@REDACTED (Perry Smith) Date: Tue, 17 Jun 2008 06:52:32 -0500 Subject: [erlang-questions] AIX and R12B-3 In-Reply-To: <944da41d0806170406n182cab69i8cf57f19f4283088@mail.gmail.com> References: <944da41d0806170406n182cab69i8cf57f19f4283088@mail.gmail.com> Message-ID: Hi Alex, Yes, I have made progress but something odd happened last night. Half my tree was missing for some reason. My file system was full briefly. So, I want to go back and redo what I did and make sure it is all working properly. I hope to be done with it sometime today. Oh, plus, my issue now is the install script is not being called correctly. Should be easy to fix. To the group: I tried "make test" and "make check" and at the top level and both were undefined. How do I run through the tests? Or is that done with just the "make". Last, should the patches be posted here or the patches list? I've subscribed to three lists at this point: questions, patches, and bugs. Thank you, Perry On Jun 17, 2008, at 6:06 AM, Alex Arnon wrote: > Have you had any luck finding the difference (or resolving the > problem)? > If so, can you post a patch? > > > On Mon, Jun 16, 2008 at 7:46 PM, Perry Smith > wrote: > I'm new and I'm trying to compile the latest on AIX 5.3. > > I get errors: > > sys/common/erl_check_io.c:534: error: 'struct erl_drv_event_data' has > no member named 'reqevents' > sys/common/erl_check_io.c:535: error: 'struct erl_drv_event_data' has > no member named 'reqevents' > > (there are other similar errors). In sys/poll.h, for 32 bit > platforms, AIX defines events to reqevents and revents to rtnevents. > Without these defines, I get errors when the poll structure is used in > erl_poll.c. Why that is being used for AIX, I don't know since select > is the preferred interface on AIX. > > I'm assuming that someone has compiled the R12B-2 version for AIX and > something changed between 2 and 3. > > I'm poking around trying to figure out how to work around this. Any > suggestions would be welcomed. > > Thanks, > Perry Smith > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cthulahoops@REDACTED Tue Jun 17 14:24:09 2008 From: cthulahoops@REDACTED (Adam Kelly) Date: Tue, 17 Jun 2008 13:24:09 +0100 Subject: [erlang-questions] lists:unzip breaks dialyzer. Message-ID: <8d1798e90806170524l34fa0adata7da4d66867a2e4a@mail.gmail.com> Hi, I've just run into a weird problem with dialyzer. A module (see below) containing a call to lists:unzip with a list of more than a single tuple as an argument seems to crash dialyzer. Am I doing something wrong? Thanks, Adam. $ cat test_dia.erl -module(test_dia). -export([u/0]). u() -> lists:unzip([{a, 1}, {b, 2}]). $ erlc +debug_info test_dia.erl $ dialyzer -c test_dia.beam Checking whether the initial PLT exists and is up-to-date... yes Proceeding with analysis... =ERROR REPORT==== 17-Jun-2008::13:04:56 === Error in process <0.29.0> with exit value: {function_clause,[{erl_types,t_tuple_args,[{c,tuple_set,[{c,tuple,[{c,atom,[a],any},{c,number,{int_set,[1]},int}],{2,{c,atom,[a],any}}},{c,tuple,[{c,atom,[b],any},{c,number,{int_set,[2]},int}],{2,{c,atom,[b],any}}}],any}]},{erl_bif_types... Analysis failed with error report: {function_clause,[{erl_types,t_tuple_args, [{c,tuple_set,[{...}|...],any}]}, {erl_bif_types,'-type/4-anonymous-373-',1}, {dialyzer_dataflow,do_call,4}, {dialyzer_dataflow,traverse,4}, {dialyzer_dataflow,do_clause,7}, {dialyzer_dataflow,handle_clauses,9}, {dialyzer_dataflow,traverse,...}, {dialyzer_dataflow,...}]} dialyzer: Internal problems were encountered in the analysis. $ dialyzer --version Dialyzer version v1.7.0 From erlang-questions_efine@REDACTED Tue Jun 17 15:01:44 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Tue, 17 Jun 2008 09:01:44 -0400 Subject: [erlang-questions] Socket keepalive option In-Reply-To: <200806162319.m5GNJUD6096992@pluto.hedeland.org> References: <60ed8a460806152357h136895d9l3b6865a37f51eda4@mail.gmail.com> <200806162319.m5GNJUD6096992@pluto.hedeland.org> Message-ID: <6c2563b20806170601u5b903706o9f09859b5f52d1bd@mail.gmail.com> Isn't he asking if there is any way to do this on a per-process basis, rather than for every process in the system? On Mon, Jun 16, 2008 at 7:19 PM, Per Hedeland wrote: > "Greg Burri" wrote: > > > >Thanks for your reply and explanation. > > > >I have theses values for a Debian distribution (Linux 2.6.25-2-amd64) : > >#> sysctl -a | grep keepalive > >net.ipv4.tcp_keepalive_time = 7200 > >net.ipv4.tcp_keepalive_probes = 9 > >net.ipv4.tcp_keepalive_intvl = 75 > > > >Theses values are in second, so the tcp_keepalive_timetime is 2 hours. > >Is it possible to set this value directly from erlang code without > >changing the global system value ? > > On Linux (only, as far as I know), you can set those parameters per > socket with setsockopt() calls, see tcp(7). Of course there is no direct > support for those in Erlang, but you should be able to do them with the > "raw" form of Option for inet:setopts/2. > > --Per > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobias.lindahl@REDACTED Tue Jun 17 15:07:09 2008 From: tobias.lindahl@REDACTED (Tobias Lindahl) Date: Tue, 17 Jun 2008 15:07:09 +0200 Subject: [erlang-questions] lists:unzip breaks dialyzer. In-Reply-To: <8d1798e90806170524l34fa0adata7da4d66867a2e4a@mail.gmail.com> References: <8d1798e90806170524l34fa0adata7da4d66867a2e4a@mail.gmail.com> Message-ID: <4857B6FD.5090301@it.uu.se> Adam Kelly wrote: > Hi, > > I've just run into a weird problem with dialyzer. A module (see > below) containing a call to lists:unzip with a list of more > than a single tuple as an argument seems to crash dialyzer. Am I > doing something wrong? No, this was a bug in Dialyzer. Sometimes it fails to find the bugs in itself... However, you are running R11B-5, and I have fixed the bug since then. I suggest you upgrade if you can. Tobias > Thanks, > Adam. > > $ cat test_dia.erl > -module(test_dia). > -export([u/0]). > > u() -> > lists:unzip([{a, 1}, {b, 2}]). > $ erlc +debug_info test_dia.erl > $ dialyzer -c test_dia.beam > Checking whether the initial PLT exists and is up-to-date... yes > Proceeding with analysis... > =ERROR REPORT==== 17-Jun-2008::13:04:56 === > Error in process <0.29.0> with exit value: > {function_clause,[{erl_types,t_tuple_args,[{c,tuple_set,[{c,tuple,[{c,atom,[a],any},{c,number,{int_set,[1]},int}],{2,{c,atom,[a],any}}},{c,tuple,[{c,atom,[b],any},{c,number,{int_set,[2]},int}],{2,{c,atom,[b],any}}}],any}]},{erl_bif_types... > > > Analysis failed with error report: > {function_clause,[{erl_types,t_tuple_args, > [{c,tuple_set,[{...}|...],any}]}, > {erl_bif_types,'-type/4-anonymous-373-',1}, > {dialyzer_dataflow,do_call,4}, > {dialyzer_dataflow,traverse,4}, > {dialyzer_dataflow,do_clause,7}, > {dialyzer_dataflow,handle_clauses,9}, > {dialyzer_dataflow,traverse,...}, > {dialyzer_dataflow,...}]} > dialyzer: Internal problems were encountered in the analysis. > $ dialyzer --version > Dialyzer version v1.7.0 > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From kostis@REDACTED Tue Jun 17 15:07:41 2008 From: kostis@REDACTED (Kostis Sagonas) Date: Tue, 17 Jun 2008 16:07:41 +0300 Subject: [erlang-questions] lists:unzip breaks dialyzer. In-Reply-To: <8d1798e90806170524l34fa0adata7da4d66867a2e4a@mail.gmail.com> References: <8d1798e90806170524l34fa0adata7da4d66867a2e4a@mail.gmail.com> Message-ID: <4857B71D.5050509@cs.ntua.gr> Adam Kelly wrote: > Hi, > > I've just run into a weird problem with dialyzer. A module (see > below) containing a call to lists:unzip with a list of more > than a single tuple as an argument seems to crash dialyzer. Am I > doing something wrong? > > Thanks, > Adam. > > $ cat test_dia.erl > -module(test_dia). > -export([u/0]). > > u() -> > lists:unzip([{a, 1}, {b, 2}]). > $ erlc +debug_info test_dia.erl > $ dialyzer -c test_dia.beam > Checking whether the initial PLT exists and is up-to-date... yes > Proceeding with analysis... > =ERROR REPORT==== 17-Jun-2008::13:04:56 === > dialyzer: Internal problems were encountered in the analysis. > $ dialyzer --version > Dialyzer version v1.7.0 With the latest Dialyzer version (the one in R12B-3), I cannot reproduce the error. I suggest you upgrade your Erlang/OTP installation to that one -- there are many reasons to do so, not only Dialyzer-specific. If, for whatever reason, this is not possible, and want to solve this issue on your own, look at the definition of lists:unzip/1 types in your Erlang/OTP installation (in the file lib/hipe/cerl/erl_bif_types.erl) and compare them with the following definition: type(lists, unzip, 1, Xs) -> strict(arg_types(lists, unzip, 1), Xs, fun ([Ps]) -> case t_is_nil(Ps) of true -> t_tuple([t_nil(), t_nil()]); false -> % Ps is a proper list of pairs TupleTypes = t_tuple_subtypes(t_list_elements(Ps)), lists:foldl(fun(Tuple, Acc) -> [A, B] = t_tuple_args(Tuple), t_sup(t_tuple([t_list(A), t_list(B)]), Acc) end, t_none(), TupleTypes) end end); for possible differences. Kostis From per@REDACTED Tue Jun 17 15:26:18 2008 From: per@REDACTED (Per Hedeland) Date: Tue, 17 Jun 2008 15:26:18 +0200 (CEST) Subject: [erlang-questions] Socket keepalive option In-Reply-To: <6c2563b20806170601u5b903706o9f09859b5f52d1bd@mail.gmail.com> Message-ID: <200806171326.m5HDQIEC014086@pluto.hedeland.org> "Edwin Fine" wrote: > >Isn't he asking if there is any way to do this on a per-process basis, >rather than for every process in the system? There's no mention of "per-process" in the question, so lacking telepathic capabilities I have to assume that he isn't - what he *is* asking is "from erlang code without changing the global system value" (where "the global system value" is obviously the same as "for every process in the system"), and that's what my answer covered. Either way, "per-socket using setsockopt(), and only on Linux" is the only answer I know of that doesn't involve changing the global system value. --Per >On Mon, Jun 16, 2008 at 7:19 PM, Per Hedeland wrote: > >> "Greg Burri" wrote: >> > >> >Thanks for your reply and explanation. >> > >> >I have theses values for a Debian distribution (Linux 2.6.25-2-amd64) : >> >#> sysctl -a | grep keepalive >> >net.ipv4.tcp_keepalive_time = 7200 >> >net.ipv4.tcp_keepalive_probes = 9 >> >net.ipv4.tcp_keepalive_intvl = 75 >> > >> >Theses values are in second, so the tcp_keepalive_timetime is 2 hours. >> >Is it possible to set this value directly from erlang code without >> >changing the global system value ? >> >> On Linux (only, as far as I know), you can set those parameters per >> socket with setsockopt() calls, see tcp(7). Of course there is no direct >> support for those in Erlang, but you should be able to do them with the >> "raw" form of Option for inet:setopts/2. >> >> --Per From raimo+erlang-questions@REDACTED Tue Jun 17 15:42:04 2008 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 17 Jun 2008 15:42:04 +0200 Subject: [erlang-questions] : efile_drv.c bug? In-Reply-To: <48579DCD.1070008@lionet.info> References: <4856C8A7.5090603@san.rr.com> <20080617101652.GA9086@erix.ericsson.se> <48579DCD.1070008@lionet.info> Message-ID: <20080617134204.GA29340@erix.ericsson.se> On Tue, Jun 17, 2008 at 04:19:41AM -0700, Lev Walkin wrote: > Jachym Holecek wrote: > >On Tue, 17 Jun 2008 12:16:52 +0200, Raimo Niskanen > > wrote: > >>On Mon, Jun 16, 2008 at 01:10:15PM -0700, Darren New wrote: > >>>Any idea why Windows (XP/SP2/32bit) can't handle files >4G but Linux can > >>>with the same release? Is this something expected to get fixed > >>>eventually? > >>If I recall correctly, file pointers (off_t) are signed in Windows > >>but there are extensions in Unix that use unsigned off_t but > >>((off_t) -1) as an error return value, so it is possible on > >>many 32 bit Unixes to have almost 4GByte files, but 32 bit > >>Windows is stuck with its 2GByte limit. > >> > >>Someone correct me!? > > > >Upon quick inspection off_t seems to be int64_t on NetBSD and > >'long int' on Linux. So it's signed on at least two Unices. > > > Single Unix Specification version 3, IEEE Std 1003.1 > specifies > > blkcnt_t and off_t shall be signed integer types. > > http://www.opengroup.org/onlinepubs/000095399/basedefs/sys/types.h.html Well, I am certain I have seen a largefile extension with unsigned 32 bit off_t. It was probably Solaris 6 (or 8 or 5). > > > -- > Lev Walkin > vlm@REDACTED -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From devdoer2@REDACTED Tue Jun 17 15:49:12 2008 From: devdoer2@REDACTED (devdoer bird) Date: Tue, 17 Jun 2008 21:49:12 +0800 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: References: Message-ID: yes, that's the one I need. 2008/6/14, Ahmed Ali : > > Hi, > > I understand what you want. Having used log4j myself, I've tried to > find something similar for my own. Eventually, I had to write it > myself. This module only supports file logs but it also have support > for multiple log levels, including critical (fatal) level, different > loggers for different modules, size-based rotation...etc. > > I haven't shared the project yet, as it has some shortcomings I need > to address before releasing it, however, I'll announce it soon. I hope > this is what you're look for. I'll keep you updated. > > Best regards, > > Ahmed Al-Issaei > > 2008/6/12 devdoer bird : > > Hi: > > > > I wonder whether any logging module for erlang exists,like python's > logging > > module which support :critical level setting ,filtering ,diffrent > handlers > > (stream handler,socket handler,ram handler,file handler...). > > > > Thanks a lot! > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlm@REDACTED Tue Jun 17 16:03:49 2008 From: vlm@REDACTED (Lev Walkin) Date: Tue, 17 Jun 2008 07:03:49 -0700 Subject: [erlang-questions] : efile_drv.c bug? In-Reply-To: <20080617134204.GA29340@erix.ericsson.se> References: <4856C8A7.5090603@san.rr.com> <20080617101652.GA9086@erix.ericsson.se> <48579DCD.1070008@lionet.info> <20080617134204.GA29340@erix.ericsson.se> Message-ID: <4857C445.1090305@lionet.info> Raimo Niskanen wrote: > On Tue, Jun 17, 2008 at 04:19:41AM -0700, Lev Walkin wrote: >> Jachym Holecek wrote: >>> On Tue, 17 Jun 2008 12:16:52 +0200, Raimo Niskanen >>> wrote: >>>> On Mon, Jun 16, 2008 at 01:10:15PM -0700, Darren New wrote: >>>>> Any idea why Windows (XP/SP2/32bit) can't handle files >4G but Linux can >>>>> with the same release? Is this something expected to get fixed >>>>> eventually? >>>> If I recall correctly, file pointers (off_t) are signed in Windows >>>> but there are extensions in Unix that use unsigned off_t but >>>> ((off_t) -1) as an error return value, so it is possible on >>>> many 32 bit Unixes to have almost 4GByte files, but 32 bit >>>> Windows is stuck with its 2GByte limit. >>>> >>>> Someone correct me!? >>> Upon quick inspection off_t seems to be int64_t on NetBSD and >>> 'long int' on Linux. So it's signed on at least two Unices. >> >> Single Unix Specification version 3, IEEE Std 1003.1 >> specifies >> >> blkcnt_t and off_t shall be signed integer types. >> >> http://www.opengroup.org/onlinepubs/000095399/basedefs/sys/types.h.html > > Well, I am certain I have seen a largefile extension with unsigned > 32 bit off_t. It was probably Solaris 6 (or 8 or 5). I believe it might have been size_t, since signed off_t is used to specify a negative seek in lseek(SEEK_{CUR,END}). Anyway newer Solaris 10 headers contain signed off_t ILP32 and LP64 modes. -- Lev Walkin vlm@REDACTED From cthulahoops@REDACTED Tue Jun 17 16:11:04 2008 From: cthulahoops@REDACTED (Adam Kelly) Date: Tue, 17 Jun 2008 15:11:04 +0100 Subject: [erlang-questions] lists:unzip breaks dialyzer. In-Reply-To: <8d1798e90806170710h2b7f0b7y2f0751c4217ffcfc@mail.gmail.com> References: <8d1798e90806170524l34fa0adata7da4d66867a2e4a@mail.gmail.com> <4857B6FD.5090301@it.uu.se> <8d1798e90806170710h2b7f0b7y2f0751c4217ffcfc@mail.gmail.com> Message-ID: <8d1798e90806170711j52a200a8n2b4f5bab77d524fb@mail.gmail.com> On Tue, Jun 17, 2008 at 2:07 PM, Tobias Lindahl wrote: > However, you are running R11B-5, and I have fixed the bug since then. I > suggest you upgrade if you can. Good to know, thank you (and Kostis). Yes, it appears we're suffering from Ubuntu packages being nearly a year old even in the latest release. I've worked around the problem for now, and we'll have to plan an upgrade. Adam. From erlang-questions_efine@REDACTED Tue Jun 17 16:22:40 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Tue, 17 Jun 2008 10:22:40 -0400 Subject: [erlang-questions] Socket keepalive option In-Reply-To: <200806171326.m5HDQIEC014086@pluto.hedeland.org> References: <6c2563b20806170601u5b903706o9f09859b5f52d1bd@mail.gmail.com> <200806171326.m5HDQIEC014086@pluto.hedeland.org> Message-ID: <6c2563b20806170722v5e0ff6cdib2d9ffc6bd597575@mail.gmail.com> You don't have telepathic abilities? Sorry to hear that :) I don't either, so I misunderstood your original answer. Thanks for clarifying it. On Tue, Jun 17, 2008 at 9:26 AM, Per Hedeland wrote: > "Edwin Fine" wrote: > > > >Isn't he asking if there is any way to do this on a per-process basis, > >rather than for every process in the system? > > There's no mention of "per-process" in the question, so lacking > telepathic capabilities I have to assume that he isn't - what he *is* > asking is "from erlang code without changing the global system value" > (where "the global system value" is obviously the same as "for every > process in the system"), and that's what my answer covered. Either way, > "per-socket using setsockopt(), and only on Linux" is the only answer I > know of that doesn't involve changing the global system value. > > --Per > > >On Mon, Jun 16, 2008 at 7:19 PM, Per Hedeland wrote: > > > >> "Greg Burri" wrote: > >> > > >> >Thanks for your reply and explanation. > >> > > >> >I have theses values for a Debian distribution (Linux 2.6.25-2-amd64) : > >> >#> sysctl -a | grep keepalive > >> >net.ipv4.tcp_keepalive_time = 7200 > >> >net.ipv4.tcp_keepalive_probes = 9 > >> >net.ipv4.tcp_keepalive_intvl = 75 > >> > > >> >Theses values are in second, so the tcp_keepalive_timetime is 2 hours. > >> >Is it possible to set this value directly from erlang code without > >> >changing the global system value ? > >> > >> On Linux (only, as far as I know), you can set those parameters per > >> socket with setsockopt() calls, see tcp(7). Of course there is no direct > >> support for those in Erlang, but you should be able to do them with the > >> "raw" form of Option for inet:setopts/2. > >> > >> --Per > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gleber.p@REDACTED Tue Jun 17 16:28:49 2008 From: gleber.p@REDACTED (Gleb Peregud) Date: Tue, 17 Jun 2008 16:28:49 +0200 Subject: [erlang-questions] lists:unzip breaks dialyzer. In-Reply-To: <8d1798e90806170711j52a200a8n2b4f5bab77d524fb@mail.gmail.com> References: <8d1798e90806170524l34fa0adata7da4d66867a2e4a@mail.gmail.com> <4857B6FD.5090301@it.uu.se> <8d1798e90806170710h2b7f0b7y2f0751c4217ffcfc@mail.gmail.com> <8d1798e90806170711j52a200a8n2b4f5bab77d524fb@mail.gmail.com> Message-ID: <14f0e3620806170728n4531487cpe4cc5f0d0c2541fe@mail.gmail.com> On Tue, Jun 17, 2008 at 4:11 PM, Adam Kelly wrote: > On Tue, Jun 17, 2008 at 2:07 PM, Tobias Lindahl wrote: >> However, you are running R11B-5, and I have fixed the bug since then. I >> suggest you upgrade if you can. > > Good to know, thank you (and Kostis). Yes, it appears we're suffering > from Ubuntu packages being nearly a year old even in the latest release. I've > worked around the problem for now, and we'll have to plan an upgrade. > > Adam. There's an updated version in intrepid repositories: http://packages.ubuntu.com/intrepid/erlang (unfortunatly it is still R12B-2) and if they become older then Debian you may rebuilt them for Ubuntu using the following instructions http://gleber.pl/blog/2008/02/12/16/building-erlang-12b-on-ubuntu-gutsy/ P.S. I just couldn't stand posting link to my own blog ^^ -- Gleb Peregud http://gleber.pl/ Every minute is to be grasped. Time waits for nobody. -- Inscription on a Zen Gong From martin.dvorak@REDACTED Tue Jun 17 16:46:32 2008 From: martin.dvorak@REDACTED (=?UTF-8?B?TWFydGluIER2b8WZw6Fr?=) Date: Tue, 17 Jun 2008 16:46:32 +0200 Subject: [erlang-questions] efile_drv.c bug? In-Reply-To: <48579DCD.1070008@lionet.info> References: <4856C8A7.5090603@san.rr.com> <20080617101652.GA9086@erix.ericsson.se> <48579DCD.1070008@lionet.info> Message-ID: <4857CE48.30903@e-fractal.cz> Hello, I think any modern OS wouldn't dare use 32 bit integers to represent file offsets, sign or no sign. For example ext3 is capable of holding a 2TB file so I'd expect the OS to be able to address any byte of it. I store ~500GB disk images on ext3 without problems (on a 32 bit architecture). However, FAT32 has a 4GB file size limit which may be the problem here. The solution may be NTFS which has 2^64 bytes limit. /Martin Lev Walkin wrote: > Jachym Holecek wrote: >> On Tue, 17 Jun 2008 12:16:52 +0200, Raimo Niskanen >> wrote: >>> On Mon, Jun 16, 2008 at 01:10:15PM -0700, Darren New wrote: >>>> Any idea why Windows (XP/SP2/32bit) can't handle files >4G but Linux can >>>> with the same release? Is this something expected to get fixed >>>> eventually? >>> If I recall correctly, file pointers (off_t) are signed in Windows >>> but there are extensions in Unix that use unsigned off_t but >>> ((off_t) -1) as an error return value, so it is possible on >>> many 32 bit Unixes to have almost 4GByte files, but 32 bit >>> Windows is stuck with its 2GByte limit. >>> >>> Someone correct me!? >> Upon quick inspection off_t seems to be int64_t on NetBSD and >> 'long int' on Linux. So it's signed on at least two Unices. > > > Single Unix Specification version 3, IEEE Std 1003.1 > specifies > > blkcnt_t and off_t shall be signed integer types. > > http://www.opengroup.org/onlinepubs/000095399/basedefs/sys/types.h.html From dnew@REDACTED Tue Jun 17 17:43:26 2008 From: dnew@REDACTED (Darren New) Date: Tue, 17 Jun 2008 08:43:26 -0700 Subject: [erlang-questions] efile_drv.c bug? In-Reply-To: <20080617101652.GA9086@erix.ericsson.se> References: <4856C8A7.5090603@san.rr.com> <20080617101652.GA9086@erix.ericsson.se> Message-ID: <4857DB9E.8000409@san.rr.com> Raimo Niskanen wrote: > I suppose you noticed that macro is not used, so there is even > a question if it was a real bug or not... I did not, actually. I was basically looking for why only 4G files were supported. Then I thought to run it under Linux, and Linux supports >4G files, so I figured it wasn't something fundamental about the file driver protocol or something. >> Any idea why Windows (XP/SP2/32bit) can't handle files >4G but Linux can >> with the same release? Is this something expected to get fixed eventually? > > If I recall correctly, file pointers (off_t) are signed in Windows > but there are extensions in Unix that use unsigned off_t but > ((off_t) -1) as an error return value, so it is possible on > many 32 bit Unixes to have almost 4GByte files, but 32 bit > Windows is stuck with its 2GByte limit. It turns out Linux will happily seek past 5G, so I don't think that's it. In private mail (likely mistakenly sent to me instead of the list ;-) I'm told it's because Windows in its structures represent file sizes as two 32-bit integers, and the code just doesn't compensate properly. Martin Dvo??k wrote: > However, FAT32 has a 4GB file size limit which may be the problem here. > The solution may be NTFS which has 2^64 bytes limit. No, I'm using a program in a different language to build a 6G file on an NTFS partition, then trying to read and append and such. Good suggestion, tho. -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From devdoer2@REDACTED Tue Jun 17 18:33:24 2008 From: devdoer2@REDACTED (devdoer bird) Date: Wed, 18 Jun 2008 00:33:24 +0800 Subject: [erlang-questions] Can one mnesia table fragment be further fragmented? Message-ID: Hi: I spilit one mnesia table into 3 fragments which distributed on 3 machines named A,B and C,but now the the machine "A" is very busy, so I decide split the fragment on the machine "A" into small fragments.Does mnesia support this? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From koops.j@REDACTED Tue Jun 17 19:34:56 2008 From: koops.j@REDACTED (Jeroen Koops) Date: Tue, 17 Jun 2008 19:34:56 +0200 Subject: [erlang-questions] Forwarding call from one gen_server to another Message-ID: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> Hi all, I was wondering what the common way is to call one gen_server, have this call forwarded to another gen_server, and have this other server reply to the client. One way of doing this is by having the first gen_server send a cast to the second gen_server, including the PID of the client, and then respond with a noreply. The second gen_server handles the cast by explicitly responding to the client using gen_server:reply() - using the PID sent by the original gen_server, of course. This seems to work, but it feels like a clumsy way to do it - isn't there a better way? At the end of the 'OTP Introduction' chapter, Armstrong's book 'Programming Erlang' hints that this is documented in the OTP Design Principles, but I cannot really find it there - am I overlooking something or is this covered elsewhere? Thanks in advance, Jeroen Koops -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim.mccoy@REDACTED Tue Jun 17 20:08:02 2008 From: jim.mccoy@REDACTED (Jim McCoy) Date: Tue, 17 Jun 2008 11:08:02 -0700 Subject: [erlang-questions] efile_drv.c bug? In-Reply-To: <4857DB9E.8000409@san.rr.com> References: <4856C8A7.5090603@san.rr.com> <20080617101652.GA9086@erix.ericsson.se> <4857DB9E.8000409@san.rr.com> Message-ID: > In private mail (likely mistakenly sent to me instead of the list ;-) > I'm told it's because Windows in its structures represent file sizes as > two 32-bit integers, and the code just doesn't compensate properly. Grrr... that nasty default reply-to... What I send to Darren was the following: On Mon, Jun 16, 2008 at 1:10 PM, Darren New wrote: > Any idea why Windows (XP/SP2/32bit) can't handle files >4G but Linux can > with the same release? Is this something expected to get fixed eventually? The win32 api has a funky way of dealing with file sizes for large files (retrofitting a 32bit api for files that have a size in excess of 2^32 I guess :) and Erlang's windows file driver does not support this. The problem code is in erts/emulator/drivers/win32/win_efile.c in the efile_openfile function. It calls GetFileInformationByHandle to get the file info, but it only returns the nFileSizeLow data. What it needs to do is to also grab the nFileSizeHigh data, shift it 32 bits left into a long and then add (nFileSizeLow & 0xFFFFFFFF) to that result. jim From pfisher@REDACTED Tue Jun 17 20:33:40 2008 From: pfisher@REDACTED (Paul Fisher) Date: Tue, 17 Jun 2008 13:33:40 -0500 Subject: [erlang-questions] Forwarding call from one gen_server to another In-Reply-To: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> References: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> Message-ID: <1213727620.6585.20.camel@localhost> On Tue, 2008-06-17 at 19:34 +0200, Jeroen Koops wrote: > I was wondering what the common way is to call one gen_server, have > this call forwarded to another gen_server, and have this other server > reply to the client. One way of doing this is by having the first > gen_server send a cast to the second gen_server, including the PID of > the client, and then respond with a noreply. The second gen_server > handles the cast by explicitly responding to the client using > gen_server:reply() - using the PID sent by the original gen_server, of > course. This seems to work, but it feels like a clumsy way to do it - > isn't there a better way? That's the way to do it (although you'll need to pass along the whole >From parameter, which is really a 2-arity tuple with the calling processes pid and a make_ref/0 specific to that call.) -- paul From morph@REDACTED Tue Jun 17 21:12:05 2008 From: morph@REDACTED (John Chandler) Date: Tue, 17 Jun 2008 15:12:05 -0400 Subject: [erlang-questions] Favored Linux distributions? In-Reply-To: <1213727620.6585.20.camel@localhost> References: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> <1213727620.6585.20.camel@localhost> Message-ID: Is there a consensus as to what distribution of Linux is most suitable to Erlang development, or does it even matter much? I am about to specify a virtual server that offers a choice of Ubuntu 7.10, Debian 4, CentOS 5, Fedora Core 7 & 8, Mandriva 2000, and OpenSUSE 10.3. -jmc From vychodil.hynek@REDACTED Tue Jun 17 21:30:33 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Tue, 17 Jun 2008 21:30:33 +0200 Subject: [erlang-questions] Favored Linux distributions? In-Reply-To: References: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> <1213727620.6585.20.camel@localhost> Message-ID: <4d08db370806171230r523c2c1dm72ecb2eca625e0b6@mail.gmail.com> It is Debian of course, what else you mean? ;-) On Tue, Jun 17, 2008 at 9:12 PM, John Chandler wrote: > Is there a consensus as to what distribution of Linux is most > suitable to > Erlang development, or does it even matter much? > > I am about to specify a virtual server that offers a choice of Ubuntu > 7.10, Debian 4, CentOS 5, Fedora Core 7 & 8, Mandriva 2000, and > OpenSUSE 10.3. > > -jmc > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From gleber.p@REDACTED Tue Jun 17 21:44:51 2008 From: gleber.p@REDACTED (Gleb Peregud) Date: Tue, 17 Jun 2008 21:44:51 +0200 Subject: [erlang-questions] Favored Linux distributions? In-Reply-To: <4d08db370806171230r523c2c1dm72ecb2eca625e0b6@mail.gmail.com> References: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> <1213727620.6585.20.camel@localhost> <4d08db370806171230r523c2c1dm72ecb2eca625e0b6@mail.gmail.com> Message-ID: <14f0e3620806171244w619cbf44k8592d34afa81adb5@mail.gmail.com> 2008/6/17 Hynek Vychodil : > It is Debian of course, what else you mean? ;-) > > On Tue, Jun 17, 2008 at 9:12 PM, John Chandler wrote: >> >> Is there a consensus as to what distribution of Linux is most >> suitable to >> Erlang development, or does it even matter much? >> >> I am about to specify a virtual server that offers a choice of Ubuntu >> 7.10, Debian 4, CentOS 5, Fedora Core 7 & 8, Mandriva 2000, and >> OpenSUSE 10.3. >> >> -jmc >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions > > > > -- > --Hynek (Pichi) Vychodil > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > I feel very comfortable with Ubuntu ;) -- Gleb Peregud http://gleber.pl/ Every minute is to be grasped. Time waits for nobody. -- Inscription on a Zen Gong From tuncer.ayaz@REDACTED Tue Jun 17 22:29:15 2008 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Tue, 17 Jun 2008 22:29:15 +0200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: <4ac8254d0806170313q1444f609p7a7f620776c937ca@mail.gmail.com> References: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> <485681E4.5080402@gmail.com> <48577F20.9030707@gmail.com> <4ac8254d0806170313q1444f609p7a7f620776c937ca@mail.gmail.com> Message-ID: <4ac8254d0806171329n1cc57811pb4ff5ef0fa221c42@mail.gmail.com> On Tue, Jun 17, 2008 at 12:13 PM, Tuncer Ayaz wrote: > On Tue, Jun 17, 2008 at 11:08 AM, Mats Cronqvist > wrote: >> Bjorn Gustavsson wrote: >>> Mats Cronqvist writes: >>> >>> >>>> i think not allowing backreferences is the correct choice for erlang. >>>> after all, erlang is about reliability. and it seems not being >>>> POSIX-compliant has worked for 20 years. >>>> >>> >>> Note that Patrik has implemented the re BIFs to be rescheduable, >>> meaning that even a badly written regular expression will not prevent >>> the emulator from running other Erlang processes (even in the non-SMP >>> emulator). >>> >> >> that is of course an example of the sound thinking that makes the >> emulator the excellent product it is. >> even so, if i was responsible for, say, a telecom server product, i >> would be worried about a feature (backreferences) where it's so easy to >> slip up and peg the CPU at 100%. i would imagine that my application >> would run into problems if someone wrote a regexp that used up a CPU-second. >> >>> As noted in the EEP, some regexp libraries would have been hard or >>> impossible to make rescheduable. >>> >> >> a crucial point. fwiw, i think using PCRE is a sound decision. the >> only reasonable alternative i'm aware of is the plan 9 regexp lib. did >> you look at that? >> "Libregexp9 is a port of Plan 9's Unicode-capable regular expression >> library. It is small and simple and provides the traditional extended >> regular expressions (without modern complications like {} and various \x >> character classes). It supports Unicode via wide character or UTF-8 >> encoding." >> >> http://swtch.com/plan9port/unix > > I hope you guys don't mind me mentioning that if we discuss > regex implementations we should at least make sure to use > one with an implementation that uses NFAs. That should not be too hard. > > http://swtch.com/~rsc/regexp/regexp1.html (Thompson NFA info) > http://en.wikipedia.org/wiki/Comparison_of_regular_expression_engines Well, I do object myself for mentioning already mentioned information :). Sorry, should have taken care to read the complete thread more carefully. Mea culpa, no excuses allowed. From nick@REDACTED Tue Jun 17 22:39:51 2008 From: nick@REDACTED (Nick Gerakines) Date: Tue, 17 Jun 2008 13:39:51 -0700 Subject: [erlang-questions] Favored Linux distributions? In-Reply-To: References: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> <1213727620.6585.20.camel@localhost> Message-ID: At Yahoo we use rhel 4.x and 5.x -- All of the erlang/otp production builds and libraries are made for it. Personally, I use debian. # Nick Gerakines On Tue, Jun 17, 2008 at 12:12 PM, John Chandler wrote: > Is there a consensus as to what distribution of Linux is most > suitable to > Erlang development, or does it even matter much? > > I am about to specify a virtual server that offers a choice of Ubuntu > 7.10, Debian 4, CentOS 5, Fedora Core 7 & 8, Mandriva 2000, and > OpenSUSE 10.3. > > -jmc > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From vances@REDACTED Tue Jun 17 23:04:59 2008 From: vances@REDACTED (Vance Shipley) Date: Tue, 17 Jun 2008 17:04:59 -0400 Subject: [erlang-questions] Favored Linux distributions? In-Reply-To: References: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> <1213727620.6585.20.camel@localhost> Message-ID: <20080617210456.GA14343@Little-Black-Book.local> On Tue, Jun 17, 2008 at 03:12:05PM -0400, John Chandler wrote: } Is there a consensus as to what distribution of Linux is most } suitable to Erlang development, or does it even matter much? Solaris is of course the most suitable Unix distribution for Erlang development. -Vance :) From rvirding@REDACTED Tue Jun 17 23:09:25 2008 From: rvirding@REDACTED (Robert Virding) Date: Tue, 17 Jun 2008 23:09:25 +0200 Subject: [erlang-questions] Dialyzer problems on Windows Message-ID: <3dbc6d1c0806171409td0018ffq1db20fadbb10f5ef@mail.gmail.com> I am running R12B-3 on Windows using the standard distribution package from erlang.org. When I try to run dialyzer from ERlang with dialyzer:gui() I get the following error: ** exception error: {dialyzer_error,"Please specify which plt to use"} in function dialyzer:gui/1 Yes, well, which plt file do I specify? Do I need to make one, and if so how? Remember I am running on a limited system. No I don't have cygwin, never needed it before. I did not get this error with R12B-2. Robert P.S. I see in R12B-2 there is a plt file on a plt directory. Can I just copy that or do I need another one? -------------- next part -------------- An HTML attachment was scrubbed... URL: From stonecypher@REDACTED Tue Jun 17 23:39:46 2008 From: stonecypher@REDACTED (John Haugeland) Date: Tue, 17 Jun 2008 15:39:46 -0600 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: <48577B13.8000809@gmail.com> References: <8f24f4b10806161212y1efc7956pa94ab0a423e057b1@mail.gmail.com> <48577B13.8000809@gmail.com> Message-ID: <8f24f4b10806171439h314ec8c3qb8cc7001d3cf8f0a@mail.gmail.com> On Tue, Jun 17, 2008 at 2:51 AM, Mats Cronqvist wrote: > John Haugeland wrote: > >> > after all, erlang is about reliability. and it seems not being >> > POSIX-compliant has worked for 20 years. >> >> By that logic, nothing in Erlang should ever change again. You might want >> to look up "argumentum ad antiquitatem". >> > By that logic, nothing in Erlang should ever change again just to become > POSIX compliant. You might want to look up "Straw man." > Sorry, no: I actually made no commentary about what erlang should or should not be doing. It's cute and all for you to repeat what someone else said, then google a fallacy list real quick-like and look for something that appears to apply, but a straw man is actually a misrepresentation of someone else's opinion, and since I made no representation of your opinion, it kind of cannot apply. If you don't understand the connection between "we shouldn't change it because it's been working for 20 years" and ad antiquitatem, well ... whatever, man. I don't think anyone has actually made the claim that things should change for the purpose of becoming POSIX compliant; I sure as hell haven't. What other people did was to say that POSIX was desirable because of linear time matching, or because of their beliefs about bloat. Nobody has ever said anything about POSIX compliance for the sake of POSIX compliance. Funny how people who claim straw man in others so often are in the process themselves. I'd recommend an introductory logic or philosophy class, so that you don't have to wipe so much egg off your face the next time you attempt to engage others in Tu Quoque. -------------- next part -------------- An HTML attachment was scrubbed... URL: From koops.j@REDACTED Tue Jun 17 23:51:53 2008 From: koops.j@REDACTED (Jeroen Koops) Date: Tue, 17 Jun 2008 23:51:53 +0200 Subject: [erlang-questions] Forwarding call from one gen_server to another In-Reply-To: <1213727620.6585.20.camel@localhost> References: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> <1213727620.6585.20.camel@localhost> Message-ID: <331a9abb0806171451q5183336q6b9b114a797709c5@mail.gmail.com> Hi Paul, Thanks for your reply. I was actually passing the From parameter back and forth without knowing what it was exactly - I just assumed it was a naked Pid. And I was worried that apparently, gen_server:call accepted a reply from a process other than the process it sent the request to, opening up the possibility for the client to interpret any message received as a response. But now that you mention that From is actually a tuple containing a reference making the call unique, it all makes sense. -- Jeroen Koops On Tue, Jun 17, 2008 at 8:33 PM, Paul Fisher wrote: > On Tue, 2008-06-17 at 19:34 +0200, Jeroen Koops wrote: > > I was wondering what the common way is to call one gen_server, have > > this call forwarded to another gen_server, and have this other server > > reply to the client. One way of doing this is by having the first > > gen_server send a cast to the second gen_server, including the PID of > > the client, and then respond with a noreply. The second gen_server > > handles the cast by explicitly responding to the client using > > gen_server:reply() - using the PID sent by the original gen_server, of > > course. This seems to work, but it feels like a clumsy way to do it - > > isn't there a better way? > > That's the way to do it (although you'll need to pass along the whole > >From parameter, which is really a 2-arity tuple with the calling > processes pid and a make_ref/0 specific to that call.) > > > -- > paul > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pedzsan@REDACTED Wed Jun 18 02:39:05 2008 From: pedzsan@REDACTED (Perry Smith) Date: Tue, 17 Jun 2008 19:39:05 -0500 Subject: [erlang-questions] AIX and R12B-3 In-Reply-To: <944da41d0806170406n182cab69i8cf57f19f4283088@mail.gmail.com> References: <944da41d0806170406n182cab69i8cf57f19f4283088@mail.gmail.com> Message-ID: Note: I did *not* cross post this to bugs or patches. I do not know the rules for this list and sometimes cross posting is really frowned upon. There are still three unsolved issues (two roughly the same issue). 1) If a java compiler is not found, the make still dives down the lib/ jinterface tree and promptly dies because it can not find a java compiler. So, what ./configure learns is not appropriately applied. 2) In the same sense, if the odbc driver is not found, the make still dives down the lib/odbc tree and soon dies. In the first case, I found a Java compiler. In the second case, I hand edited the Makefile and removed the call to odbc. (but that change is not in these diffs) 3) The make files assume that you can call install -d to create a directory. Not true for AIX. On AIX, I used: gmake MKSUBDIRS='mkdir -p' install I *believe* (but could be full of it) that mkdir -p is more universal than install -d. I used gcc 4.0.2, AIX's ld, and GNU's make (called gmake on my system). There is a "odd" issue which probably everyone knows about but I found it confusing. The distribution comes with precompiled files. "make clean" removes these files. That is not the usual case. "make clean" usually just removes files created post configure. Some other target in make is usually used to remove any precompiled files. Aside from that, attached is my diff of my changes. Oh, one explanation. As is somewhat typical of ./configure, things which are platform dependent, compiler dependent, and linker dependent get smerged. Some of the changes are to check if gcc is being used (I used it in my case) and if it is, then the linker flags are change from "-a -b -c" to -Wl,-a,-b,-c. This assumes that the linker flags are added to the gcc line. But, in some cases, they are not. The compiler is called and the ld is called directly. All this makes the configure scripts nearly impossible to get 100% correct. I keep waiting and hoping for some replacement to them. HTH, Perry On Jun 17, 2008, at 6:06 AM, Alex Arnon wrote: > Have you had any luck finding the difference (or resolving the > problem)? > If so, can you post a patch? > > > On Mon, Jun 16, 2008 at 7:46 PM, Perry Smith > wrote: > I'm new and I'm trying to compile the latest on AIX 5.3. > > I get errors: > > sys/common/erl_check_io.c:534: error: 'struct erl_drv_event_data' has > no member named 'reqevents' > sys/common/erl_check_io.c:535: error: 'struct erl_drv_event_data' has > no member named 'reqevents' > > (there are other similar errors). In sys/poll.h, for 32 bit > platforms, AIX defines events to reqevents and revents to rtnevents. > Without these defines, I get errors when the poll structure is used in > erl_poll.c. Why that is being used for AIX, I don't know since select > is the preferred interface on AIX. > > I'm assuming that someone has compiled the R12B-2 version for AIX and > something changed between 2 and 3. > > I'm poking around trying to figure out how to work around this. Any > suggestions would be welcomed. > > Thanks, > Perry Smith > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: erlang-aix.diff Type: application/octet-stream Size: 6671 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Jun 18 04:04:18 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 18 Jun 2008 14:04:18 +1200 Subject: [erlang-questions] power(N,P), cant get it to work In-Reply-To: <765646.51510.qm@web28308.mail.ukl.yahoo.com> References: <765646.51510.qm@web28308.mail.ukl.yahoo.com> Message-ID: <124090E7-C35F-4C49-85B9-2104E2C35695@cs.otago.ac.nz> This is another case of missing 'end's. power/2 would be more cleanly written as power(N, P) when is_integer(N), N >= 0 -> ipower(N, P, 1); power(N, P) when is_integer(N), N < 0 -> 1/ipower(N, P, 1). ipower(0, _, R) -> R; ipower(N, P, R) -> ipower(N-1, P, R*P). (a) use pattern matching and guards (b) common case first [power/2] base case first [ipower/3] (c) tail recursion (d) don't keep on checking for < 0 (e) help Dialyzer help you (the is_integer/1 tests). (f) public checks [power/2] private trusts [ipower/3]. It would be even better if written to take logarithmic time, but that's another story. While we're at it, fac(N) when is_integer(N), N >= 1 -> ifac(N, 1); fac(0) -> 1. ifac(1, R) -> R; ifac(N, R) -> ifac(N-1, R*N). The same ideas apply. From andreas.hillqvist@REDACTED Wed Jun 18 08:36:00 2008 From: andreas.hillqvist@REDACTED (Andreas Hillqvist) Date: Wed, 18 Jun 2008 08:36:00 +0200 Subject: [erlang-questions] power(N,P), cant get it to work In-Reply-To: <124090E7-C35F-4C49-85B9-2104E2C35695@cs.otago.ac.nz> References: <765646.51510.qm@web28308.mail.ukl.yahoo.com> <124090E7-C35F-4C49-85B9-2104E2C35695@cs.otago.ac.nz> Message-ID: <8268eea30806172336p2221fd9dyd792ef5234bd0439@mail.gmail.com> I can not see how the code will terminate for a negative N? power(N, P) when is_integer(N), N < 0 -> 1/ipower(-1 * N, P, 1). Or can "-" be used as an unary operation: power(N, P) when is_integer(N), N < 0 -> 1/ipower(-N, P, 1). Kind regards Andreas Hillqvist 2008/6/18, Richard A. O'Keefe : > This is another case of missing 'end's. > > power/2 would be more cleanly written as > > power(N, P) when is_integer(N), N >= 0 -> > ipower(N, P, 1); > power(N, P) when is_integer(N), N < 0 -> > 1/ipower(N, P, 1). > > ipower(0, _, R) -> R; > ipower(N, P, R) -> ipower(N-1, P, R*P). > > (a) use pattern matching and guards > (b) common case first [power/2] > base case first [ipower/3] > (c) tail recursion > (d) don't keep on checking for < 0 > (e) help Dialyzer help you (the is_integer/1 tests). > (f) public checks [power/2] private trusts [ipower/3]. > > It would be even better if written to take logarithmic time, > but that's another story. > > While we're at it, > > fac(N) when is_integer(N), N >= 1 -> > ifac(N, 1); > fac(0) -> 1. > > ifac(1, R) -> R; > ifac(N, R) -> ifac(N-1, R*N). > > The same ideas apply. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From raimo+erlang-questions@REDACTED Wed Jun 18 09:01:00 2008 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 18 Jun 2008 09:01:00 +0200 Subject: [erlang-questions] : efile_drv.c bug? In-Reply-To: References: <4856C8A7.5090603@san.rr.com> <20080617101652.GA9086@erix.ericsson.se> <4857DB9E.8000409@san.rr.com> Message-ID: <20080618070100.GA22112@erix.ericsson.se> On Tue, Jun 17, 2008 at 11:08:02AM -0700, Jim McCoy wrote: > > In private mail (likely mistakenly sent to me instead of the list ;-) > > I'm told it's because Windows in its structures represent file sizes as > > two 32-bit integers, and the code just doesn't compensate properly. > > Grrr... that nasty default reply-to... Most decent mail clients can be configured to use the list address in the To: field for replies to mailing lists. > > What I send to Darren was the following: > On Mon, Jun 16, 2008 at 1:10 PM, Darren New wrote: > > Any idea why Windows (XP/SP2/32bit) can't handle files >4G but Linux can > > with the same release? Is this something expected to get fixed eventually? > > The win32 api has a funky way of dealing with file sizes for large > files (retrofitting a 32bit api for files that have a size in excess > of 2^32 I guess :) and Erlang's windows file driver does not support > this. > > The problem code is in erts/emulator/drivers/win32/win_efile.c in the > efile_openfile function. It calls GetFileInformationByHandle to get > the file info, but it only returns the nFileSizeLow data. What it > needs to do is to also grab the nFileSizeHigh data, shift it 32 bits > left into a long and then add (nFileSizeLow & 0xFFFFFFFF) to that > result. Thanks a lot! I will have a look at it and investigate if there are backwards compatibility issues when using the nFileSizeHigh field or with other API functions win_efile.c uses. Apart from that it sounds like an easy enough fix. > > jim > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From raimo+erlang-questions@REDACTED Wed Jun 18 09:23:30 2008 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 18 Jun 2008 09:23:30 +0200 Subject: [erlang-questions] Favored Linux distributions? In-Reply-To: References: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> <1213727620.6585.20.camel@localhost> Message-ID: <20080618072330.GB22112@erix.ericsson.se> On Tue, Jun 17, 2008 at 03:12:05PM -0400, John Chandler wrote: > Is there a consensus as to what distribution of Linux is most > suitable to > Erlang development, or does it even matter much? > > I am about to specify a virtual server that offers a choice of Ubuntu > 7.10, Debian 4, CentOS 5, Fedora Core 7 & 8, Mandriva 2000, and > OpenSUSE 10.3. Currently, the major platforms we build and daily test are SuSE Linux Enterprise Server 10 SP1 and Solaris 8/9/10. But that is mostly because ClearCase runs on these, and does not necessarily make them the best platform to develop or run production on. We also test on Fedora, OpenSuSE, Ubuntu et.al. Quote from README: At Ericsson we have a "Daily Build and Test" that runs on: Operating system Versions ----------------------------------------------------------- Solaris/Sparc32 8, 9, 10 Linux/Suse x86 9.3, 10.1 Linux/Suse Amd64 9.3, 10.1 Linux/Montavista PPC 3.1 FreeBSD x86 5.0 Mac OS X/Intel 10.4.11 (Tiger) Mac OS X/Intel 10.5.3 (Leopard) Windows x86 XP SP2, 2003, Vista End quote. The platform you choose to develop on and the one you choose to run production on is probably up to your own demands, like our ClearCase demand. The fancy modern distros a'la Ubuntu, Fedora, OpenSuSE are probably nice to develop on because all the other tools they contain. Debian is probably more stable for production and Solaris 10 often outperform Linux for high load and also have a stability reputation. Even e.g OpenBSD might be a production choice for security if you do not need SMP performance. If you just need the latest Erlang/OTP I do not know which distros have got the most updated packages, but if you build from source you know what you have. > > -jmc > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From zerthurd@REDACTED Wed Jun 18 09:47:53 2008 From: zerthurd@REDACTED (Maxim Treskin) Date: Wed, 18 Jun 2008 14:47:53 +0700 Subject: [erlang-questions] Favored Linux distributions? In-Reply-To: <20080618072330.GB22112@erix.ericsson.se> References: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> <1213727620.6585.20.camel@localhost> <20080618072330.GB22112@erix.ericsson.se> Message-ID: I do it with Gentoo On 18/06/2008, Raimo Niskanen wrote: > On Tue, Jun 17, 2008 at 03:12:05PM -0400, John Chandler wrote: >> Is there a consensus as to what distribution of Linux is most >> suitable to >> Erlang development, or does it even matter much? >> >> I am about to specify a virtual server that offers a choice of Ubuntu >> 7.10, Debian 4, CentOS 5, Fedora Core 7 & 8, Mandriva 2000, and >> OpenSUSE 10.3. > > Currently, the major platforms we build and daily test are > SuSE Linux Enterprise Server 10 SP1 and Solaris 8/9/10. > But that is mostly because ClearCase runs on these, > and does not necessarily make them the best platform > to develop or run production on. > > We also test on Fedora, OpenSuSE, Ubuntu et.al. Quote from README: > > At Ericsson we have a "Daily Build and Test" that runs on: > > Operating system Versions > ----------------------------------------------------------- > Solaris/Sparc32 8, 9, 10 > Linux/Suse x86 9.3, 10.1 > Linux/Suse Amd64 9.3, 10.1 > Linux/Montavista PPC 3.1 > FreeBSD x86 5.0 > Mac OS X/Intel 10.4.11 (Tiger) > Mac OS X/Intel 10.5.3 (Leopard) > Windows x86 XP SP2, 2003, Vista > > End quote. > > The platform you choose to develop on and the one you > choose to run production on is probably up to your own > demands, like our ClearCase demand. The fancy modern > distros a'la Ubuntu, Fedora, OpenSuSE are probably > nice to develop on because all the other tools they > contain. Debian is probably more stable for production > and Solaris 10 often outperform Linux for high load > and also have a stability reputation. Even e.g OpenBSD > might be a production choice for security if you do > not need SMP performance. > > If you just need the latest Erlang/OTP I do not know > which distros have got the most updated packages, > but if you build from source you know what you have. > >> >> -jmc >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- Maxim Treskin From erlangy@REDACTED Wed Jun 18 09:57:52 2008 From: erlangy@REDACTED (ERLANG) Date: Wed, 18 Jun 2008 09:57:52 +0200 Subject: [erlang-questions] Favored Linux distributions? In-Reply-To: <20080618072330.GB22112@erix.ericsson.se> References: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> <1213727620.6585.20.camel@localhost> <20080618072330.GB22112@erix.ericsson.se> Message-ID: <66BB9C14-B7D7-4952-B2E1-CA684F140A4C@gmail.com> Hi Raimo, > contain. Debian is probably more stable for production > and Solaris 10 often outperform Linux for high load Could you please provide some details about how Solaris10 can outperfom Linux under high load? Any benchmarks? Regards Y. From tobias.lindahl@REDACTED Wed Jun 18 10:30:09 2008 From: tobias.lindahl@REDACTED (Tobias Lindahl) Date: Wed, 18 Jun 2008 10:30:09 +0200 Subject: [erlang-questions] Dialyzer problems on Windows In-Reply-To: <3dbc6d1c0806171409td0018ffq1db20fadbb10f5ef@mail.gmail.com> References: <3dbc6d1c0806171409td0018ffq1db20fadbb10f5ef@mail.gmail.com> Message-ID: <4858C791.3070100@it.uu.se> Hi Robert, I'm sorry that I didn't foresee this problem. I tend to forget about the Windows distribution. In the Dialyzer chapter of the reference manual the new plt handling is described (http://www.erlang.org/doc/apps/dialyzer). I haven't described how to do the same thing from the Erlang shell since I didn't see a need for this, but perhaps there is one on Windows. Use the dialyzer:run/1 function to perform the analysis to build the plt. The instructions are the same as for the command line version in the reference manual, but the options for analysis_type are --add_to_plt => plt_add --build_plt => plt_build --check_plt => plt_check --remove_from_plt => plt_remove The plt is no longer distributed by default, and the format has changed so you cannot use plt:s built with an older Dialyzer. To build a plt from the same files that used to be in the distributed version, the following command should work dialyzer:run([{files_rec, ["/lib/kernel/ebin", "/lib/stdlib/ebin", "/lib/mnesia/ebin"]}, {analysis_type, plt_build}, {output_plt, }]). Once the plt is built, you start the gui with dialyzer:gui([{init_plt, }]). You can also use the environment variable $DIALYZER_PLT to specify the location of the plt, both for builds and for later analyses. Although it is a bit more work to set up now, you get better control over what files to include in the plt. For example, you can add your own files to the plt and get them checked for consistency when you analyze. Also, you can build a plt for your whole project, and then just use --check_plt when something changes, and you will get warnings for the changed files (and all files that depend on them). (*) Hope this helps. Tobias (*) Note that the default is to not spit out the warnings when manipulating the plt. If you want to get the warnings you need to specify --get_warnings (or {get_warnings, true} in the shell version). Robert Virding wrote: > I am running R12B-3 on Windows using the standard distribution package from > erlang.org. > > When I try to run dialyzer from ERlang with dialyzer:gui() I get the > following error: > > ** exception error: {dialyzer_error,"Please specify which plt to use"} > in function dialyzer:gui/1 > > Yes, well, which plt file do I specify? Do I need to make one, and if so > how? Remember I am running on a limited system. No I don't have cygwin, > never needed it before. > > I did not get this error with R12B-2. > > Robert > > P.S. I see in R12B-2 there is a plt file on a plt directory. Can I just copy > that or do I need another one? > > > > ------------------------------------------------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From aicean@REDACTED Wed Jun 18 10:39:57 2008 From: aicean@REDACTED (aicean) Date: Wed, 18 Jun 2008 16:39:57 +0800 Subject: [erlang-questions] Is there size limit for socket buffer? Message-ID: <638555680806180139x13cad832m786fc55d7300b7dd@mail.gmail.com> I'm use socket receive data from other program.When data size is over 1492 bit,I can't get any the remaining data. Any one has a clue? Thanks! -- aicean -------------- next part -------------- An HTML attachment was scrubbed... URL: From jilani@REDACTED Wed Jun 18 11:38:35 2008 From: jilani@REDACTED (Jilani Khaldi) Date: Wed, 18 Jun 2008 11:38:35 +0200 Subject: [erlang-questions] Favored Linux distributions? In-Reply-To: <20080618072330.GB22112@erix.ericsson.se> References: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> <1213727620.6585.20.camel@localhost> <20080618072330.GB22112@erix.ericsson.se> Message-ID: <4858D79B.7070603@cheapnet.it> > Is there a consensus as to what distribution of Linux is most > suitable to Erlang development, or does it even matter much? I have been using (seldomly) Erlang for the last 7 years and compiling it from source on Slackaware Linux (from version 7 to 12) using 3 PC (almost all old) and I *never* had a problem running it and trying a lot of stuff. -- Jilani KHALDI --------------------- http://www.jkhaldi.eu From dot@REDACTED Mon Jun 16 19:09:54 2008 From: dot@REDACTED (Tony Finch) Date: Mon, 16 Jun 2008 18:09:54 +0100 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> References: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> Message-ID: On Mon, 16 Jun 2008, Robert Virding wrote: > One quick comment is that I wonder if not part of the problem with PCRE and > Perl regular expressions is that in one respect they are integrated into > Perl and contain features to interact with the rest of Perl. If I remember > correctly, for example, you can set Perl variables directly from the RE. Is > this what became the named subexprs in PCRE? Named subpatterns came to PCRE from Python. There isn't a syntax for setting arbitrary Perl variables from a subpattern (nor is there in Python). In perl-5.10, which is the first version that supports named subexpressions, the named captures become values in a hash table. In Python the MatchObject result has a group() method for accessing captures by number or name. Tony. -- f.anthony.n.finch http://dotat.at/ HUMBER THAMES: VARIABLE 3 OR 4 BECOMING SOUTHWEST 4 OR 5, OCCASIONALLY 6 LATER. SLIGHT OR MODERATE. SHOWERS. GOOD. From adam@REDACTED Wed Jun 18 12:34:44 2008 From: adam@REDACTED (Adam Lindberg) Date: Wed, 18 Jun 2008 12:34:44 +0200 Subject: [erlang-questions] Mnesia second index keys? Message-ID: <6344005f0806180334w2f25f94ci9117b257b162adbb@mail.gmail.com> Hi all! I'm working with some Mnesia tables which have several indexes created with mnesia:add_table_index(...) or as a start option. Is there any way to get the keys for those indexes? Something which would work as mnesia:index_all_keys(Tab, Pos) if it existed. I'd hate to go inside Mnesia's guts and work with the ets tables directly... The only way I can think of right now is to parse the whole table, however I expect it to be large and having second indexes was to speed up performance in those cases. Cheers! Adam -- Adam Lindberg http://www.erlang-consulting.com From raimo+erlang-questions@REDACTED Wed Jun 18 13:54:03 2008 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 18 Jun 2008 13:54:03 +0200 Subject: [erlang-questions] : Favored Linux distributions? In-Reply-To: <38B5AADC-B73A-402A-B3BB-1295C200EE57@gmail.com> References: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> <1213727620.6585.20.camel@localhost> <20080618072330.GB22112@erix.ericsson.se> <38B5AADC-B73A-402A-B3BB-1295C200EE57@gmail.com> Message-ID: <20080618115403.GA5740@erix.ericsson.se> On Wed, Jun 18, 2008 at 09:56:44AM +0200, ERLANG wrote: > Hi Raimo, > > > >contain. Debian is probably more stable for production > >and Solaris 10 often outperform Linux for high load > > Could you please provide some details about how > Solaris10 can outperfom Linux under high load? Just a rumor from a corridor neighbour that when running Erlang SMP and doing some load tests he beleived it scaled better for many scheduler threads. He thought the Solaris process scheduling might be more evolved. And this was on Sun hardware (Sun Ultra 24). Linux also has different shedulers to choose from, especially in some distros. > > Any benchmarks? Nope. > > Regards > Y. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From vychodil.hynek@REDACTED Wed Jun 18 14:10:25 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Wed, 18 Jun 2008 14:10:25 +0200 Subject: [erlang-questions] builtin regexp functions in Erlang/OTP R12B-3 In-Reply-To: <3dbc6d1c0806161231s32220744v88cf2dae83b54307@mail.gmail.com> References: <3dbc6d1c0806160715x50339988q3c8bbe744d3a4136@mail.gmail.com> <3dbc6d1c0806161231s32220744v88cf2dae83b54307@mail.gmail.com> Message-ID: <4d08db370806180510g3ab2dc13he2baf3214efdd298@mail.gmail.com> If you look in perldoc perlre, there is many more strange features in perl "regular" expressions. For example: "branch reset" (?|pattern) - fix backrefs numbers after this group "call to perl code" (?{ code }) - very good joke :-) "code generated pattern" (??{ code }) - dtto "parentheses back referencing" (?PARNO) - as previous, very good for making recursive patterns conditional patterns and more and more. I like it, but I argue don't implement it any more outside perl world :-) In result, perl regular expressions are very irregular. 2008/6/16 Robert Virding : > 2008/6/16 Tony Finch : > >> On Mon, 16 Jun 2008, Robert Virding wrote: >> >> > One quick comment is that I wonder if not part of the problem with PCRE >> and >> > Perl regular expressions is that in one respect they are integrated into >> > Perl and contain features to interact with the rest of Perl. If I >> remember >> > correctly, for example, you can set Perl variables directly from the RE. >> Is >> > this what became the named subexprs in PCRE? >> >> Named subpatterns came to PCRE from Python. There isn't a syntax for >> setting arbitrary Perl variables from a subpattern (nor is there in >> Python). In perl-5.10, which is the first version that supports named >> subexpressions, the named captures become values in a hash table. In >> Python the MatchObject result has a group() method for accessing captures >> by number or name. > > > Then I am at least partially right, not variables but keys in table. Still > you can access them by name anyway which means that the name exporting from > the regexp into the surrounding programs works. > > Robert > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From circularfunc@REDACTED Wed Jun 18 13:47:11 2008 From: circularfunc@REDACTED (Circular Function) Date: Wed, 18 Jun 2008 11:47:11 +0000 (GMT) Subject: [erlang-questions] guard sequences problem Message-ID: <696973.19438.qm@web28312.mail.ukl.yahoo.com> 36> 5<4,12<2,12>4. true 37> 5<4;12<2;12>4. * 1: syntax error before: 12 37> so apparently when separating with , it just evaluates the last expression, or at leats only returns the result of that. and if i understand the manual right guard sequences should be written like that but apparently not. first expression is short for 40> begin 4<2, 3>2 end. true ? ___________________________________________________ S?k efter k?rleken! Hitta din tvillingsj?l p? Yahoo! Dejting: http://ad.doubleclick.net/clk;185753627;24584539;x?http://se.meetic.yahoo.net/index.php?mtcmk=148783 -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Wed Jun 18 14:17:49 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Wed, 18 Jun 2008 14:17:49 +0200 Subject: [erlang-questions] : Favored Linux distributions? In-Reply-To: <20080618115403.GA5740@erix.ericsson.se> References: <331a9abb0806171034x2c13711cw9e5d37d3185c22b9@mail.gmail.com> <1213727620.6585.20.camel@localhost> <20080618072330.GB22112@erix.ericsson.se> <38B5AADC-B73A-402A-B3BB-1295C200EE57@gmail.com> <20080618115403.GA5740@erix.ericsson.se> Message-ID: <4d08db370806180517s3aa83e56ic1494825de31e585@mail.gmail.com> On Wed, Jun 18, 2008 at 1:54 PM, Raimo Niskanen < raimo+erlang-questions@REDACTED> wrote: > On Wed, Jun 18, 2008 at 09:56:44AM +0200, ERLANG wrote: > > Hi Raimo, > > > > > > >contain. Debian is probably more stable for production > > >and Solaris 10 often outperform Linux for high load > > > > Could you please provide some details about how > > Solaris10 can outperfom Linux under high load? > > Just a rumor from a corridor neighbour that when running > Erlang SMP and doing some load tests he beleived it > scaled better for many scheduler threads. He thought > the Solaris process scheduling might be more evolved. > And this was on Sun hardware (Sun Ultra 24). > > Linux also has different shedulers to choose from, > especially in some distros. Linux currently have only one process cheduler - CFS (Completely Fair Scheduler). Different schedulers for choice are only IO schedulers. > > > > > > Any benchmarks? > > Nope. > > > > > Regards > > Y. > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From circularfunc@REDACTED Wed Jun 18 14:08:52 2008 From: circularfunc@REDACTED (Circular Function) Date: Wed, 18 Jun 2008 12:08:52 +0000 (GMT) Subject: [erlang-questions] list comprehensions with predefined functions? Message-ID: <978949.66431.qm@web28314.mail.ukl.yahoo.com> how do i use a predefined function in a listcomprehension? and if i have 2 variables how do I write then? 42> c(mymath).?????????? ? {ok,mymath} 43> [mymath:power || X <- [1,2,3] Y <- 3]. * 1: syntax error before: Y 43> [mymath:power || X <- [1,2,3], Y <- 3]. * 1: illegal expression 44> [mymath:cube || X <- [1,2,3]].??????? ? * 1: illegal expression 45> [X*2 || X <- [1,2,3]].??????????????? ? [2,4,6] 46> [X*X || X <- [1,2,3]]. [1,4,9] 47> [X*2 || X <- [1,2,3]].?????? ? [2,4,6] 48> [mymath:sq || X <- [1,2,3]]. * 1: illegal expression 49> __________________________________________________________ G?r det l?ngsamt? Skaffa dig en snabbare bredbandsuppkoppling. S?k och j?mf?r hos Yahoo! Shopping. http://shopping.yahoo.se/c-100015813-bredband.html?partnerId=96914325 -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.hillqvist@REDACTED Wed Jun 18 14:48:05 2008 From: andreas.hillqvist@REDACTED (Andreas Hillqvist) Date: Wed, 18 Jun 2008 14:48:05 +0200 Subject: [erlang-questions] list comprehensions with predefined functions? In-Reply-To: <978949.66431.qm@web28314.mail.ukl.yahoo.com> References: <978949.66431.qm@web28314.mail.ukl.yahoo.com> Message-ID: <8268eea30806180548l67120211r27c81641905f07f3@mail.gmail.com> Try: > [mymath:power(X, Y) || X <- [1,2,3] Y <- 3]. > [mymath:cube(X) || X <- [1,2,3]]. > [mymath:sq(X) || X <- [1,2,3]]. Kind regards Andreas Hillqvist Den 2008-06-18 skrev Circular Function : > > how do i use a predefined function in a listcomprehension? > > and if i have 2 variables how do I write then? > > > 42> c(mymath). > {ok,mymath} > 43> [mymath:power || X <- [1,2,3] Y <- 3]. > * 1: syntax error before: Y > 43> [mymath:power || X <- [1,2,3], Y <- 3]. > * 1: illegal expression > 44> [mymath:cube || X <- [1,2,3]]. > * 1: illegal expression > 45> [X*2 || X <- [1,2,3]]. > [2,4,6] > 46> [X*X || X <- [1,2,3]]. > [1,4,9] > 47> [X*2 || X <- [1,2,3]]. > [2,4,6] > 48> [mymath:sq || X <- [1,2,3]]. > * 1: illegal expression > 49> > > ------------------------------ > G?r det l?ngsamt? Skaffa dig en snabbare bredbandsuppkoppling. > S?k och j?mf?r hos Yahoo! Shopping. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Wed Jun 18 14:55:37 2008 From: erlang@REDACTED (Dominic Williams) Date: Wed, 18 Jun 2008 08:55:37 -0400 (EDT) Subject: [erlang-questions] list comprehensions with predefined functions? In-Reply-To: <978949.66431.qm@web28314.mail.ukl.yahoo.com> References: <978949.66431.qm@web28314.mail.ukl.yahoo.com> Message-ID: <48a0d1c2534aa02f9a70bf9c786929a8.squirrel@www.geekisp.com> > how do i use a predefined function in a listcomprehension? > > and if i have 2 variables how do I write then? > > > 42> c(mymath). > {ok,mymath} > 43> [mymath:power || X <- [1,2,3] Y <- 3]. > * 1: syntax error before: Y [mymath: power (X, 3) || X <- [1,2,3]]. Regards, Dominic Williams http://dominicwilliams.net ---- From jilani@REDACTED Wed Jun 18 15:53:37 2008 From: jilani@REDACTED (Jilani Khaldi) Date: Wed, 18 Jun 2008 15:53:37 +0200 Subject: [erlang-questions] list comprehensions with predefined functions? In-Reply-To: <8268eea30806180548l67120211r27c81641905f07f3@mail.gmail.com> References: <978949.66431.qm@web28314.mail.ukl.yahoo.com> <8268eea30806180548l67120211r27c81641905f07f3@mail.gmail.com> Message-ID: <48591361.2060009@cheapnet.it> Andreas Hillqvist wrote: > Try: > > [mymath:power(X, Y) || X <- [1,2,3] Y <- 3]. Try: [mymath:power(X, Y) || X <- [1,2,3], Y <- [3]]. [mymath:power(X, Y) || X <- [1,2,3], Y <- [3, 4]]. % modify N in -N for N < 0 power(N, P) when is_integer(N), N >= 0 -> ipower(N, P, 1); power(N, P) when is_integer(N), N < 0 -> 1/ipower(-N, P, 1). ipower(0, _, R) -> R; ipower(N, P, R) -> ipower(N-1, P, R*P). -- Jilani KHALDI --------------------- http://www.jkhaldi.eu From circularfunc@REDACTED Wed Jun 18 18:44:32 2008 From: circularfunc@REDACTED (Circular Function) Date: Wed, 18 Jun 2008 16:44:32 +0000 (GMT) Subject: [erlang-questions] or? Message-ID: <997099.93066.qm@web28315.mail.ukl.yahoo.com> if i instead of : fib(N) -> ??????? if ??????????????? N > 1 -> ??????????????????????? fib(N-1) + fib(N-2); ??????????????? N == 1 -> ??????????????????????? 1; ??????????????? N == 0 -> ??????????????????????? 0 ??????? end. want to do: fib(N) -> ??????? if ??????????????? N > 1 -> ??????????????????????? fib(N-1) + fib(N-2); ??????????????? N == 1 or N == 0 -> ??????????????????????? N ??????? end. it doenst work, how do i use or? http://www.erlang.org/doc/reference_manual/part_frame.html doesnt provide an actual example of how to use it. __________________________________________________________ Ta semester! - s?k efter resor hos Yahoo! Shopping. J?mf?r pris p? flygbiljetter och hotellrum h?r: http://shopping.yahoo.se/c-169901-resor-biljetter.html?partnerId=96914052 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.polvorin@REDACTED Wed Jun 18 20:01:57 2008 From: pablo.polvorin@REDACTED (Pablo Polvorin) Date: Wed, 18 Jun 2008 15:01:57 -0300 Subject: [erlang-questions] or? In-Reply-To: <997099.93066.qm@web28315.mail.ukl.yahoo.com> References: <997099.93066.qm@web28315.mail.ukl.yahoo.com> Message-ID: <1ffe809c0806181101u5e7de5a0s31e749571fb8db92@mail.gmail.com> fib(N) -> if N > 1 -> fib(N-1) + fib(N-2); N == 1; N == 0 -> N end. Or using brackets: fib2(N) -> if N > 1 -> fib2(N-1) + fib2(N-2); (N == 1) or (N == 0) -> N end. written in more idiomatic erlang: fib3(0) -> 0; fib3(1) -> 1; fib3(N) when N > 1 -> fib3(N-1) + fib3(N-2). 2008/6/18, Circular Function : > if i instead of : > > fib(N) -> > if > N > 1 -> > fib(N-1) + fib(N-2); > N == 1 -> > 1; > N == 0 -> > 0 > end. > > want to do: > > fib(N) -> > if > N > 1 -> > fib(N-1) + fib(N-2); > N == 1 or N == 0 -> > N > end. > > it doenst work, how do i use or? > http://www.erlang.org/doc/reference_manual/part_frame.html > doesnt provide an actual example of how to use it. > > > > __________________________________________________________ > Ta semester! - s?k efter resor hos Yahoo! Shopping. > J?mf?r pris p? flygbiljetter och hotellrum h?r: > http://shopping.yahoo.se/c-169901-resor-biljetter.html?partnerId=96914052 -- -- pablo http://ppolv.wordpress.com ---- From bioprogrammer@REDACTED Wed Jun 18 20:36:32 2008 From: bioprogrammer@REDACTED (Caitlin Gibbons) Date: Wed, 18 Jun 2008 11:36:32 -0700 Subject: [erlang-questions] Help with tuple pattern match? Message-ID: Hi all. I'm looking at: Person={person,{name,{first,joe},{last,armstrong}},{footsize,42}}. and trying to extract the value of 'last', i.e., 'armstrong'. {_,{_,{_,{_,Last},_}=Person. was unsuccessful. Any help? Thanks, Caitlin -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmercer@REDACTED Wed Jun 18 20:42:36 2008 From: dmercer@REDACTED (David Mercer) Date: Wed, 18 Jun 2008 13:42:36 -0500 Subject: [erlang-questions] Help with tuple pattern match? In-Reply-To: References: Message-ID: <021b01c8d173$14b612a0$f21ea8c0@SSI.CORP> 1> Person={person,{name,{first,joe},{last,armstrong}},{footsize,42}}. {person,{name,{first,joe},{last,armstrong}},{footsize,42}} 2> {_,{_,_,{_,Last}},_} = Person. {person,{name,{first,joe},{last,armstrong}},{footsize,42}} 3> Last. armstrong _____ From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Caitlin Gibbons Sent: Wednesday, June 18, 2008 13:37 To: erlang-questions@REDACTED Subject: [erlang-questions] Help with tuple pattern match? Hi all. I'm looking at: Person={person,{name,{first,joe},{last,armstrong}},{footsize,42}}. and trying to extract the value of 'last', i.e., 'armstrong'. {_,{_,{_,{_,Last},_}=Person. was unsuccessful. Any help? Thanks, Caitlin -------------- next part -------------- An HTML attachment was scrubbed... URL: From circularfunc@REDACTED Wed Jun 18 20:19:17 2008 From: circularfunc@REDACTED (Circular Function) Date: Wed, 18 Jun 2008 18:19:17 +0000 (GMT) Subject: [erlang-questions] function with no parameter? Message-ID: <195708.60961.qm@web28313.mail.ukl.yahoo.com> -export([hej/0,fib/1,fibx/1,fac/1, fac2/1, sq/1,cube/1,power/2, powerc/2,nCk/2]). hej() -> 12. this compiles but it doesnt work when calling in the shell. 36> hej(). ** exception error: undefined function shell_default:hej/0 does this have anything to do with functions should be pure in erlang thus calling a fucntion with no parameter isnt allowed? but then why is creating them allowed? ___________________________________________________ S?k efter k?rleken! Hitta din tvillingsj?l p? Yahoo! Dejting: http://ad.doubleclick.net/clk;185753627;24584539;x?http://se.meetic.yahoo.net/index.php?mtcmk=148783 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodahummont@REDACTED Wed Jun 18 19:50:10 2008 From: rodahummont@REDACTED (Rodrigo Ahumada) Date: Wed, 18 Jun 2008 13:50:10 -0400 Subject: [erlang-questions] or? In-Reply-To: <997099.93066.qm@web28315.mail.ukl.yahoo.com> References: <997099.93066.qm@web28315.mail.ukl.yahoo.com> Message-ID: <1213811410.22655.4.camel@joyita> El mi??, 18-06-2008 a las 16:44 +0000, Circular Function escribi??: > if i instead of : > > fib(N) -> > if > N > 1 -> > fib(N-1) + fib(N-2); > N == 1 -> > 1; > N == 0 -> > 0 > end. > > want to do: > > fib(N) -> > if > N > 1 -> > fib(N-1) + fib(N-2); > N == 1 or N == 0 -> > N > end. > > it doenst work, how do i use or? > http://www.erlang.org/doc/reference_manual/part_frame.html > doesnt provide an actual example of how to use it. > > in the "if" you have to put Guard sequences, not boolean expressions something like this: N == 1; N == 0 -> N the ; is the prolog OR. it's a case of atavism i think. > > ______________________________________________________________________ > L??na pengar utan s??kerhet. > S??k och j??mf??r hos Yahoo! Shopping. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions __________________________________________________ Correo Yahoo! Espacio para todos tus mensajes, antivirus y antispam ?gratis! ?Abr? tu cuenta ya! - http://correo.yahoo.com.ar From igorrs@REDACTED Wed Jun 18 20:49:52 2008 From: igorrs@REDACTED (Igor Ribeiro Sucupira) Date: Wed, 18 Jun 2008 15:49:52 -0300 Subject: [erlang-questions] guard sequences problem In-Reply-To: <696973.19438.qm@web28312.mail.ukl.yahoo.com> References: <696973.19438.qm@web28312.mail.ukl.yahoo.com> Message-ID: 2008/6/18 Circular Function : > 36> 5<4,12<2,12>4. > true Both were evaluated and the last result was shown. I don't this is a guard, as it's not being used within an "if", a "when" or the like... Igor. > 37> 5<4;12<2;12>4. > * 1: syntax error before: 12 > 37> > > so apparently when separating with , it just evaluates the last expression, > or at leats only returns the result of that. > and if i understand the manual right guard sequences should be written like > that but apparently not. > first expression is short for > 40> begin 4<2, 3>2 end. > true > ? > > > > ________________________________ > Ta semester! - s?k efter resor hos Yahoo! Shopping. > J?mf?r pris p? flygbiljetter och hotellrum: > http://shopping.yahoo.se/c-169901-resor-biljetter.html > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From dmercer@REDACTED Wed Jun 18 20:57:24 2008 From: dmercer@REDACTED (David Mercer) Date: Wed, 18 Jun 2008 13:57:24 -0500 Subject: [erlang-questions] function with no parameter? In-Reply-To: <195708.60961.qm@web28313.mail.ukl.yahoo.com> References: <195708.60961.qm@web28313.mail.ukl.yahoo.com> Message-ID: <022c01c8d175$25b38540$f21ea8c0@SSI.CORP> You should prefix the function call with the module. E.g., ?Module:hej().? _____ From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Circular Function Sent: Wednesday, June 18, 2008 13:19 To: erlang-questions@REDACTED Subject: [erlang-questions] function with no parameter? -export([hej/0,fib/1,fibx/1,fac/1, fac2/1, sq/1,cube/1,power/2, powerc/2,nCk/2]). hej() -> 12. this compiles but it doesnt work when calling in the shell. 36> hej(). ** exception error: undefined function shell_default:hej/0 does this have anything to do with functions should be pure in erlang thus calling a fucntion with no parameter isnt allowed? but then why is creating them allowed? _____ G?r det l?ngsamt? Skaffa dig en snabbare bredbandsuppkoppling. S?k och j?mf?r hos Yahoo! Shopping. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rapsey@REDACTED Wed Jun 18 20:57:53 2008 From: rapsey@REDACTED (Rapsey) Date: Wed, 18 Jun 2008 20:57:53 +0200 Subject: [erlang-questions] Help with tuple pattern match? In-Reply-To: References: Message-ID: <97619b170806181157y4a8ef7b7q240e5f57dd7fb61d@mail.gmail.com> {_,{_,_,{_,Last}},_} = Person. Sergej 2008/6/18 Caitlin Gibbons : > Hi all. > > I'm looking at: > > Person={person,{name,{first,joe},{last,armstrong}},{footsize,42}}. > > and trying to extract the value of 'last', i.e., 'armstrong'. > > {_,{_,{_,{_,Last},_}=Person. > > was unsuccessful. Any help? > > Thanks, > > Caitlin > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bioprogrammer@REDACTED Wed Jun 18 21:04:59 2008 From: bioprogrammer@REDACTED (Caitlin Gibbons) Date: Wed, 18 Jun 2008 12:04:59 -0700 Subject: [erlang-questions] Help with tuple pattern match? In-Reply-To: <021b01c8d173$14b612a0$f21ea8c0@SSI.CORP> References: <021b01c8d173$14b612a0$f21ea8c0@SSI.CORP> Message-ID: Hi David. Thanks for the prompt reply. Could you explain the details of your answer? I interpreted the structure of this form of pattern matching as: Person={person,{name,{first,joe},{last,armstrong}},{footsize,42}}. {person,_,{name,_{first,_{last,_}},{footsize,_}=Person. Is this correct? From left to right, one counts the atoms, i.e., person,name,first,last,etc. and 'aligns' them with the underscores? Thanks again, Caitlin On 6/18/08, David Mercer wrote: > > 1> Person={person,{name,{first,joe},{last,armstrong}},{footsize,42}}. > > {person,{name,{first,joe},{last,armstrong}},{footsize,42}} > > 2> {_,{_,_,{_,Last}},_} = Person. > > {person,{name,{first,joe},{last,armstrong}},{footsize,42}} > > 3> Last. > > armstrong > > > ------------------------------ > > *From:* erlang-questions-bounces@REDACTED [mailto: > erlang-questions-bounces@REDACTED] *On Behalf Of *Caitlin Gibbons > *Sent:* Wednesday, June 18, 2008 13:37 > *To:* erlang-questions@REDACTED > *Subject:* [erlang-questions] Help with tuple pattern match? > > > > Hi all. > > I'm looking at: > > Person={person,{name,{first,joe},{last,armstrong}},{footsize,42}}. > > and trying to extract the value of 'last', i.e., 'armstrong'. > > {_,{_,{_,{_,Last},_}=Person. > > was unsuccessful. Any help? > > Thanks, > > Caitlin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rapsey@REDACTED Wed Jun 18 21:06:16 2008 From: rapsey@REDACTED (Rapsey) Date: Wed, 18 Jun 2008 21:06:16 +0200 Subject: [erlang-questions] function with no parameter? In-Reply-To: <195708.60961.qm@web28313.mail.ukl.yahoo.com> References: <195708.60961.qm@web28313.mail.ukl.yahoo.com> Message-ID: <97619b170806181206l7add3e6aj6d3521bf42bca615@mail.gmail.com> Are you sure you didn't forget to call the function with the module name? module:hej(). 2008/6/18 Circular Function : > -export([hej/0,fib/1,fibx/1,fac/1, fac2/1, sq/1,cube/1,power/2, > powerc/2,nCk/2]). > > hej() -> 12. > > this compiles but it doesnt work when calling in the shell. > > 36> hej(). > ** exception error: undefined function shell_default:hej/0 > > does this have anything to do with functions should be pure in erlang thus > calling a fucntion with no parameter isnt allowed? but then why is creating > them allowed? > > ------------------------------ > G?r det l?ngsamt? Skaffa dig en snabbare bredbandsuppkoppling. > S?k och j?mf?r hos Yahoo! Shopping. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dougedmunds@REDACTED Wed Jun 18 21:09:52 2008 From: dougedmunds@REDACTED (Doug Edmunds) Date: Wed, 18 Jun 2008 12:09:52 -0700 Subject: [erlang-questions] function with no parameter? In-Reply-To: <195708.60961.qm@web28313.mail.ukl.yahoo.com> References: <195708.60961.qm@web28313.mail.ukl.yahoo.com> Message-ID: I doubt any of them work unless you prefix the function with the name of the module (unless you are shadowing a BIF, in which case you are really calling the BIF and not yours. the tipoff is 'undefined function' try this: whatever_the_module_name_is:hej(). which will work, as long as the module is in the search path. -dae 2008/6/18 Circular Function : > -export([hej/0,fib/1,fibx/1,fac/1, fac2/1, sq/1,cube/1,power/2, > powerc/2,nCk/2]). > > hej() -> 12. > > this compiles but it doesnt work when calling in the shell. > > 36> hej(). > ** exception error: undefined function shell_default:hej/0 > > does this have anything to do with functions should be pure in erlang thus > calling a fucntion with no parameter isnt allowed? but then why is creating > them allowed? > > ------------------------------ > G?r det l?ngsamt? Skaffa dig en snabbare bredbandsuppkoppling. > S?k och j?mf?r hos Yahoo! Shopping. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hasan.veldstra@REDACTED Wed Jun 18 21:19:36 2008 From: hasan.veldstra@REDACTED (Hasan Veldstra) Date: Wed, 18 Jun 2008 20:19:36 +0100 Subject: [erlang-questions] Help with tuple pattern match? In-Reply-To: References: <021b01c8d173$14b612a0$f21ea8c0@SSI.CORP> Message-ID: <48595FC8.3000104@gmail.com> > answer? I interpreted the structure of this form of pattern matching as: > > Person={person,{name,{first,joe},{last,armstrong}},{footsize,42}}. > > {person,_,{name,_{first,_{last,_}},{footsize,_}=Person. > > Is this correct? From left to right, one counts the atoms, i.e., > person,name,first,last,etc. and 'aligns' them with the underscores? Underscore matches terms (e.g. whole tuples), not just atoms. From dougedmunds@REDACTED Wed Jun 18 21:25:39 2008 From: dougedmunds@REDACTED (Doug Edmunds) Date: Wed, 18 Jun 2008 12:25:39 -0700 Subject: [erlang-questions] Help with tuple pattern match? In-Reply-To: References: <021b01c8d173$14b612a0$f21ea8c0@SSI.CORP> Message-ID: Also count the braces. You need an equal number of left {'s and right }'s, regardless of the underlying structure of tuple. Your {person,_,{name,_{first,_{last ,_}},{footsize,_}=Person. has 5 left but only 3 right. -dae -------------- next part -------------- An HTML attachment was scrubbed... URL: From circularfunc@REDACTED Wed Jun 18 22:30:34 2008 From: circularfunc@REDACTED (Circular Function) Date: Wed, 18 Jun 2008 20:30:34 +0000 (GMT) Subject: [erlang-questions] Idiomatic Erlang, style/performance question Message-ID: <310053.70921.qm@web28314.mail.ukl.yahoo.com> Is there a styleguide somewhere on Erlang? Im currently going through the Erlang reference manual and you can see some examples, are those generally referred to as idiomatic Erlang? Some examples of implementations of the power-function below. The last function someone on the mailing list provided as a correction to the second one. The first one uses pattern matching and guard sequences right? Is this a good way? I dont quite understand the advantage of the last function. However, the shell complains about 88> c(mymath). ../mymath.erl:24: Warning: variable 'N' is unused {ok,mymath} Which is the powerx(N, 0) -> 1; line. However i saw some code that used _ and i tried: powerx(_, 0) -> 1; so _ is the correct way to write this function? Hmm while writing I got a thought. Is the last function tail-recursive? I have only heard that term but never used it. Is tail-recursion faster or uses less memory, what is the advantage/disadvantage? powerx(N, 0) -> 1; powerx(N, P) when is_integer(P), P > 0 -> ??????? N * powerx(N, P - 1); powerx(N, P) when is_integer(P), P < 0 -> ??????? 1 / powerx(N, -1 * P). ? powerc(N, P) -> ??????? if ??????????????? P > 0 -> ??????????????????????? N * powerc(N, P - 1); ??????????????? P < 0 -> ??????????????????????? 1 / powerc(N, -1 * P); ??????????????? P == 0 -> ??????????????????????? 1 ??????? end. power(N, P) when is_integer(N), N >= 0 -> ??????? ipower(N, P, 1); ??? power(N, P) when is_integer(N), N < 0 -> ??????? 1/ipower(-N, P, 1). ??? ipower(0, _, R) -> R; ??? ipower(N, P, R) -> ipower(N-1, P, R*P). __________________________________________________________ Ta semester! - s?k efter resor hos Yahoo! Shopping. J?mf?r pris p? flygbiljetter och hotellrum h?r: http://shopping.yahoo.se/c-169901-resor-biljetter.html?partnerId=96914052 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Wed Jun 18 23:00:58 2008 From: rvirding@REDACTED (Robert Virding) Date: Wed, 18 Jun 2008 23:00:58 +0200 Subject: [erlang-questions] New patch of LFE - Lisp Flavoured Erlang Message-ID: <3dbc6d1c0806181400x48feedf8w72ab742555e5c5b6@mail.gmail.com> This patch contains two major changes: - list/binary comprehensions have been added (contains a test and bind variables option) - extra option for cond to test and bind variables (this is a big win) - updated documentation plus some internal changes and new example files (test files really). This should be the last minor version before the next full release which should have the new (and final) syntax. The system can be found in the "User Contributions" section at trapexit.org, http://forum.trapexit.org/viewtopic.php?p=43950#43950 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dbarker@REDACTED Wed Jun 18 23:19:19 2008 From: dbarker@REDACTED (Deryk Barker) Date: Wed, 18 Jun 2008 14:19:19 -0700 Subject: [erlang-questions] Idiomatic Erlang, style/performance question In-Reply-To: <310053.70921.qm@web28314.mail.ukl.yahoo.com> References: <310053.70921.qm@web28314.mail.ukl.yahoo.com> Message-ID: <48597BD7.6080708@camosun.bc.ca> Circular Function wrote: > Is there a styleguide somewhere on Erlang? Im currently going through > the Erlang reference manual and you can see some examples, are those > generally referred to as idiomatic Erlang? > > Some examples of implementations of the power-function below. > The last function someone on the mailing list provided as a correction > to the second one. > The first one uses pattern matching and guard sequences right? Is this > a good way? I dont quite understand the advantage of the last function. > However, the shell complains about > 88> c(mymath). > ./mymath.erl:24: Warning: variable 'N' is unused > {ok,mymath} > Which is the powerx(N, 0) -> 1; line. > However i saw some code that used _ and i tried: > powerx(_, 0) -> 1; > so _ is the correct way to write this function? > I claim no great expertise, but I believe that power(_N, 0) -> 1; is the preferred style today: the compiler gives no warning but, should the need arise, you can refer to the first parameter as _N. (The _Name style also allows for name which have documentary value) > > Hmm while writing I got a thought. Is the last function > tail-recursive? I have only heard that term but never used it. > Is tail-recursion faster or uses less memory, what is the > advantage/disadvantage? > > power(N, P) when is_integer(N), N >= 0 -> > ipower(N, P, 1); > power(N, P) when is_integer(N), N < 0 -> > 1/ipower(-N, P, 1). > > ipower(0, _, R) -> R; > ipower(N, P, R) -> ipower(N-1, P, R*P). > > Yup, this looks tail recursive, because the recursive call to ipower is the last thing in the function. TR functions can run in constant stack space; to write a server you must use tail recursion. -- |Deryk Barker, Computer Science Dept. | Music does not have to be understood| |Camosun College, Victoria, BC, Canada| It has to be listened to. | |email: dbarker@REDACTED | | |phone: +1 250 370 4452 | Hermann Scherchen. | From dmercer@REDACTED Wed Jun 18 23:20:53 2008 From: dmercer@REDACTED (David Mercer) Date: Wed, 18 Jun 2008 16:20:53 -0500 Subject: [erlang-questions] Help with tuple pattern match? In-Reply-To: References: <021b01c8d173$14b612a0$f21ea8c0@SSI.CORP> Message-ID: <026d01c8d189$31692c00$f21ea8c0@SSI.CORP> Let's take a look at the tuple: {person,{name,{first,joe},{last,armstrong}},{footsize,42}}. The last name is the second element of a 2-tuple, whose first element is the atom "last", so we would match that with: {last,Last} or since we don't care about the first element: {_,Last} That tuple, however, is the third element of a 3-tuple, so that is matched thus: {_,_,{_,Last}} Finally, that tuple is the second element of a 3-tuple, resulting in the final match expression: {_,{_,_,{_,Last}},_} DBM _____ From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Caitlin Gibbons Sent: Wednesday, June 18, 2008 14:05 To: erlang-questions@REDACTED Subject: Re: [erlang-questions] Help with tuple pattern match? Hi David. Thanks for the prompt reply. Could you explain the details of your answer? I interpreted the structure of this form of pattern matching as: Person={person,{name,{first,joe},{last,armstrong}},{footsize,42}}. {person,_,{name,_{first,_{last,_}},{footsize,_}=Person. Is this correct? From left to right, one counts the atoms, i.e., person,name,first,last,etc. and 'aligns' them with the underscores? Thanks again, Caitlin On 6/18/08, David Mercer wrote: 1> Person={person,{name,{first,joe},{last,armstrong}},{footsize,42}}. {person,{name,{first,joe},{last,armstrong}},{footsize,42}} 2> {_,{_,_,{_,Last}},_} = Person. {person,{name,{first,joe},{last,armstrong}},{footsize,42}} 3> Last. armstrong _____ From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Caitlin Gibbons Sent: Wednesday, June 18, 2008 13:37 To: erlang-questions@REDACTED Subject: [erlang-questions] Help with tuple pattern match? Hi all. I'm looking at: Person={person,{name,{first,joe},{last,armstrong}},{footsize,42}}. and trying to extract the value of 'last', i.e., 'armstrong'. {_,{_,{_,{_,Last},_}=Person. was unsuccessful. Any help? Thanks, Caitlin -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Wed Jun 18 23:59:04 2008 From: rvirding@REDACTED (Robert Virding) Date: Wed, 18 Jun 2008 23:59:04 +0200 Subject: [erlang-questions] Dialyzer problems on Windows In-Reply-To: <4858C791.3070100@it.uu.se> References: <3dbc6d1c0806171409td0018ffq1db20fadbb10f5ef@mail.gmail.com> <4858C791.3070100@it.uu.se> Message-ID: <3dbc6d1c0806181459u74b7512fja3f4dac41292120d@mail.gmail.com> Hi Tobias, I see the benefits of the plt handling but it would be very nice to have a default plt containing the standard stuff in a known place which is used unless you tell it not to. Users/apps could then have their own plts which augment the standard one. Or completely replace it for building special apps. Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt@REDACTED Thu Jun 19 00:07:25 2008 From: matt@REDACTED (Matt Harrison) Date: Wed, 18 Jun 2008 23:07:25 +0100 Subject: [erlang-questions] Reading large (1GB+) XML files. Message-ID: <1213826845.6813.1.camel@matt> I have to extract information from some extremely large XML files, 6GB+ as well. All signs point to xmerl_eventp but the documentation is non exactly fleshed out and as I've only been doing erlang for a few weeks I am really struggling. Did anyone go down the eventp route and did you find any documentation or can any provide a basic example of the callback module that is required. Any help so greatly appreciated... regards, Matt P.s. Great Pragmatic book Joe. From dbarker@REDACTED Wed Jun 18 20:59:15 2008 From: dbarker@REDACTED (Deryk Barker) Date: Wed, 18 Jun 2008 11:59:15 -0700 Subject: [erlang-questions] Help with tuple pattern match? In-Reply-To: References: Message-ID: <48595B03.7020706@camosun.bc.ca> Caitlin Gibbons wrote: > Hi all. > > I'm looking at: > > Person={person,{name,{first,joe},{last,armstrong}},{footsize,42}}. > > and trying to extract the value of 'last', i.e., 'armstrong'. > > {_,{_,{_,{_,Last},_}=Person. > > was unsuccessful. Any help? I think your tuple nesting went slightly awry; shouldn't that be: {_,{_,{_,_},{_,Last}},{_,_}} = Person. or - from my limited experience - preferably: {person,{_Name,{first,_First},{last,Last}},{footsize,_Footsize}} = Person. which only matches tuples with the appropriate tags in them. -- |Deryk Barker, Computer Science Dept. | Music does not have to be understood| |Camosun College, Victoria, BC, Canada| It has to be listened to. | |email: dbarker@REDACTED | | |phone: +1 250 370 4452 | Hermann Scherchen. | From dking@REDACTED Thu Jun 19 00:09:57 2008 From: dking@REDACTED (David King) Date: Wed, 18 Jun 2008 15:09:57 -0700 Subject: [erlang-questions] Idiomatic Erlang, style/performance question In-Reply-To: <48597BD7.6080708@camosun.bc.ca> References: <310053.70921.qm@web28314.mail.ukl.yahoo.com> <48597BD7.6080708@camosun.bc.ca> Message-ID: <30894E33-7785-4AF6-91DB-95051DC80388@ketralnis.com> >> Hmm while writing I got a thought. Is the last function >> tail-recursive? I have only heard that term but never used it. >> Is tail-recursion faster or uses less memory, what is the >> advantage/disadvantage? >> power(N, P) when is_integer(N), N >= 0 -> >> ipower(N, P, 1); >> power(N, P) when is_integer(N), N < 0 -> >> 1/ipower(-N, P, 1). This one is not tail-recursive (the latter clause most perform a division before turning >> >> >> ipower(0, _, R) -> R; >> ipower(N, P, R) -> ipower(N-1, P, R*P). >> >> > Yup, this looks tail recursive, because the recursive call to ipower > is > the last thing in the function. > > TR functions can run in constant stack space; to write a server you > must > use tail recursion. > > -- > |Deryk Barker, Computer Science Dept. | Music does not have to be > understood| > |Camosun College, Victoria, BC, Canada| It has to be listened > to. | > |email: dbarker@REDACTED > | | > |phone: +1 250 370 4452 | Hermann > Scherchen. | > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From ok@REDACTED Thu Jun 19 00:38:13 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 19 Jun 2008 10:38:13 +1200 Subject: [erlang-questions] power(N,P), cant get it to work In-Reply-To: <8268eea30806172336p2221fd9dyd792ef5234bd0439@mail.gmail.com> References: <765646.51510.qm@web28308.mail.ukl.yahoo.com> <124090E7-C35F-4C49-85B9-2104E2C35695@cs.otago.ac.nz> <8268eea30806172336p2221fd9dyd792ef5234bd0439@mail.gmail.com> Message-ID: <0774C280-FD6F-4693-A0A9-79EA6597CCB6@cs.otago.ac.nz> On 18 Jun 2008, at 6:36 pm, Andreas Hillqvist wrote: > I can not see how the code will terminate for a negative N? Yes, my stupid mistake. Of course unary minus is available. http://www.erlang.org/doc/reference_manual/part_frame.html Select section 6.12 "Arithmetic expressions" from the table of contents on the left. From mog-lists@REDACTED Thu Jun 19 00:45:33 2008 From: mog-lists@REDACTED (mog) Date: Wed, 18 Jun 2008 17:45:33 -0500 Subject: [erlang-questions] Pretty print and clear in the shell Message-ID: <4859900D.8030609@rldn.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all I thought it would be cool to have color printing like 1> io:format("\033[31m Hello \033[0m~n"). Hello ok (that would be in color if this wasn't plain text ^_^. However this is currently escaped over the shell and ssh. Also included in this patch is clear support for the shell. typing ^L clears the screen as does io:clear() or l(). Hope this helps Mog PS sorry for cross post. I was just hoping for more feedback. Now to go implement .erlang-history.... -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIWZALeq+tARrxhnsRAvzLAJ4kxKuD/h78r1J7gS/LAs3i2xKmNwCgpayR xoPDYQzYWizmjvBlqDfKy0g= =YHY/ -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: print-clear.diff Type: text/x-patch Size: 6879 bytes Desc: not available URL: From ok@REDACTED Thu Jun 19 01:00:23 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 19 Jun 2008 11:00:23 +1200 Subject: [erlang-questions] guard sequences problem In-Reply-To: <696973.19438.qm@web28312.mail.ukl.yahoo.com> References: <696973.19438.qm@web28312.mail.ukl.yahoo.com> Message-ID: On 18 Jun 2008, at 11:47 pm, Circular Function wrote: > 36> 5<4,12<2,12>4. > true This evaluates 5 < 4 and throws the result away; it then evaluates 12 < 2 and throws the result away; it last evaluates 12 > 4 and returns a result. > > > 37> 5<4;12<2;12>4. > * 1: syntax error before: 12 > 37> > This is indeed a syntax error. You cannot use ';' to separate EXPRESSIONS. > so apparently when separating with , it just evaluates the last > expression, No, EXPRESSIONS separated by commas are ALL evaluated. The RESULT is the result of the last expression. > or at leats only returns the result of that. That's it. The comma operation in Erlang expressions is *exactly* the same as the comma operation in C expressions. In GUARDS, the comma is like a C "&&". > > and if i understand the manual right guard sequences > should be written like that but apparently not. Yes, but neither of your examples is a GUARD. > > first expression is short for > 40> begin 4<2, 3>2 end. > true > ? Obviously not. The first expression has two commas and includes 12 twice; this one has one comma, and includes 12 not at all. Part 1 of the old Erlang book is available from www.erlang.org. Joe Armstrong's new Erlang book is well worth a read. Several reads, in fact. And of course the reference manual is on-line. It would be a really good idea to put spaces around the comparison operators, "4<2" is unreadable as well as untrue. To me, this question is a perfect illustration of why I was extremely dismayed when the decision to allow individual guard tests as expressions was made. It invited precisely this confusion. For example, before this decision, the comma operator had one consistent reading: "sequence", and the and-like effect was a simple understandable consequence of "exception=failure" in guards. Now we have the surprising result that in expressions, (1 > 2, f(X)) *will* call f/1. We are stuck with the language we have, but it would be rather nice to have a style warning "Warning! Guard test used as expression! Beware of the inconsistencies!" From ok@REDACTED Thu Jun 19 01:07:38 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 19 Jun 2008 11:07:38 +1200 Subject: [erlang-questions] list comprehensions with predefined functions? In-Reply-To: <978949.66431.qm@web28314.mail.ukl.yahoo.com> References: <978949.66431.qm@web28314.mail.ukl.yahoo.com> Message-ID: <39541EF0-6E4B-489A-B8D0-357BBC6EC54A@cs.otago.ac.nz> On 19 Jun 2008, at 12:08 am, Circular Function wrote: > how do i use a predefined function in a list comprehension? The same way you use it anywhere else. > > > and if i have 2 variables how do I write then? The same way you write them anywhere else. > > > > 42> c(mymath). > {ok,mymath} > 43> [mymath:power || X <- [1,2,3] Y <- 3]. > * 1: syntax error before: Y You are missing a comma exactly where the error message tells you it is in trouble. [...... || X <- [1,2,3], Y <- 3]. Y <- 3 is pretty meaningless; a "generator" using <- expects to generate the elements of a LIST, and 3 is not a list. So [...... || X <- [1,2,3], Y <- [3]]. > > 43> [mymath:power || X <- [1,2,3], Y <- 3]. > * 1: illegal expression Where are the arguments of your function? [mymath:power(X, Y) || X <- [1,2,3], Y <- [3]]. > > 44> [mymath:cube || X <- [1,2,3]]. > * 1: illegal expression > 45> [X*2 || X <- [1,2,3]]. This should have given you a clue: the thing between the "[" and the "||" is a perfectly ordinary expression where all the normal rules about expressions apply. The only difference is that the variables bound by the generators may be used in this expression. > > [2,4,6] > 46> [X*X || X <- [1,2,3]]. > [1,4,9] > 47> [X*2 || X <- [1,2,3]]. > [2,4,6] > 48> [mymath:sq || X <- [1,2,3]]. > * 1: illegal expression Once again: where is the argument of mymath:sq? : is NEVER meaningful in Erlang. If you want to refer to a function, you always need to specify its arity, and in any case, the expression here should be a normal expression. By the way, there is nothing unusual about Erlang here. All the languages I know that have list/array/set/tuple/... comprehensions do exactly the same thing: From stonecypher@REDACTED Thu Jun 19 01:38:04 2008 From: stonecypher@REDACTED (John Haugeland) Date: Wed, 18 Jun 2008 17:38:04 -0600 Subject: [erlang-questions] Favored Linux distributions? Message-ID: <8f24f4b10806181638w1da4c05cxdf3c15a2f3e625a@mail.gmail.com> > Is there a consensus as to what distribution of Linux is most suitable to Erlang development, or does it even matter much? As any real developer will tell you, code should be tested on multiple platforms. I do my development between BSD, Centos and Vista. Enforcing broad portability is a great way to drive incorrect assumptions out of your code before they become problems. -------------- next part -------------- An HTML attachment was scrubbed... URL: From morph@REDACTED Thu Jun 19 01:55:26 2008 From: morph@REDACTED (John Chandler) Date: Wed, 18 Jun 2008 19:55:26 -0400 Subject: [erlang-questions] Favored Linux distributions? In-Reply-To: <8f24f4b10806181638w1da4c05cxdf3c15a2f3e625a@mail.gmail.com> References: <8f24f4b10806181638w1da4c05cxdf3c15a2f3e625a@mail.gmail.com> Message-ID: <9D42DA59-F291-44A9-A0AB-9F8BE3045712@growler.net> On Jun 18, 2008, at 7:38 PM, John Haugeland wrote: > > Is there a consensus as to what distribution of Linux is most > > suitable to Erlang development, or does it even matter much? > > As any real developer will tell you, code should be tested on > multiple platforms. I do my development between BSD, Centos and > Vista. > > Enforcing broad portability is a great way to drive incorrect > assumptions out of your code before they become problems. Well, sure. Client code in particular needs to be tested on whatever range of platforms you intend it to eventually run on. With regard to server code, though, it could be that one platform is consistently more or less troublesome than others in one way or another, and it might be worth considering that when specifying a server. From fritchie@REDACTED Thu Jun 19 02:21:33 2008 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Wed, 18 Jun 2008 19:21:33 -0500 Subject: [erlang-questions] Can one mnesia table fragment be further fragmented? In-Reply-To: Message of "Wed, 18 Jun 2008 00:33:24 +0800." Message-ID: <62588.1213834893@snookles.snookles.com> devdoer bird wrote: dd> I spilit one mnesia table into 3 fragments which distributed on 3 dd> machines named A,B and C,but now the the machine "A" is very busy, dd> so I decide split the fragment on the machine "A" into small dd> fragments.Does mnesia support this? No, that isn't possible with Mnesia, not as you wish/state above. If you have a table 'tab' split into fragments 'tab', 'tab_frag2', and 'tab_frag3', you can add more fragments: 'tab_frag4', 'tab_frag5', etc. You can choose to put only 1 fragment on node A, 4 fragments on node B, and 4 fragments on node C. Assuming that access to the table is evenly distributed over all fragments, then node A should be getting only 1/9 of the load, and B & C will get 4/9 of the load. -Scott From ok@REDACTED Thu Jun 19 02:31:52 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 19 Jun 2008 12:31:52 +1200 Subject: [erlang-questions] or? In-Reply-To: <997099.93066.qm@web28315.mail.ukl.yahoo.com> References: <997099.93066.qm@web28315.mail.ukl.yahoo.com> Message-ID: On 19 Jun 2008, at 4:44 am, Circular Function wrote: > > fib(N) -> > if > N > 1 -> > fib(N-1) + fib(N-2); > N == 1 or N == 0 -> > N > end. > > it doenst work, how do i use or? Don't. Don't use 'and' or 'or' in Erlang without thinking about whether they really REALLY do what you want. You almost always want something else. When you have determined that they really are what you want, don't use them anyway, because people reading your code will _think_ you made a mistake and should have used 'andalso' or 'orelse' instead. > > http://www.erlang.org/doc/reference_manual/part_frame.html > doesnt provide an actual example of how to use it. Section 6.24 of that very manual tells you exactly what to do in its very first paragraph. You need fib(N) -> if N > 1 -> fib(N-1) + fib(N-2) ; N == 1 ; N == 0 -> N end. If you know that N is an integer, it is simpler to do fib(N) -> if N > 1 -> fib(N-1) + fib(N-2) ; N >= 0 -> N % 0 or 1 end. In this case, it would be better to do fib(N) when is_integer(N), N >= 0 -> fib_aux(N). fib_aux(N) when N > 1 -> fib_aux(N-1) + fib_aux(N-2); fib_aux(N) -> N. Better still, of course, would be to use the standard linear-time tail-recursive version, or even better, the less well known logarithmic time version (an approach that works for other recurrences as well). From ok@REDACTED Thu Jun 19 02:47:12 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 19 Jun 2008 12:47:12 +1200 Subject: [erlang-questions] Help with tuple pattern match? In-Reply-To: References: <021b01c8d173$14b612a0$f21ea8c0@SSI.CORP> Message-ID: <6A338CB7-24A4-47D5-A898-E5DDA8D9200A@cs.otago.ac.nz> On 19 Jun 2008, at 7:04 am, Caitlin Gibbons wrote: > Thanks for the prompt reply. Could you explain the details of your > answer? I interpreted the structure of this form of pattern matching > as: > > Person={person,{name,{first,joe},{last,armstrong}},{footsize,42}}. > > {person,_,{name,_{first,_{last,_}},{footsize,_}=Person. > You have {person,{name,{first,joe},{last,armstrong}},{footsize,42}} you want ^^^^^^^^^ you write {person,{name,{first,_}, {last,Last }}, {footsize,_ }} or else {_ ,{_ ,_ ,{_ ,Last }},_ } replacing all the bits you are not interested in with wild-cards. The first approach keeps the 'attribute name' atoms and replaces all the uninteresting 'attribute value' constants with wild-cards. That's good to use if you are not perfectly sure that your source will be providing you with data of the correct form. The second approach replaces _everything_ except the part you want and the lists or tuples above it with wild-cards. Me, I'd go for the first approach because it is much much easier to get the right number of curly braces and commas that way. From stonecypher@REDACTED Thu Jun 19 03:14:33 2008 From: stonecypher@REDACTED (John Haugeland) Date: Wed, 18 Jun 2008 19:14:33 -0600 Subject: [erlang-questions] Favored Linux distributions? In-Reply-To: <9D42DA59-F291-44A9-A0AB-9F8BE3045712@growler.net> References: <8f24f4b10806181638w1da4c05cxdf3c15a2f3e625a@mail.gmail.com> <9D42DA59-F291-44A9-A0AB-9F8BE3045712@growler.net> Message-ID: <8f24f4b10806181814o2ee08424n5196b7ccda4e36d9@mail.gmail.com> It is my opinion that even when you intend to deploy on a single well known environment, and even if you ignore that things change in the future unexpectedly, the extra rigor applied by the shifting demands of alternate environments is desirable. But hey, one man's wine is another man's poison, no? On Wed, Jun 18, 2008 at 5:55 PM, John Chandler wrote: > > On Jun 18, 2008, at 7:38 PM, John Haugeland wrote: > > > Is there a consensus as to what distribution of Linux is most >> > suitable to Erlang development, or does it even matter much? >> >> As any real developer will tell you, code should be tested on >> multiple platforms. I do my development between BSD, Centos and >> Vista. >> >> Enforcing broad portability is a great way to drive incorrect >> assumptions out of your code before they become problems. >> > > Well, sure. Client code in particular needs to be tested on > whatever range of platforms you intend it to eventually run on. > > With regard to server code, though, it could be that one platform > is consistently more or less troublesome than others in one way > or another, and it might be worth considering that when > specifying a server. > > -- --- GuaranteedVPS.com - bandwidth commitments and root starting from $12.98/mo -------------- next part -------------- An HTML attachment was scrubbed... URL: From morph@REDACTED Thu Jun 19 03:48:55 2008 From: morph@REDACTED (John Chandler) Date: Wed, 18 Jun 2008 21:48:55 -0400 Subject: [erlang-questions] erl: "Can't set long node name!" In-Reply-To: <8f24f4b10806181814o2ee08424n5196b7ccda4e36d9@mail.gmail.com> References: <8f24f4b10806181638w1da4c05cxdf3c15a2f3e625a@mail.gmail.com> <9D42DA59-F291-44A9-A0AB-9F8BE3045712@growler.net> <8f24f4b10806181814o2ee08424n5196b7ccda4e36d9@mail.gmail.com> Message-ID: <27E00D81-9A25-4CB2-887F-A49A19ACB6FC@growler.net> I was following an example and found that my configuration lacks the ability to "set long node name." I checked the crash dump, and it wasn't informative to me. I would attach it, but it's 180K, so I'll only do that if asked to. So, any idea what I should change in my configuration? -jmc ==== $ erl -name foo {error_logger,{{2008,6,18},{21,43,13}},"Can't set long node name! \nPlease check your configuration\n",[]} {error_logger,{{2008,6,18},{21,43,13}},crash_report,[[{pid,<0.20.0>}, {registered_name,net_kernel},{error_info,{error,badarg}}, {initial_call,{gen,init_it,[gen_server,<0.17.0>,<0.17.0>, {local,net_kernel},net_kernel,{foo,longnames,15000},[]]}},{ancestors, [net_sup,kernel_sup,<0.8.0>]},{messages,[]},{links,[<0.17.0>]}, {dictionary,[{longnames,true}]},{trap_exit,true},{status,running}, {heap_size,233},{stack_size,21},{reductions,363}],[]]} {error_logger,{{2008,6,18},{21,43,13}},supervisor_report,[{supervisor, {local,net_sup}},{errorContext,start_error},{reason, {'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel}, {mfa,{net_kernel,start_link,[[foo,longnames]]}}, {restart_type,permanent},{shutdown,2000},{child_type,worker}]}]} {error_logger,{{2008,6,18},{21,43,13}},supervisor_report,[{supervisor, {local,kernel_sup}},{errorContext,start_error},{reason,shutdown}, {offender,[{pid,undefined},{name,net_sup},{mfa, {erl_distribution,start_link,[]}},{restart_type,permanent}, {shutdown,infinity},{child_type,supervisor}]}]} {error_logger,{{2008,6,18},{21,43,13}},crash_report,[[{pid,<0.7.0>}, {registered_name,[]},{error_info,{shutdown,{kernel,start,[normal, []]}}},{initial_call,{application_master,init,[<0.5.0>,<0.6.0>, {appl_data,kernel, [application_controller,erl_reply,auth,boot_server,code_server,disk_log_ server,disk_log_sup,erl_prim_loader,error_logger,file_server_2,fixtable_ server,global_group,global_name_server,heart,init,kernel_config,kernel_s up,net_kernel,net_sup,rex,user,os_server,ddll_server,erl_epmd,inet_db,pg 2],undefined,{kernel,[]}, [application,application_controller,application_master,application_start er,auth,code,code_aux,packages,code_server,dist_util,erl_boot_server,erl _distribution,erl_prim_loader,erl_reply,erlang,error_handler,error_logge r,file,file_server,file_io_server,prim_file,global,global_group,global_s earch,group,heart,hipe_unified_loader,inet6_tcp,inet6_tcp_dist,inet6_udp ,inet_config,inet_hosts,inet_gethost_native,inet_tcp_dist,init,kernel,ke rnel_config,net,net_adm,net_kernel,os,ram_file,rpc,user,user_drv,user_su p,disk_log,disk_log_1,disk_log_server,disk_log_sup,dist_ac,erl_ddll,erl_ epmd,erts_debug,gen_tcp,gen_udp,gen_sctp,prim_inet,inet,inet_db,inet_dns ,inet_parse,inet_res,inet_tcp,inet_udp,inet_sctp,pg2,seq_trace,wrap_log_ reader,zlib,otp_ring0],[],infinity,infinity},normal]}},{ancestors, [<0.6.0>]},{messages,[{'EXIT',<0.8.0>,normal}]},{links, [<0.6.0>,<0.5.0>]},{dictionary,[]},{trap_exit,true},{status,running}, {heap_size,987},{stack_size,21},{reductions,2063}],[]]} {error_logger,{{2008,6,18},{21,43,13}},std_info,[{application,kernel}, {exited,{shutdown,{kernel,start,[normal,[]]}}},{type,permanent}]} {"Kernel pid terminated",application_controller,"{application_start_failure,kernel, {shutdown,{kernel,start,[normal,[]]}}}"} Crash dump was written to: erl_crash.dump Kernel pid terminated (application_controller) ({application_start_failure,kernel,{shutdown,{kernel,start,[normal, []]}}}) From devdoer2@REDACTED Thu Jun 19 04:03:46 2008 From: devdoer2@REDACTED (devdoer bird) Date: Thu, 19 Jun 2008 10:03:46 +0800 Subject: [erlang-questions] Can one mnesia table fragment be further fragmented? In-Reply-To: <62588.1213834893@snookles.snookles.com> References: <62588.1213834893@snookles.snookles.com> Message-ID: Thanks. Thant means ,I need fragment table many times not for load balance but for decreasing one specific node's (the node is not as powerful as others ) load. Eg. Node A has 2 fragments,Node B has 2 fragments. Each fragment has equal records(say 100 records each fragment). Now the Node B is heavily loaded.If I adding one or two fragments in Node A , the load of Node B won't be decreased ,for the records from Node A will be splited and moved to the new fragments,the records on Node A is 50+50+50+50=200 ,the total number of Node B's record is still 200. In order to decreas the load on Node B,I have to calcaulate carefully how many new fragments should be added. If there's a way for mnesia to specify which node's fragment should be splited when addin a new fragment,the problems above will be solved.Take the example above ,when adding a new fragment,I can tell mnesia to split Node B's fragment. 2008/6/19, Scott Lystig Fritchie : > > devdoer bird wrote: > > dd> I spilit one mnesia table into 3 fragments which distributed on 3 > dd> machines named A,B and C,but now the the machine "A" is very busy, > > dd> so I decide split the fragment on the machine "A" into small > dd> fragments.Does mnesia support this? > > No, that isn't possible with Mnesia, not as you wish/state above. > > If you have a table 'tab' split into fragments 'tab', 'tab_frag2', and > 'tab_frag3', you can add more fragments: 'tab_frag4', 'tab_frag5', etc. > You can choose to put only 1 fragment on node A, 4 fragments on node B, > and 4 fragments on node C. Assuming that access to the table is evenly > distributed over all fragments, then node A should be getting only 1/9 > of the load, and B & C will get 4/9 of the load. > > -Scott > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang-questions_efine@REDACTED Thu Jun 19 04:44:43 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Wed, 18 Jun 2008 22:44:43 -0400 Subject: [erlang-questions] Reading large (1GB+) XML files. In-Reply-To: <1213826845.6813.1.camel@matt> References: <1213826845.6813.1.camel@matt> Message-ID: <6c2563b20806181944h23f9e479h91680742845569c4@mail.gmail.com> I don't know xmerl eventp, but for such large XML files it may be prudent performance-wise to interface to a fast 'C' SAX parser such as expat. The ejabberd project has an Erlang driver written to use the expat parser. It's also some pretty good code for a proven real-world system that one could learn a lot from. You may be able to adapt the code (I think it's called xml_stream.erl and the 'C' files under c_src) although if, as you write, you have only been doing Erlang for a few weeks it may be a bit of a leap to get into linked-in drivers. Hope this helps. On Wed, Jun 18, 2008 at 6:07 PM, Matt Harrison wrote: > I have to extract information from some extremely large XML files, 6GB+ > as well. All signs point to xmerl_eventp but the documentation is non > exactly fleshed out and as I've only been doing erlang for a few weeks I > am really struggling. > > Did anyone go down the eventp route and did you find any documentation > or can any provide a basic example of the callback module that is > required. > > Any help so greatly appreciated... > > regards, > > Matt > > P.s. Great Pragmatic book Joe. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang-questions_efine@REDACTED Thu Jun 19 05:00:03 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Wed, 18 Jun 2008 23:00:03 -0400 Subject: [erlang-questions] erl: "Can't set long node name!" In-Reply-To: <27E00D81-9A25-4CB2-887F-A49A19ACB6FC@growler.net> References: <8f24f4b10806181638w1da4c05cxdf3c15a2f3e625a@mail.gmail.com> <9D42DA59-F291-44A9-A0AB-9F8BE3045712@growler.net> <8f24f4b10806181814o2ee08424n5196b7ccda4e36d9@mail.gmail.com> <27E00D81-9A25-4CB2-887F-A49A19ACB6FC@growler.net> Message-ID: <6c2563b20806182000kcc1fa1eh581443848b1aa484@mail.gmail.com> Is your OS set up to have a long hostname? What does hostname --long show? I think Erlang expects to see something like mybox.domain.name. Very often, home or personal Linux installations are created without a domain name, so hostname --long returns something just like "mybox". You need to change this so that you have a domain name. To do this, I think you would need to set your box to have a static IP address (not DHCP) and modify /etc/hosts so that the static IP address for your box is known by a fully qualified name, e.g. /etc/hosts: 127.0.0.1 localhost 192.168.1.100 mybox.mydomain.com If you're not actually on a real domain (which is probable), you'll need to make one up. Hope this helps. It worked for me. On Wed, Jun 18, 2008 at 9:48 PM, John Chandler wrote: > I was following an example and found that my configuration lacks the > ability to "set long node name." I checked the crash dump, and it > wasn't informative to me. I would attach it, but it's 180K, so I'll > only do that if asked to. > > So, any idea what I should change in my configuration? > > -jmc > ==== > $ erl -name foo > {error_logger,{{2008,6,18},{21,43,13}},"Can't set long node name! > \nPlease check your configuration\n",[]} > {error_logger,{{2008,6,18},{21,43,13}},crash_report,[[{pid,<0.20.0>}, > {registered_name,net_kernel},{error_info,{error,badarg}}, > {initial_call,{gen,init_it,[gen_server,<0.17.0>,<0.17.0>, > {local,net_kernel},net_kernel,{foo,longnames,15000},[]]}},{ancestors, > [net_sup,kernel_sup,<0.8.0>]},{messages,[]},{links,[<0.17.0>]}, > {dictionary,[{longnames,true}]},{trap_exit,true},{status,running}, > {heap_size,233},{stack_size,21},{reductions,363}],[]]} > {error_logger,{{2008,6,18},{21,43,13}},supervisor_report,[{supervisor, > {local,net_sup}},{errorContext,start_error},{reason, > {'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel}, > {mfa,{net_kernel,start_link,[[foo,longnames]]}}, > {restart_type,permanent},{shutdown,2000},{child_type,worker}]}]} > {error_logger,{{2008,6,18},{21,43,13}},supervisor_report,[{supervisor, > {local,kernel_sup}},{errorContext,start_error},{reason,shutdown}, > {offender,[{pid,undefined},{name,net_sup},{mfa, > {erl_distribution,start_link,[]}},{restart_type,permanent}, > {shutdown,infinity},{child_type,supervisor}]}]} > {error_logger,{{2008,6,18},{21,43,13}},crash_report,[[{pid,<0.7.0>}, > {registered_name,[]},{error_info,{shutdown,{kernel,start,[normal, > []]}}},{initial_call,{application_master,init,[<0.5.0>,<0.6.0>, > {appl_data,kernel, > [application_controller,erl_reply,auth,boot_server,code_server,disk_log_ > server,disk_log_sup,erl_prim_loader,error_logger,file_server_2,fixtable_ > server,global_group,global_name_server,heart,init,kernel_config,kernel_s > up,net_kernel,net_sup,rex,user,os_server,ddll_server,erl_epmd,inet_db,pg > 2],undefined,{kernel,[]}, > [application,application_controller,application_master,application_start > er,auth,code,code_aux,packages,code_server,dist_util,erl_boot_server,erl > _distribution,erl_prim_loader,erl_reply,erlang,error_handler,error_logge > r,file,file_server,file_io_server,prim_file,global,global_group,global_s > earch,group,heart,hipe_unified_loader,inet6_tcp,inet6_tcp_dist,inet6_udp > ,inet_config,inet_hosts,inet_gethost_native,inet_tcp_dist,init,kernel,ke > rnel_config,net,net_adm,net_kernel,os,ram_file,rpc,user,user_drv,user_su > p,disk_log,disk_log_1,disk_log_server,disk_log_sup,dist_ac,erl_ddll,erl_ > epmd,erts_debug,gen_tcp,gen_udp,gen_sctp,prim_inet,inet,inet_db,inet_dns > ,inet_parse,inet_res,inet_tcp,inet_udp,inet_sctp,pg2,seq_trace,wrap_log_ > reader,zlib,otp_ring0],[],infinity,infinity},normal]}},{ancestors, > [<0.6.0>]},{messages,[{'EXIT',<0.8.0>,normal}]},{links, > [<0.6.0>,<0.5.0>]},{dictionary,[]},{trap_exit,true},{status,running}, > {heap_size,987},{stack_size,21},{reductions,2063}],[]]} > {error_logger,{{2008,6,18},{21,43,13}},std_info,[{application,kernel}, > {exited,{shutdown,{kernel,start,[normal,[]]}}},{type,permanent}]} > {"Kernel pid > terminated",application_controller,"{application_start_failure,kernel, > {shutdown,{kernel,start,[normal,[]]}}}"} > > Crash dump was written to: erl_crash.dump > Kernel pid terminated (application_controller) > ({application_start_failure,kernel,{shutdown,{kernel,start,[normal, > []]}}}) > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang-questions_efine@REDACTED Thu Jun 19 05:12:07 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Wed, 18 Jun 2008 23:12:07 -0400 Subject: [erlang-questions] Is there size limit for socket buffer? In-Reply-To: <638555680806180139x13cad832m786fc55d7300b7dd@mail.gmail.com> References: <638555680806180139x13cad832m786fc55d7300b7dd@mail.gmail.com> Message-ID: <6c2563b20806182012r175d1c34la8e9e3c5fad079c5@mail.gmail.com> I assume you mean 1492 bytes, not bits. You don't show how you opened the socket, or say what happens after 1492 bytes, so I'll assume it was opened in passive binary mode and you only did one read. If these assumptions are correct, you need to read the socket in a loop, appending the data to a binary, until all the data has been received. How do you know when all the data has been received? You need a protocol of some sort. It could be as simple as expecting the first 4 bytes to hold the length of the data (not very reliable), or something like byte-stuffed data. Hope this helps. 2008/6/18 aicean : > I'm use socket receive data from other program.When data size is over 1492 > bit,I can't get any the remaining data. > Any one has a clue? > > Thanks! > > -- > aicean > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From morph@REDACTED Thu Jun 19 05:39:53 2008 From: morph@REDACTED (John Chandler) Date: Wed, 18 Jun 2008 23:39:53 -0400 Subject: [erlang-questions] erl: "Can't set long node name!" In-Reply-To: <6c2563b20806182000kcc1fa1eh581443848b1aa484@mail.gmail.com> References: <8f24f4b10806181638w1da4c05cxdf3c15a2f3e625a@mail.gmail.com> <9D42DA59-F291-44A9-A0AB-9F8BE3045712@growler.net> <8f24f4b10806181814o2ee08424n5196b7ccda4e36d9@mail.gmail.com> <27E00D81-9A25-4CB2-887F-A49A19ACB6FC@growler.net> <6c2563b20806182000kcc1fa1eh581443848b1aa484@mail.gmail.com> Message-ID: Thanks. I think you may be right. I'll have to work on how to set this up, because this is a Mac and the Unix networking tools only *sort of* work. In particular, "hostname" doesn't have a "--long" option. Static IP may be a bit challenging, or at least laborious, because my machine is a laptop moved relatively frequently to different locations. And I'm not clear on how to set up a "domain" in the sense you mean. I have a couple of them, but they're not DNS'd to my laptop. I wonder if Erlang will evolve to match the way people use computers now (and didn't 10, 20 years ago). -jmc On Jun 18, 2008, at 11:00 PM, Edwin Fine wrote: > Is your OS set up to have a long hostname? What does > hostname --long show? I think Erlang expects to see > something like mybox.domain.name. Very often, home or > personal Linux installations are created without a domain > name, so hostname --long returns something just like > "mybox". You need to change this so that you have a domain > name. To do this, I think you would need to set your box to > have a static IP address (not DHCP) and modify /etc/hosts so > that the static IP address for your box is known by a fully > qualified name, e.g. > > /etc/hosts: > 127.0.0.1 localhost > 192.168.1.100 mybox.mydomain.com > > If you're not actually on a real domain (which is probable), you'll > need to make one up. > > Hope this helps. It worked for me. > > On Wed, Jun 18, 2008 at 9:48 PM, John Chandler > wrote: > I was following an example and found that my configuration lacks the > ability to "set long node name." I checked the crash dump, and it > wasn't informative to me. I would attach it, but it's 180K, so I'll > only do that if asked to. > > So, any idea what I should change in my configuration? > > -jmc > ==== > $ erl -name foo > {error_logger,{{2008,6,18},{21,43,13}},"Can't set long node name! > \nPlease check your configuration\n",[]} > {error_logger,{{2008,6,18},{21,43,13}},crash_report,[[{pid,<0.20.0>}, > {registered_name,net_kernel},{error_info,{error,badarg}}, > {initial_call,{gen,init_it,[gen_server,<0.17.0>,<0.17.0>, > {local,net_kernel},net_kernel,{foo,longnames,15000},[]]}},{ancestors, > [net_sup,kernel_sup,<0.8.0>]},{messages,[]},{links,[<0.17.0>]}, > {dictionary,[{longnames,true}]},{trap_exit,true},{status,running}, > {heap_size,233},{stack_size,21},{reductions,363}],[]]} > {error_logger,{{2008,6,18},{21,43,13}},supervisor_report,[{supervisor, > {local,net_sup}},{errorContext,start_error},{reason, > {'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel}, > {mfa,{net_kernel,start_link,[[foo,longnames]]}}, > {restart_type,permanent},{shutdown,2000},{child_type,worker}]}]} > {error_logger,{{2008,6,18},{21,43,13}},supervisor_report,[{supervisor, > {local,kernel_sup}},{errorContext,start_error},{reason,shutdown}, > {offender,[{pid,undefined},{name,net_sup},{mfa, > {erl_distribution,start_link,[]}},{restart_type,permanent}, > {shutdown,infinity},{child_type,supervisor}]}]} > {error_logger,{{2008,6,18},{21,43,13}},crash_report,[[{pid,<0.7.0>}, > {registered_name,[]},{error_info,{shutdown,{kernel,start,[normal, > []]}}},{initial_call,{application_master,init,[<0.5.0>,<0.6.0>, > {appl_data,kernel, > [application_controller,erl_reply,auth,boot_server,code_server,disk_lo > g_ > server,disk_log_sup,erl_prim_loader,error_logger,file_server_2,fixtabl > e_ > server,global_group,global_name_server,heart,init,kernel_config,kernel > _s > up,net_kernel,net_sup,rex,user,os_server,ddll_server,erl_epmd,inet_db, > pg > 2],undefined,{kernel,[]}, > [application,application_controller,application_master,application_sta > rt > er,auth,code,code_aux,packages,code_server,dist_util,erl_boot_server,e > rl > _distribution,erl_prim_loader,erl_reply,erlang,error_handler,error_log > ge > r,file,file_server,file_io_server,prim_file,global,global_group,global > _s > earch,group,heart,hipe_unified_loader,inet6_tcp,inet6_tcp_dist,inet6_u > dp > ,inet_config,inet_hosts,inet_gethost_native,inet_tcp_dist,init,kernel, > ke > rnel_config,net,net_adm,net_kernel,os,ram_file,rpc,user,user_drv,user_ > su > p,disk_log,disk_log_1,disk_log_server,disk_log_sup,dist_ac,erl_ddll,er > l_ > epmd,erts_debug,gen_tcp,gen_udp,gen_sctp,prim_inet,inet,inet_db,inet_d > ns > ,inet_parse,inet_res,inet_tcp,inet_udp,inet_sctp,pg2,seq_trace,wrap_lo > g_ > reader,zlib,otp_ring0],[],infinity,infinity},normal]}},{ancestors, > [<0.6.0>]},{messages,[{'EXIT',<0.8.0>,normal}]},{links, > [<0.6.0>,<0.5.0>]},{dictionary,[]},{trap_exit,true},{status,running}, > {heap_size,987},{stack_size,21},{reductions,2063}],[]]} > {error_logger,{{2008,6,18},{21,43,13}},std_info,[{application,kernel}, > {exited,{shutdown,{kernel,start,[normal,[]]}}},{type,permanent}]} > {"Kernel pid > terminated",application_controller,"{application_start_failure,kernel, > {shutdown,{kernel,start,[normal,[]]}}}"} > > Crash dump was written to: erl_crash.dump > Kernel pid terminated (application_controller) > ({application_start_failure,kernel,{shutdown,{kernel,start,[normal, > []]}}}) > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dnew@REDACTED Thu Jun 19 07:17:44 2008 From: dnew@REDACTED (Darren New) Date: Wed, 18 Jun 2008 22:17:44 -0700 Subject: [erlang-questions] erl: "Can't set long node name!" In-Reply-To: References: <8f24f4b10806181638w1da4c05cxdf3c15a2f3e625a@mail.gmail.com> <9D42DA59-F291-44A9-A0AB-9F8BE3045712@growler.net> <8f24f4b10806181814o2ee08424n5196b7ccda4e36d9@mail.gmail.com> <27E00D81-9A25-4CB2-887F-A49A19ACB6FC@growler.net> <6c2563b20806182000kcc1fa1eh581443848b1aa484@mail.gmail.com> Message-ID: <4859EBF8.2040101@san.rr.com> John Chandler wrote: > I wonder if Erlang will evolve to match the way people use > computers now (and didn't 10, 20 years ago). What application are you using that needs a long name that doesn't work with a short name? -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From paul-trapexit@REDACTED Thu Jun 19 09:14:44 2008 From: paul-trapexit@REDACTED (Paul Mineiro) Date: Thu, 19 Jun 2008 00:14:44 -0700 (PDT) Subject: [erlang-questions] Can one mnesia table fragment be further fragmented? In-Reply-To: References: <62588.1213834893@snookles.snookles.com> Message-ID: Re: picking what gets split, not with the default frag_hash. You can see what it's doing in mnesia_frag_hash.erl ... it basically splits the fragments in order. This means that unless there is a power-of-two number of fragments they will be of different magnitudes. You could leverage that for your load balancing by placing bigger fragments on less noded nodes, etc. In fact, maybe this is why you are seeing an uneven distribution to begin with? -- p p.z. If you are feeling your wheaties, you can make your own frag hash, and specify the module in the frag_properties when you create the table. http://www.erlang.org/doc/apps/mnesia/Mnesia_chap5.html#5.3 Don't be tempted to just use phash2 or something like that; in order to go from N fragments to M fragments you have to pass through all the intermediate stages N + 1, ..., M - 1, M ; you do not want to rehash all the data in all the buckets at each step. On Thu, 19 Jun 2008, devdoer bird wrote: > Thanks. > > Thant means ,I need fragment table many times not for load balance but for > decreasing one specific node's (the node is not as powerful as others ) > load. > > Eg. Node A has 2 fragments,Node B has 2 fragments. Each fragment has equal > records(say 100 records each fragment). > > Now the Node B is heavily loaded.If I adding one or two fragments in Node > A , the load of Node B won't be > > decreased ,for the records from Node A will be splited and moved to the new > fragments,the records on Node A is 50+50+50+50=200 ,the total number of > Node B's record is still 200. > > In order to decreas the load on Node B,I have to calcaulate carefully how > many new fragments should be added. > > If there's a way for mnesia to specify which node's fragment should be > splited when addin a new fragment,the problems above will be solved.Take the > example above ,when adding a new fragment,I can tell mnesia to split Node > B's fragment. > > 2008/6/19, Scott Lystig Fritchie : > > > > devdoer bird wrote: > > > > dd> I spilit one mnesia table into 3 fragments which distributed on 3 > > dd> machines named A,B and C,but now the the machine "A" is very busy, > > > > dd> so I decide split the fragment on the machine "A" into small > > dd> fragments.Does mnesia support this? > > > > No, that isn't possible with Mnesia, not as you wish/state above. > > > > If you have a table 'tab' split into fragments 'tab', 'tab_frag2', and > > 'tab_frag3', you can add more fragments: 'tab_frag4', 'tab_frag5', etc. > > You can choose to put only 1 fragment on node A, 4 fragments on node B, > > and 4 fragments on node C. Assuming that access to the table is evenly > > distributed over all fragments, then node A should be getting only 1/9 > > of the load, and B & C will get 4/9 of the load. > > > > -Scott > > > In an artificial world, only extremists live naturally. -- Paul Graham From raimo+erlang-questions@REDACTED Thu Jun 19 10:20:37 2008 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 19 Jun 2008 10:20:37 +0200 Subject: [erlang-questions] : erl: "Can't set long node name!" In-Reply-To: References: <8f24f4b10806181638w1da4c05cxdf3c15a2f3e625a@mail.gmail.com> <9D42DA59-F291-44A9-A0AB-9F8BE3045712@growler.net> <8f24f4b10806181814o2ee08424n5196b7ccda4e36d9@mail.gmail.com> <27E00D81-9A25-4CB2-887F-A49A19ACB6FC@growler.net> <6c2563b20806182000kcc1fa1eh581443848b1aa484@mail.gmail.com> Message-ID: <20080619082037.GA6443@erix.ericsson.se> On Wed, Jun 18, 2008 at 11:39:53PM -0400, John Chandler wrote: > Thanks. I think you may be right. I'll have to work on how to set > this up, because this is a Mac and the Unix networking tools only > *sort of* work. In particular, "hostname" doesn't have a "--long" > option. On a Mac(OS X), `hostname' gives you the fully qualified hostname and 'hostname -s' gives the short name. Try `man hostname', it works on most platforms. Why do you not run short names a'la `erl -sname foo'? If you know your hosts DNS domainname you can force the node name a'la `erl -name foo@REDACTED'. To make `erl -name foo' work, a name lookup of the short hostname must return the fully qualified hostname. Try from erlang with `inet:gethostbyname("foo").', if that does not give you a long hostname as the second element of the returned hostent tuple; long node names will not work without giving a long node name to the -name argument > > Static IP may be a bit challenging, or at least laborious, because > my machine is a laptop moved relatively frequently to different > locations. It works just as well with dynamic IP, if the name resolver (often DNS resolver) returns a fully qualified name. > > And I'm not clear on how to set up a "domain" in the sense you > mean. I have a couple of them, but they're not DNS'd to my > laptop. If you have no clear domain, use short node names; that is what they are for. Long node names are only necessary when you have to distinguish between different domains. > > I wonder if Erlang will evolve to match the way people use > computers now (and didn't 10, 20 years ago). I am afraid computer networking has not changed much the last 10, 20 years... > > -jmc > > On Jun 18, 2008, at 11:00 PM, Edwin Fine wrote: > > >Is your OS set up to have a long hostname? What does > >hostname --long show? I think Erlang expects to see > >something like mybox.domain.name. Very often, home or > >personal Linux installations are created without a domain > >name, so hostname --long returns something just like > >"mybox". You need to change this so that you have a domain > >name. To do this, I think you would need to set your box to > >have a static IP address (not DHCP) and modify /etc/hosts so > >that the static IP address for your box is known by a fully > >qualified name, e.g. > > > >/etc/hosts: > >127.0.0.1 localhost > >192.168.1.100 mybox.mydomain.com > > > >If you're not actually on a real domain (which is probable), you'll > >need to make one up. > > > >Hope this helps. It worked for me. > > > >On Wed, Jun 18, 2008 at 9:48 PM, John Chandler > >wrote: > >I was following an example and found that my configuration lacks the > >ability to "set long node name." I checked the crash dump, and it > >wasn't informative to me. I would attach it, but it's 180K, so I'll > >only do that if asked to. > > > >So, any idea what I should change in my configuration? > > > >-jmc > >==== > > $ erl -name foo > >{error_logger,{{2008,6,18},{21,43,13}},"Can't set long node name! > >\nPlease check your configuration\n",[]} > >{error_logger,{{2008,6,18},{21,43,13}},crash_report,[[{pid,<0.20.0>}, > >{registered_name,net_kernel},{error_info,{error,badarg}}, > >{initial_call,{gen,init_it,[gen_server,<0.17.0>,<0.17.0>, > >{local,net_kernel},net_kernel,{foo,longnames,15000},[]]}},{ancestors, > >[net_sup,kernel_sup,<0.8.0>]},{messages,[]},{links,[<0.17.0>]}, > >{dictionary,[{longnames,true}]},{trap_exit,true},{status,running}, > >{heap_size,233},{stack_size,21},{reductions,363}],[]]} > >{error_logger,{{2008,6,18},{21,43,13}},supervisor_report,[{supervisor, > >{local,net_sup}},{errorContext,start_error},{reason, > >{'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel}, > >{mfa,{net_kernel,start_link,[[foo,longnames]]}}, > >{restart_type,permanent},{shutdown,2000},{child_type,worker}]}]} > >{error_logger,{{2008,6,18},{21,43,13}},supervisor_report,[{supervisor, > >{local,kernel_sup}},{errorContext,start_error},{reason,shutdown}, > >{offender,[{pid,undefined},{name,net_sup},{mfa, > >{erl_distribution,start_link,[]}},{restart_type,permanent}, > >{shutdown,infinity},{child_type,supervisor}]}]} > >{error_logger,{{2008,6,18},{21,43,13}},crash_report,[[{pid,<0.7.0>}, > >{registered_name,[]},{error_info,{shutdown,{kernel,start,[normal, > >[]]}}},{initial_call,{application_master,init,[<0.5.0>,<0.6.0>, > >{appl_data,kernel, > >[application_controller,erl_reply,auth,boot_server,code_server,disk_lo > >g_ > >server,disk_log_sup,erl_prim_loader,error_logger,file_server_2,fixtabl > >e_ > >server,global_group,global_name_server,heart,init,kernel_config,kernel > >_s > >up,net_kernel,net_sup,rex,user,os_server,ddll_server,erl_epmd,inet_db, > >pg > >2],undefined,{kernel,[]}, > >[application,application_controller,application_master,application_sta > >rt > >er,auth,code,code_aux,packages,code_server,dist_util,erl_boot_server,e > >rl > >_distribution,erl_prim_loader,erl_reply,erlang,error_handler,error_log > >ge > >r,file,file_server,file_io_server,prim_file,global,global_group,global > >_s > >earch,group,heart,hipe_unified_loader,inet6_tcp,inet6_tcp_dist,inet6_u > >dp > >,inet_config,inet_hosts,inet_gethost_native,inet_tcp_dist,init,kernel, > >ke > >rnel_config,net,net_adm,net_kernel,os,ram_file,rpc,user,user_drv,user_ > >su > >p,disk_log,disk_log_1,disk_log_server,disk_log_sup,dist_ac,erl_ddll,er > >l_ > >epmd,erts_debug,gen_tcp,gen_udp,gen_sctp,prim_inet,inet,inet_db,inet_d > >ns > >,inet_parse,inet_res,inet_tcp,inet_udp,inet_sctp,pg2,seq_trace,wrap_lo > >g_ > >reader,zlib,otp_ring0],[],infinity,infinity},normal]}},{ancestors, > >[<0.6.0>]},{messages,[{'EXIT',<0.8.0>,normal}]},{links, > >[<0.6.0>,<0.5.0>]},{dictionary,[]},{trap_exit,true},{status,running}, > >{heap_size,987},{stack_size,21},{reductions,2063}],[]]} > >{error_logger,{{2008,6,18},{21,43,13}},std_info,[{application,kernel}, > >{exited,{shutdown,{kernel,start,[normal,[]]}}},{type,permanent}]} > >{"Kernel pid > >terminated",application_controller,"{application_start_failure,kernel, > >{shutdown,{kernel,start,[normal,[]]}}}"} > > > >Crash dump was written to: erl_crash.dump > >Kernel pid terminated (application_controller) > >({application_start_failure,kernel,{shutdown,{kernel,start,[normal, > >[]]}}}) > >_______________________________________________ > >erlang-questions mailing list > >erlang-questions@REDACTED > >http://www.erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From bjorn@REDACTED Thu Jun 19 10:24:34 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 19 Jun 2008 10:24:34 +0200 Subject: [erlang-questions] Dialyzer problems on Windows In-Reply-To: <3dbc6d1c0806181459u74b7512fja3f4dac41292120d@mail.gmail.com> References: <3dbc6d1c0806171409td0018ffq1db20fadbb10f5ef@mail.gmail.com> <4858C791.3070100@it.uu.se> <3dbc6d1c0806181459u74b7512fja3f4dac41292120d@mail.gmail.com> Message-ID: "Robert Virding" writes: > Hi Tobias, > > I see the benefits of the plt handling but it would be very nice to have a > default plt containing the standard stuff in a known place which is used > unless you tell it not to. Users/apps could then have their own plts which > augment the standard one. Or completely replace it for building special > apps. It is non-trivial to implement support for more than one PLT, because of dependecies between applications. Therefore Dialyzer only uses a single PLT. We don't want to have the PLT inside OTP itself (as it was earlier), because that causes problems if OTP is installed in a read-only file-system or in Clearcase. However, we might somehow provide a pre-built PLT that you copy to your home directory to get started quickly. /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From rapsey@REDACTED Thu Jun 19 10:55:22 2008 From: rapsey@REDACTED (Rapsey) Date: Thu, 19 Jun 2008 10:55:22 +0200 Subject: [erlang-questions] why is gen_tcp:send slow? Message-ID: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> I have a streaming server written in Erlang. When it was pushing 200-300 mb/s the CPU was getting completely hammered. I traced the problem to gen_tcp:send. So instead of sending every audio/video packet with a single gen_tcp:send call, I buffer 3 packets and then send them all at once. CPU consumption dropped dramatically. On one of the servers I have a simple proxy, the main process that sends packets between the client and some other server looks like this: transmit_loop({tcp, Sock, Data}, P) when P#transdat.client == Sock -> gen_tcp:send(P#transdat.server, Data), inet:setopts(P#transdat.client, [{active, once}]), {ok, P}; transmit_loop({tcp, Sock, Data}, P) when P#transdat.server == Sock -> gen_tcp:send(P#transdat.client, Data), inet:setopts(P#transdat.server, [{active, once}]), {ok, P}; transmit_loop({start, ServerPort}, P) -> {ok, Sock} = gen_tcp:connect("127.0.0.1", ServerPort, [binary, {active, once}, {packet, 0}]), {ok, P#transdat{server = Sock}}; transmit_loop({tcp_closed, _}, _) -> exit(stop). The proxy is eating more CPU time than the streaming server. Is this normal behavior? The server is running OSX 10.4 Sergej -------------- next part -------------- An HTML attachment was scrubbed... URL: From morph@REDACTED Thu Jun 19 13:04:12 2008 From: morph@REDACTED (John Chandler) Date: Thu, 19 Jun 2008 07:04:12 -0400 Subject: [erlang-questions] : erl: "Can't set long node name!" In-Reply-To: <20080619082037.GA6443@erix.ericsson.se> References: <8f24f4b10806181638w1da4c05cxdf3c15a2f3e625a@mail.gmail.com> <9D42DA59-F291-44A9-A0AB-9F8BE3045712@growler.net> <8f24f4b10806181814o2ee08424n5196b7ccda4e36d9@mail.gmail.com> <27E00D81-9A25-4CB2-887F-A49A19ACB6FC@growler.net> <6c2563b20806182000kcc1fa1eh581443848b1aa484@mail.gmail.com> <20080619082037.GA6443@erix.ericsson.se> Message-ID: On Jun 19, 2008, at 4:20 AM, Raimo Niskanen wrote: > On Wed, Jun 18, 2008 at 11:39:53PM -0400, John Chandler wrote: >> Thanks. I think you may be right. I'll have to work on how to set >> this up, because this is a Mac and the Unix networking tools only >> *sort of* work. In particular, "hostname" doesn't have a "--long" >> option. > > On a Mac(OS X), `hostname' gives you the fully qualified > hostname and 'hostname -s' gives the short name. Try > `man hostname', it works on most platforms. "hostname" works fine, it just doesn't have the same options as on Linux. But if your network needs to be re-initialized, you can't use "ifconfig blah blah down" and "ifconfig blah blah up" to accomplish that, you have to do some GUI hoohah. But I digress. On my machine, "hostname" with or without the "-s" yields a fully-unqualified hostname. > Why do you not run short names a'la `erl -sname foo'? Because I am/was unaware of the option. I am following an example in an online article that doesn't mention that those trying to follow along on a machine that boots up with DHCP from a consumer-level private-network router will get a not-entirely-helpful screenful of information and probably ought to use this other option. > If you know your hosts DNS domainname you can force > the node name a'la `erl -name foo@REDACTED'. > > To make `erl -name foo' work, a name lookup of the > short hostname must return the fully qualified > hostname. Try from erlang with `inet:gethostbyname("foo").', > if that does not give you a long hostname as the > second element of the returned hostent tuple; > long node names will not work without giving > a long node name to the -name argument This makes sense. I have no particular commitment to using long node names, as long as I can get nodes to work properly without them. >> >> Static IP may be a bit challenging, or at least laborious, because >> my machine is a laptop moved relatively frequently to different >> locations. > > It works just as well with dynamic IP, if the name > resolver (often DNS resolver) returns a fully qualified name. There is no FQDN that resolves to my laptop, unless I'm mistaken. >> >> And I'm not clear on how to set up a "domain" in the sense you >> mean. I have a couple of them, but they're not DNS'd to my >> laptop. > > If you have no clear domain, use short node names; > that is what they are for. Long node names are > only necessary when you have to distinguish between > different domains. I think this may be the answer. >> I wonder if Erlang will evolve to match the way people use >> computers now (and didn't 10, 20 years ago). > > I am afraid computer networking has not changed much > the last 10, 20 years... My remark was ill-advised, and I apologize. Erlang covers the case I'm concerned with, it's just the article I was following that didn't. And I was reading the article in order to reduce, at least a little, my vast ignorance of Erlang. Still, 10, 20 years ago people did not commonly pick up their computers, take them home, or to work, or to Charbucks, or to a friend's house, and (reasonably) expect them to be able to join the local net. Back in the day, if you took your workstation to another branch of your large company, someone would have to add your MAC address to one NIS map, and give you an IP and a hostname before your machine would boot properly. -jmc From raimo+erlang-questions@REDACTED Thu Jun 19 14:44:19 2008 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 19 Jun 2008 14:44:19 +0200 Subject: [erlang-questions] : : erl: "Can't set long node name!" In-Reply-To: References: <8f24f4b10806181638w1da4c05cxdf3c15a2f3e625a@mail.gmail.com> <9D42DA59-F291-44A9-A0AB-9F8BE3045712@growler.net> <8f24f4b10806181814o2ee08424n5196b7ccda4e36d9@mail.gmail.com> <27E00D81-9A25-4CB2-887F-A49A19ACB6FC@growler.net> <6c2563b20806182000kcc1fa1eh581443848b1aa484@mail.gmail.com> <20080619082037.GA6443@erix.ericsson.se> Message-ID: <20080619124419.GC6443@erix.ericsson.se> On Thu, Jun 19, 2008 at 07:04:12AM -0400, John Chandler wrote: > On Jun 19, 2008, at 4:20 AM, Raimo Niskanen wrote: > >On Wed, Jun 18, 2008 at 11:39:53PM -0400, John Chandler wrote: > >>Thanks. I think you may be right. I'll have to work on how to set > >>this up, because this is a Mac and the Unix networking tools only > >>*sort of* work. In particular, "hostname" doesn't have a "--long" > >>option. > > > >On a Mac(OS X), `hostname' gives you the fully qualified > >hostname and 'hostname -s' gives the short name. Try > >`man hostname', it works on most platforms. > > "hostname" works fine, it just doesn't have the same options > as on Linux. But if your network needs to be > re-initialized, you can't use "ifconfig blah blah down" and > "ifconfig blah blah up" to accomplish that, you have to do > some GUI hoohah. But I digress. On my machine, > "hostname" with or without the "-s" yields a > fully-unqualified hostname. > > >Why do you not run short names a'la `erl -sname foo'? > > Because I am/was unaware of the option. I am following an > example in an online article that doesn't mention that those > trying to follow along on a machine that boots up with DHCP > from a consumer-level private-network router will get a > not-entirely-helpful screenful of information and probably > ought to use this other option. That was nicely put. The error symptom has lots of room for improvement, I agree. Too bad the online article did not mention -sname... > > >If you know your hosts DNS domainname you can force > >the node name a'la `erl -name foo@REDACTED'. > > > >To make `erl -name foo' work, a name lookup of the > >short hostname must return the fully qualified > >hostname. Try from erlang with `inet:gethostbyname("foo").', > >if that does not give you a long hostname as the > >second element of the returned hostent tuple; > >long node names will not work without giving > >a long node name to the -name argument > > This makes sense. I have no particular commitment > to using long node names, as long as I can get nodes > to work properly without them. > > >> > >>Static IP may be a bit challenging, or at least laborious, because > >>my machine is a laptop moved relatively frequently to different > >>locations. > > > >It works just as well with dynamic IP, if the name > >resolver (often DNS resolver) returns a fully qualified name. > > There is no FQDN that resolves to my laptop, unless I'm > mistaken. > > >> > >>And I'm not clear on how to set up a "domain" in the sense you > >>mean. I have a couple of them, but they're not DNS'd to my > >>laptop. > > > >If you have no clear domain, use short node names; > >that is what they are for. Long node names are > >only necessary when you have to distinguish between > >different domains. > > I think this may be the answer. > > >>I wonder if Erlang will evolve to match the way people use > >>computers now (and didn't 10, 20 years ago). > > > >I am afraid computer networking has not changed much > >the last 10, 20 years... > > My remark was ill-advised, and I apologize. Erlang covers > the case I'm concerned with, it's just the article I was > following that didn't. And I was reading the article in > order to reduce, at least a little, my vast ignorance of > Erlang. Apology accepted. Either Erlang/OTP will have to evolve, or maybe just OTP, or maybe just the applications for Erlang/OTP. > > Still, 10, 20 years ago people did not commonly pick up > their computers, take them home, or to work, or to > Charbucks, or to a friend's house, and (reasonably) expect > them to be able to join the local net. Back in the day, if > you took your workstation to another branch of your large > company, someone would have to add your MAC address to one > NIS map, and give you an IP and a hostname before your > machine would boot properly. Quite right. And now such features as Zeroconf and Apple Bonjour are emerging. We will see what features Erlang will need to adapt. Erlang/OTP was designed for duplicated servers in a static environment. Now computer networking is becoming more dynamic, based on the old networking technology. > > -jmc -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From dot@REDACTED Sun Jun 15 19:35:57 2008 From: dot@REDACTED (Tony Finch) Date: Sun, 15 Jun 2008 18:35:57 +0100 Subject: [erlang-questions] clueless performance question In-Reply-To: References: <91a2ba3e0806101327w66ee3ff9yf2e2f90695208f81@mail.gmail.com> <490D297E-0151-489D-BF26-355E9AB486B2@cs.otago.ac.nz> Message-ID: On Fri, 13 Jun 2008, Richard A. O'Keefe wrote: > On 11 Jun 2008, at 11:57 pm, Mike Shaver wrote: > > > > It's probably[*] going to be less efficient than early binding on > > first call, but there's no reason that it can't be equivalent to early > > binding on every subsequent call, until a module replacement triggers > > the flushing of a JIT cache or what have you. > > The startup costs can be high. Yes for traditional JIT. Tracing JITs are much more lightweight. > > Work on tracing JITs indicates that the overhead of late binding can > > be made to disappear completely in the loops and functions that > > comprise virtually all of a program's work > > I keep hearing this. Please provide some references. Short overview: One method at a time is quite a waste of time http://www.ics.uci.edu/~franz/Site/pubs-pdf/C44Prepub.pdf More in depth: HotpathVM: an effective JIT compiler for resource-constrained devices http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.108.8090&rep=rep1&type=pdf Incremental dynamic code generation with trace trees http://www.ics.uci.edu/~franz/Site/pubs-pdf/ICS-TR-06-16.pdf Efficient Just-In-Time Execution of Dynamically Typed Languages Via Code Specialization Using Precise Runtime Type Inference http://www.ics.uci.edu/~franz/Site/pubs-pdf/ICS-TR-07-10.pdf Someone else's short overview: Mike Pall's notes on a trace JIT for Lua http://article.gmane.org/gmane.comp.lang.lua.general/44781 Less polished discussion: Andreas Gal's blog has lots of stuff about applying his work to the Tamarin JavaScript interpreter/JIT. http://andreasgal.com/ (look at the "recent posts" links) Tony. -- f.anthony.n.finch http://dotat.at/ CROMARTY FORTH TYNE: NORTHWEST BACKING SOUTH 3 OR 4, OCCASIONALLY 5 LATER. MODERATE BECOMING SLIGHT. SHOWERS. GOOD. From erlang@REDACTED Thu Jun 19 15:43:58 2008 From: erlang@REDACTED (Dominic Williams) Date: Thu, 19 Jun 2008 09:43:58 -0400 (EDT) Subject: [erlang-questions] Idiomatic Erlang, style/performance question In-Reply-To: <30894E33-7785-4AF6-91DB-95051DC80388@ketralnis.com> References: <310053.70921.qm@web28314.mail.ukl.yahoo.com> <48597BD7.6080708@camosun.bc.ca> <30894E33-7785-4AF6-91DB-95051DC80388@ketralnis.com> Message-ID: >>> Hmm while writing I got a thought. Is the last function >>> tail-recursive? I have only heard that term but never used it. >>> Is tail-recursion faster or uses less memory, what is the >>> advantage/disadvantage? It can be slower. The advantage is that the stack doesn't grow, so you can use the function on lists of any size without risking running out of memory. You don't always need that. >>> power(N, P) when is_integer(N), N >= 0 -> >>> ipower(N, P, 1); >>> power(N, P) when is_integer(N), N < 0 -> >>> 1/ipower(-N, P, 1). > > This one is not tail-recursive (the latter clause most perform a > division before turning It's a call to a different function (ipower), so it's not recursive at all. Had it been a call to itself, your remark would be quite correct. >>> ipower(0, _, R) -> R; >>> ipower(N, P, R) -> ipower(N-1, P, R*P). ipower is tail-recursive. Regards, Dominic Williams http://dominicwilliams.net From colm.dougan@REDACTED Thu Jun 19 18:05:43 2008 From: colm.dougan@REDACTED (Colm Dougan) Date: Thu, 19 Jun 2008 17:05:43 +0100 Subject: [erlang-questions] Signal handling (TERM, INT etc) Message-ID: <24d4f39c0806190905w1d628859x2efcc47b2ca972c4@mail.gmail.com> Hi, I don't seem to be able to find any way to trap/handle OS signals in erlang (TERM, INT etc). Am I missing something or is this just "not the erlang way", and if the latter then is there some alternative? I'd like to be able to trap a TERM signal and shutdown cleanly The best I could come up with was to start my erlang process from a shell script which itself trapped signals and then sent the erlang node a rpc message to stop cleanly. Is there a better way? Thanks, Colm From zerthurd@REDACTED Thu Jun 19 18:59:42 2008 From: zerthurd@REDACTED (Maxim Treskin) Date: Thu, 19 Jun 2008 23:59:42 +0700 Subject: [erlang-questions] Signal handling (TERM, INT etc) In-Reply-To: <24d4f39c0806190905w1d628859x2efcc47b2ca972c4@mail.gmail.com> References: <24d4f39c0806190905w1d628859x2efcc47b2ca972c4@mail.gmail.com> Message-ID: Hello Handling of UNIX-signal is not portable feature through many OS. So, you must write OS-specified driver for this operations. On 19/06/2008, Colm Dougan wrote: > Hi, > > I don't seem to be able to find any way to trap/handle OS signals in > erlang (TERM, INT etc). Am I missing something or is this just "not > the erlang way", and if the latter then is there some alternative? > I'd like to be able to trap a TERM signal and shutdown cleanly The > best I could come up with was to start my erlang process from a shell > script which itself trapped signals and then sent the erlang node a > rpc message to stop cleanly. Is there a better way? > > Thanks, > Colm > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- Maxim Treskin From erlang-questions_efine@REDACTED Thu Jun 19 19:39:47 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Thu, 19 Jun 2008 13:39:47 -0400 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> Message-ID: <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> How large is each packet? Can multiple packets fit into one TCP window? Have you looked at the TCP/IP wire-level data with Wireshark/Ethereal to see if the packets are being combined at the TCP level? If you see that you are only getting one packet per TCP frame (assuming a packet is much smaller than the window size), you might be falling foul of the Nagle congestion algorithm. The fact that manually buffering your packets improves performance suggests this may be the case. Nagle says that send, send, send is OK, receive, receive, receive is ok, and even send, receive, send, receive is ok, but you get into trouble if sends and receives are mixed asymmetrically on the same socket (e.g. send, send, receive). Also, I don't understand your transmit_loop. Where is it looping (or am I misunderstanding something)? >From what I have seen, people writing Erlang TCP/IP code do an {active, once} receive, and when getting the first packet, drop into another loop that does a passive receive until there's no data waiting, then go back into the {active, once} receive. Are you doing this? I am not sure, but I fear that if all your receives are {active, once} it will incur more CPU overhead than the active/passive split. It's hard to know because I can't see enough of your code to know what you are doing overall. Disclaimer: I'm no Erlang or TCP/IP expert. Hope this helps. 2008/6/19 Rapsey : > I have a streaming server written in Erlang. When it was pushing 200-300 > mb/s the CPU was getting completely hammered. I traced the problem to > gen_tcp:send. > So instead of sending every audio/video packet with a single gen_tcp:send > call, I buffer 3 packets and then send them all at once. CPU consumption > dropped dramatically. > On one of the servers I have a simple proxy, the main process that sends > packets between the client and some other server looks like this: > > transmit_loop({tcp, Sock, Data}, P) when P#transdat.client == Sock -> > gen_tcp:send(P#transdat.server, Data), > inet:setopts(P#transdat.client, [{active, once}]), > {ok, P}; > transmit_loop({tcp, Sock, Data}, P) when P#transdat.server == Sock -> > gen_tcp:send(P#transdat.client, Data), > inet:setopts(P#transdat.server, [{active, once}]), > {ok, P}; > transmit_loop({start, ServerPort}, P) -> > {ok, Sock} = gen_tcp:connect("127.0.0.1", ServerPort, [binary, > {active, once}, {packet, 0}]), > {ok, P#transdat{server = Sock}}; > transmit_loop({tcp_closed, _}, _) -> > exit(stop). > > The proxy is eating more CPU time than the streaming server. > Is this normal behavior? The server is running OSX 10.4 > > > Sergej > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rapsey@REDACTED Thu Jun 19 20:06:13 2008 From: rapsey@REDACTED (Rapsey) Date: Thu, 19 Jun 2008 20:06:13 +0200 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> Message-ID: <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> It loops from another module, that way I can update the code at any time without disrupting anything. The packets are generally a few hundred bytes big, except keyframes which tend to be in the kB range. I haven't tried looking with wireshark. Still it seems a bit odd that a large CPU consumption would be the symptom. The traffic is strictly one way. Either someone is sending the stream or receiving it. The transmit could of course be written with a passive receive, but the code would be significantly uglier. I'm sure someone here knows if setting {active, once} every packet is CPU intensive or not. It seems the workings of gen_tcp is quite platform dependent. If I run the code in windows, sending more than 128 bytes per gen_tcp call significantly decreases network output. Oh and I forgot to mention I use R12B-3. Sergej On Thu, Jun 19, 2008 at 7:39 PM, Edwin Fine wrote: > How large is each packet? Can multiple packets fit into one TCP window? > Have you looked at the TCP/IP wire-level data with Wireshark/Ethereal to see > if the packets are being combined at the TCP level? If you see that you are > only getting one packet per TCP frame (assuming a packet is much smaller > than the window size), you might be falling foul of the Nagle congestion > algorithm. The fact that manually buffering your packets improves > performance suggests this may be the case. Nagle says that send, send, send > is OK, receive, receive, receive is ok, and even send, receive, send, > receive is ok, but you get into trouble if sends and receives are mixed > asymmetrically on the same socket (e.g. send, send, receive). > > Also, I don't understand your transmit_loop. Where is it looping (or am I > misunderstanding something)? > > From what I have seen, people writing Erlang TCP/IP code do an {active, > once} receive, and when getting the first packet, drop into another loop > that does a passive receive until there's no data waiting, then go back into > the {active, once} receive. Are you doing this? I am not sure, but I fear > that if all your receives are {active, once} it will incur more CPU overhead > than the active/passive split. It's hard to know because I can't see enough > of your code to know what you are doing overall. Disclaimer: I'm no Erlang > or TCP/IP expert. > > Hope this helps. > > 2008/6/19 Rapsey : > >> I have a streaming server written in Erlang. When it was pushing 200-300 >> mb/s the CPU was getting completely hammered. I traced the problem to >> gen_tcp:send. >> So instead of sending every audio/video packet with a single gen_tcp:send >> call, I buffer 3 packets and then send them all at once. CPU consumption >> dropped dramatically. >> On one of the servers I have a simple proxy, the main process that sends >> packets between the client and some other server looks like this: >> >> transmit_loop({tcp, Sock, Data}, P) when P#transdat.client == Sock -> >> gen_tcp:send(P#transdat.server, Data), >> inet:setopts(P#transdat.client, [{active, once}]), >> {ok, P}; >> transmit_loop({tcp, Sock, Data}, P) when P#transdat.server == Sock -> >> gen_tcp:send(P#transdat.client, Data), >> inet:setopts(P#transdat.server, [{active, once}]), >> {ok, P}; >> transmit_loop({start, ServerPort}, P) -> >> {ok, Sock} = gen_tcp:connect("127.0.0.1", ServerPort, [binary, >> {active, once}, {packet, 0}]), >> {ok, P#transdat{server = Sock}}; >> transmit_loop({tcp_closed, _}, _) -> >> exit(stop). >> >> The proxy is eating more CPU time than the streaming server. >> Is this normal behavior? The server is running OSX 10.4 >> >> >> Sergej >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From petr.sturc@REDACTED Thu Jun 19 22:12:01 2008 From: petr.sturc@REDACTED (Petr Sturc) Date: Thu, 19 Jun 2008 22:12:01 +0200 Subject: [erlang-questions] esdl - Driver Failed {error, driver_incorrect_version} Message-ID: <73b10e5b0806191312u6d39c9bdha17919038ce41876@mail.gmail.com> Hi, I would like to try esdl on windows with Erlang OTP R12B. When running the example testsprite:go() I am getting: 1> testsprite:go(). Driver Failed {error,driver_incorrect_version} ** exception error: einval in function open_port/2 called as open_port({spawn,"sdl_driver"},[binary]) in call from sdl:init/1 in call from testsprite:go/1 IMO it is because the binaries are compiled against R11. Am I right? Does anyone made ESDL windows binary for R12? Regards, Petr From matt@REDACTED Thu Jun 19 23:24:15 2008 From: matt@REDACTED (Matt Harrison) Date: Thu, 19 Jun 2008 22:24:15 +0100 Subject: [erlang-questions] Erlsom it's so close to being SAXy Message-ID: <485ACE7F.901@lummie.co.uk> See what I did there with the subject title :) All Has anyone used Erlsom for sax parsing straight from a file, i.e without using file:read_file to load a whole file. I can't seem to find an appropriate call, and the docs don't seem to cover it. I have large files 6gb+ that I need to SAX parse, the main requirement being able to parse them with about 1gb of memory. Erlsom works a treat with files that are a few 100Mb but requires the whole file loaded in memory, which kinda kills the main benefit of sax parsing in my opinion. I am not especially bothered about speed, (it was suggested that I look at c parsers linked into erlang) as this is for a data import process that will only happen rarely (mainly for development and testing purposes). I still don't seem to be able to find any xmerl_eventp examples so if you have one please let me know. regards and thanks, Matt I haven't discarded the using a c library I'm just fairly new to erlang and would prefer an erlang solution if possible as I don't want to venture into ports quite yet :) From dougedmunds@REDACTED Fri Jun 20 01:00:57 2008 From: dougedmunds@REDACTED (Doug Edmunds) Date: Thu, 19 Jun 2008 16:00:57 -0700 Subject: [erlang-questions] testing a list against an ets table Message-ID: Scenario: The goal is determine if any tuples in the List are not in the Table (and if so, identify which are missing). I wrote two functions, one which uses ets:match_object/2 and the other which uses lists:member/2. Function parameters: List is a list of tuples [{a,b}, {c,d,e}, {a,d}, ...]. Table is an open ets table of type 'bag', which means there can be more than one item with the same key in the table. Which of these methods is preferable, and why? %% a quick display utility function io(X) -> io:format("~p~n",[X]). findlist1(List,Table) -> R1 = [E || E <- List, length(ets:match_object(Table, E)) == 0], io(R1), %% elements not found in table io(length(R1)). %% if length(R1) == 0, all are in list findlist2(List, Table) -> AsList = ets:tab2list(Table), R1 = [E || E <- List, not lists:member(E,AsList)], io(R1), %% same as above io(length(R1)). length(R1) could be turned into true/false thus: case length(R1) of 0 -> true; % all in table _Other -> false end. What are some other alternatives? Also, I would appreciate suggestions on how to short-circuit the evaluation (stopping as soon as one element of the list is not found in the table). Thanks. -dae -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Jun 20 01:58:35 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 20 Jun 2008 11:58:35 +1200 Subject: [erlang-questions] : erl: "Can't set long node name!" In-Reply-To: <20080619082037.GA6443@erix.ericsson.se> References: <8f24f4b10806181638w1da4c05cxdf3c15a2f3e625a@mail.gmail.com> <9D42DA59-F291-44A9-A0AB-9F8BE3045712@growler.net> <8f24f4b10806181814o2ee08424n5196b7ccda4e36d9@mail.gmail.com> <27E00D81-9A25-4CB2-887F-A49A19ACB6FC@growler.net> <6c2563b20806182000kcc1fa1eh581443848b1aa484@mail.gmail.com> <20080619082037.GA6443@erix.ericsson.se> Message-ID: <622B4A7C-1D94-49C9-8542-85A194ED8E98@cs.otago.ac.nz> On 19 Jun 2008, at 8:20 pm, Raimo Niskanen wrote: > On Wed, Jun 18, 2008 at 11:39:53PM -0400, John Chandler wrote: >> Thanks. I think you may be right. I'll have to work on how to set >> this up, because this is a Mac and the Unix networking tools only >> *sort of* work. In particular, "hostname" doesn't have a "--long" >> option. "--" is a good sign of a GNU extension. I'm using Solaris 2.10, and hostname doesn't have _any_ options. It's just a wrapper around gethostname(), which on some systems gives you a long name and on some a short name. That's why I have my own hostname with the following options: hostname -a list the Aliases hostname -d give the Domain part (after 1st .) hostname -f Full name (short.domain) hostname -s Short name hostname -w Whatever gethostname() gives you hostname same as -w I attach it in case anyone else finds it useful. -------------- next part -------------- A non-text attachment was scrubbed... Name: hostname.c Type: application/octet-stream Size: 3876 bytes Desc: not available URL: -------------- next part -------------- From hokan.stenholm@REDACTED Fri Jun 20 01:56:09 2008 From: hokan.stenholm@REDACTED (=?ISO-8859-1?Q?H=E5kan_Stenholm?=) Date: Fri, 20 Jun 2008 01:56:09 +0200 Subject: [erlang-questions] testing a list against an ets table In-Reply-To: References: Message-ID: <485AF219.9090302@bredband.net> Doug Edmunds wrote: > Scenario: > The goal is determine if any tuples in the List are not > in the Table (and if so, identify which are missing). > > I wrote two functions, one which uses ets:match_object/2 and > the other which uses lists:member/2. > > Function parameters: > List is a list of tuples [{a,b}, {c,d,e}, {a,d}, ...]. > Table is an open ets table of type 'bag', which means > there can be more than one item with the same key in > the table. > > Which of these methods is preferable, and why? > > %% a quick display utility function > io(X) -> io:format("~p~n",[X]). > > findlist1(List,Table) -> > R1 = [E || E <- List, length(ets:match_object(Table, E)) == 0], > io(R1), %% elements not found in table > io(length(R1)). %% if length(R1) == 0, all are in list > I guess that this is something like O(N) or O(N log N), as ets:match_object/2 is fast if the key is defined in the match pattern (E). > findlist2(List, Table) -> > AsList = ets:tab2list(Table), > R1 = [E || E <- List, not lists:member(E,AsList)], > io(R1), %% same as above > io(length(R1)). > This is O(N^2) - it does a O(N) scan over the list AsList for each element in List. There is also a minor O(N) cost of copying the Table into a list (AsList). > length(R1) could be turned into true/false thus: > > case length(R1) of > > 0 -> true; % all in table > _Other -> false > end. > > What are some other alternatives? > You could use sets (sets, ordset, ...) find_tuples_not_in_table(List, Table) -> LSet = sets:from_list(List), TSet = sets:from_list(ets:tab2list(Table)), sets:subtract(LSet, TSet). This should (probably) be O(N log N) to build the sets and O(N) to do the set subtraction - this may depend on the set module used. > > Also, I would appreciate suggestions on > how to short-circuit the evaluation > (stopping as soon as one element of the > list is not found in the table). %% short circuit loop by doing manual List traversal findlist1([],Table) -> true; findlist1([E | List], Table) -> case ets:match_object(Table, E) of [] -> false; %% stop when E not found _ -> findlist1(List, Table) end. %% short circuit using exceptions, this allow the use of standard %% list traversal functions like map, foldl, ... although it's kind %% of ugly to use exceptions for something thats not really a error findlist1(List,Table) -> F = fun(E, Acc) -> case ets:match_object(Table, E) of [] -> throw(end); _ -> Acc end, try lists:foldl(F, true, List) catch throw:end -> false end. > > Thanks. > > -dae > > > ------------------------------------------------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From colm.dougan@REDACTED Fri Jun 20 03:28:21 2008 From: colm.dougan@REDACTED (Colm Dougan) Date: Fri, 20 Jun 2008 02:28:21 +0100 Subject: [erlang-questions] Signal handling (TERM, INT etc) In-Reply-To: References: <24d4f39c0806190905w1d628859x2efcc47b2ca972c4@mail.gmail.com> Message-ID: <24d4f39c0806191828t52e8b2fbvbfd02512ce3990d6@mail.gmail.com> On Thu, Jun 19, 2008 at 5:59 PM, Maxim Treskin wrote: > Hello > > Handling of UNIX-signal is not portable feature through many OS. So, > you must write OS-specified driver for this operations. Thanks - that is the answer I expected. So what do people generally do in situations that normally call for signal handling? For example if I shutdown a machine and all my processes get sent a TERM signal but I don't want my erlang process to get killed mercilessly. What do people normally do? An OS specific driver seems like a relatively difficult option. Do people normally wrap the erlang process in another program/script that knows how to intercept signals and which then sends RPC to the erlang process? I'm just seeking guidance on the standard practice (I already looked at what yaws and ejabberd do on Linux). Thanks, Colm From circularfunc@REDACTED Fri Jun 20 03:13:35 2008 From: circularfunc@REDACTED (Circular Function) Date: Fri, 20 Jun 2008 01:13:35 +0000 (GMT) Subject: [erlang-questions] good project for beginner erlanger? Message-ID: <222076.55508.qm@web28304.mail.ukl.yahoo.com> Im learning Erlang like some of you have noticed perhaps by my earlier post.. I am quite experienced with programming though, having good experience with python and some other languages adn Im not new to functional and celarative programming having used both lisp and prolog. anyone have a good recommendation for an application to write as my first a little bigger project? it shouldnt be huge, but perhaps a few hundred lines of code and really capture the spirit of erlang. Erlang is made for concurrency so something with that. i was thinking a chat-program but im not sure how big that is and it is also hard to test by myself. any tips? ___________________________________________________ S?k efter k?rleken! Hitta din tvillingsj?l p? Yahoo! Dejting: http://ad.doubleclick.net/clk;185753627;24584539;x?http://se.meetic.yahoo.net/index.php?mtcmk=148783 -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlangy@REDACTED Fri Jun 20 03:46:34 2008 From: erlangy@REDACTED (ERLANG) Date: Fri, 20 Jun 2008 03:46:34 +0200 Subject: [erlang-questions] Kilim: Fast, lightweight, cheap message passing in Java In-Reply-To: <485AF219.9090302@bredband.net> References: <485AF219.9090302@bredband.net> Message-ID: <1A07A2A4-2564-4180-A975-572A3B672906@gmail.com> Hi List, I found this recent GoogleTechTalk presentation interesting: http://video.google.com/videoplay?docid=-2327417501531223380&q=erlang+%22Google+Tech+Talks%22+duration%3Along+%28site%3Avideo.google.com+OR+site%3Ayoutube.com%29&hl=fr Crazy, look to the abstract: Kilim: Fast, lightweight, cheap message passing in Java. A million actors, 3x faster than Erlang. The message passing (MP) paradigm is often seen as a superior ... cheers Y. -------------- next part -------------- An HTML attachment was scrubbed... URL: From litaocheng@REDACTED Fri Jun 20 03:49:19 2008 From: litaocheng@REDACTED (=?UTF-8?B?5oiQ56uL5rab?=) Date: Fri, 20 Jun 2008 09:49:19 +0800 Subject: [erlang-questions] Bit syntax decscription in erlang reference manual? Message-ID: In erlang reference mannual, the Bit Syntax Expressions section, about the binary Size, the manual said: "In matching, this default value is only valid for the very last element. All other bit string or binary elements in the matching must have a size specification." we know the default size for integer is 8, float is 64, binary is the whole bytes. for example: > <> = <<1, 2, 3>>. I think in this pattern match, the variables A, B, C all use the dafault Size. Is it a conflict with reference manual? thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlm@REDACTED Fri Jun 20 03:55:18 2008 From: vlm@REDACTED (Lev Walkin) Date: Thu, 19 Jun 2008 18:55:18 -0700 Subject: [erlang-questions] Signal handling (TERM, INT etc) In-Reply-To: <24d4f39c0806191828t52e8b2fbvbfd02512ce3990d6@mail.gmail.com> References: <24d4f39c0806190905w1d628859x2efcc47b2ca972c4@mail.gmail.com> <24d4f39c0806191828t52e8b2fbvbfd02512ce3990d6@mail.gmail.com> Message-ID: <485B0E06.8080804@lionet.info> Colm Dougan wrote: > On Thu, Jun 19, 2008 at 5:59 PM, Maxim Treskin wrote: >> Hello >> >> Handling of UNIX-signal is not portable feature through many OS. So, >> you must write OS-specified driver for this operations. > > Thanks - that is the answer I expected. So what do people generally > do in situations that normally call for signal handling? For example > if I shutdown a machine and all my processes get sent a TERM signal > but I don't want my erlang process to get killed mercilessly. What do > people normally do? An OS specific driver seems like a relatively > difficult option. Do people normally wrap the erlang process in > another program/script that knows how to intercept signals and which > then sends RPC to the erlang process? I'm just seeking guidance on > the standard practice (I already looked at what yaws and ejabberd do > on Linux). You could create a server listening to TCP or UDP port and have a matching script (e.g. `echo "msg" | telnet localhost 1234`) that you can invoke during shutdown to tell Erlang to terminate. Second, you can create a linked-in driver which sends a message back to a corresponding server. This allows you to intercept signals directly instead of relying on some external notificator. -- Lev Walkin vlm@REDACTED From dbyrne@REDACTED Fri Jun 20 04:17:20 2008 From: dbyrne@REDACTED (Dennis Byrne) Date: Thu, 19 Jun 2008 21:17:20 -0500 Subject: [erlang-questions] good project for beginner erlanger? Message-ID: An HTML attachment was scrubbed... URL: From dbyrne@REDACTED Fri Jun 20 04:24:41 2008 From: dbyrne@REDACTED (Dennis Byrne) Date: Thu, 19 Jun 2008 21:24:41 -0500 Subject: [erlang-questions] Kilim: Fast, lightweight, cheap message passing in Java Message-ID: An HTML attachment was scrubbed... URL: From erlang-questions_efine@REDACTED Fri Jun 20 05:04:34 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Thu, 19 Jun 2008 23:04:34 -0400 Subject: [erlang-questions] Bit syntax decscription in erlang reference manual? In-Reply-To: References: Message-ID: <6c2563b20806192004k18a9c45bk2e055da10edc9b55@mail.gmail.com> There's a subtlety here that you may have missed. The manual says All other *bit string or binary* elements in the matching must have a size specification. In <> = <<1,2,3>>, all elements on the left hand side are by default of type integer, not bitstring or binary. Therefore sizes are assumed to be 1 byte each. Your example is the same as writing <> = <<1,2,3>>. This below, for example, would NOT work: <> = <<1,2,3>>. You would have to write <> = <<1,2,3>>. Example: 1> <> = <<1,2,3>>. * 1: a binary field without size is only allowed at the end of a binary pattern 2> <> = <<1,2,3>>. <<1,2,3>> 3> A. <<1>> 4> B. <<2,3>> 5> Hope this helps. 2008/6/19 ??? : > In erlang reference mannual, the Bit Syntax Expressions section, about the > binary Size, the manual said: > "In matching, this default value is only valid for the very last element. > All other bit string or binary elements in the matching must have a size > specification." > we know the default size for integer is 8, float is 64, binary is the whole > bytes. > for example: > > <> = <<1, 2, 3>>. > I think in this pattern match, the variables A, B, C all use the dafault > Size. > Is it a conflict with reference manual? > thanks! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toby@REDACTED Mon Jun 16 00:20:53 2008 From: toby@REDACTED (Toby Thain) Date: Sun, 15 Jun 2008 19:20:53 -0300 Subject: [erlang-questions] Erlang/OTP R12B-3 has been released In-Reply-To: <3dbc6d1c0806121259g2877fb21s96c26730ba09abf8@mail.gmail.com> References: <3dbc6d1c0806121259g2877fb21s96c26730ba09abf8@mail.gmail.com> Message-ID: On 12-Jun-08, at 4:59 PM, Robert Virding wrote: > 2008/6/12 Torbjorn Tornkvist : > > Great work! > > A little warning: there is a risk for confusion here > because the new regexp module has been named 're' which > also is the name of the regexp module that Robert Virding > has put on User Contrib at trapexit.org. > > Cheers, Tobbe > > Yes, I know. I will have to start a name war to get back my name. > Infect R12 with a virus which crashes all using the new built in > module re. Or change the name of my module. :-) > How about 'rv' ? --Toby > Robert > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From litaocheng@REDACTED Fri Jun 20 06:00:08 2008 From: litaocheng@REDACTED (=?UTF-8?B?5oiQ56uL5rab?=) Date: Fri, 20 Jun 2008 12:00:08 +0800 Subject: [erlang-questions] Bit syntax decscription in erlang reference manual? In-Reply-To: <6c2563b20806192004k18a9c45bk2e055da10edc9b55@mail.gmail.com> References: <6c2563b20806192004k18a9c45bk2e055da10edc9b55@mail.gmail.com> Message-ID: yeah. I see. Thank you very much Edwin. my opinion: in binary syntax, the integer, float can always use their default size, but binary(bits and bytes) must declare its size specification except the very last binary element. Is't right? I come from china, so my english is not good, please don't mind my expression. :) 2008/6/20 Edwin Fine : > There's a subtlety here that you may have missed. The manual says > > All other *bit string or binary* elements in the matching must have a size > specification. > > In <> = <<1,2,3>>, all elements on the left hand side are by > default of type integer, not bitstring or binary. Therefore sizes are > assumed to be 1 byte each. > > Your example is the same as writing > > <> = <<1,2,3>>. > > This below, for example, would NOT work: > > <> = <<1,2,3>>. > > You would have to write > > <> = <<1,2,3>>. > > Example: > 1> <> = <<1,2,3>>. > * 1: a binary field without size is only allowed at the end of a binary > pattern > 2> <> = <<1,2,3>>. > <<1,2,3>> > 3> A. > <<1>> > 4> B. > <<2,3>> > 5> > > Hope this helps. > 2008/6/19 ??? : > >> In erlang reference mannual, the Bit Syntax Expressions section, about the >> binary Size, the manual said: >> "In matching, this default value is only valid for the very last element. >> All other bit string or binary elements in the matching must have a size >> specification." >> we know the default size for integer is 8, float is 64, binary is the >> whole bytes. >> for example: >> > <> = <<1, 2, 3>>. >> I think in this pattern match, the variables A, B, C all use the dafault >> Size. >> Is it a conflict with reference manual? >> thanks! >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrickdlogan@REDACTED Fri Jun 20 06:14:53 2008 From: patrickdlogan@REDACTED (Patrick Logan) Date: Thu, 19 Jun 2008 21:14:53 -0700 Subject: [erlang-questions] Kilim: Fast, lightweight, cheap message passing in Java Message-ID: "Any chance you caught whether or not this tool allows shared state? If so, pass by reference would explain 3x claim. Objects are cheap, so that explains the actor counts. But at the end of the day I'd want to know how much context switching was used for these benchmarks before I got excited." After a very preliminary review: * Messages and objects are distinct. * Messages are trees, i.e. no cycles. * Messages are constructed in a "controlled" way to form a tree. * Message state moves from "free" (under construction) to "constant" (constructed, read-only) to "invalid". * Kind of a linear type system ensures the proper construction and "linear" hand offs of messages from actor to actor. * Actors are implemented (I think completely isolated from each other) using a continuation-passing style transformation to be much lighter wieght than java threads. * Message boxes are implemented as a "library" so various designs are possible, the current one is multiple senders and single consumer (enforced only at runtime, throwing an exception in the second consumer, I recall). Looks worth exploring. From patrickdlogan@REDACTED Fri Jun 20 06:23:23 2008 From: patrickdlogan@REDACTED (Patrick Logan) Date: Thu, 19 Jun 2008 21:23:23 -0700 Subject: [erlang-questions] Kilim: Fast, lightweight, cheap message passing in Java Message-ID: "Any chance you caught whether or not this tool allows shared state? If so, pass by reference would explain 3x claim." Yeah, I understand copying is avoided, using the linear types and message types. Apparently there is a "sharable" attribute to allow shared-memory object access on a case by case explicit basis. From toby@REDACTED Sun Jun 15 20:44:10 2008 From: toby@REDACTED (Toby Thain) Date: Sun, 15 Jun 2008 15:44:10 -0300 Subject: [erlang-questions] US DOD Certification of Erlang? In-Reply-To: <00d501c8cb86$15eba4c0$6501a8c0@moneymaker2> References: <00d501c8cb86$15eba4c0$6501a8c0@moneymaker2> Message-ID: <0802F660-6BE1-41DE-BBE1-BE53AFEB95D2@telegraphics.com.au> On 11-Jun-08, at 2:43 AM, Valentin Micic wrote: > IMHO, open source may introduce security risks that might be very > dificult > to quantify. Maybe a sound approach would be to see if they ever > certified > any open-source environment (how's Java doin', for example) and see > if it is > even possible. But I have a strong feeling that DoD would not certify > something that does not have a "Made in U.S.A." tag, woulld > they? ;-) I > mean, just think about it -- if they can run it, so can Russia, > Iran... > where's the competitive advantage? The ludicrousness of this position is clear when one realises that the home-grown product is Windows. :-) --Toby > > V. > > ----- Original Message ----- > From: "Adam Wagner" > To: > Sent: Wednesday, June 11, 2008 3:07 AM > Subject: [erlang-questions] US DOD Certification of Erlang? > > >> I currently work for a US defense contractor, but am relatively >> new to >> government work. From what I've learned so far about Erlang I can >> see >> programs on the horizon where it would be a great fit. Whenever I've >> mentioned the idea to those higher up, I'm told that it would never >> happen because the DOD would never "certify" it. Has anybody else >> gone down this road with Erlang that could offer advice? >> >> Thanks, >> Adam >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From ok@REDACTED Fri Jun 20 06:23:49 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 20 Jun 2008 16:23:49 +1200 Subject: [erlang-questions] Idiomatic Erlang, style/performance question In-Reply-To: <310053.70921.qm@web28314.mail.ukl.yahoo.com> References: <310053.70921.qm@web28314.mail.ukl.yahoo.com> Message-ID: On 19 Jun 2008, at 8:30 am, Circular Function wrote: > Is there a styleguide somewhere on Erlang? "Do what Joe Armstrong does in his book." > Hmm while writing I got a thought. Is the last function tail- > recursive? > I have only heard that term but never used it. > Is tail-recursion faster or uses less memory, what is the advantage/ > disadvantage? I prefer the term "Tail-Call Optimisation" because it is more accurate. The idea (as explained in "Lambda - the Ultimate Goto") is that if the dynamically last thing a function does (in any language) is to return the result of another function call, then there is no point in keeping the current stack frame. You might as well arrange for the new function to return directly to the original caller. Think of *all* tail calls as "argument binding + GOTO". It is always better if a compiler does tail call optimisation (correctly) than if it doesn't. Even the C compiler I normally does it. (It may not be a coincidence that the machine is a SPARC where stack frames are larger than you would think.) It doesn't cost you anything in time, and it's a win in (stack) space. [I am ignoring debugging issues here, which is why Smalltalk doesn't do TCO.] However, there is a *different* question, which is whether writing a function so as to exploit tail call optimisation is always a win over writing in some other fashion. And that has the answer, "no, sometimes one way is better, sometimes the other, depending on what you are doing." The main advantage of TCO is the way it lets you think: control structure = procedure call. You don't have to choose between tail recursion and iteration, because they *are the same thing*. So it's OK for Erlang not to offer a 'while' loop, for example. > > powerx(N, 0) -> 1; > powerx(N, P) when is_integer(P), P > 0 -> > N * powerx(N, P - 1); > powerx(N, P) when is_integer(P), P < 0 -> > 1 / powerx(N, -P). > There are no tail calls here; after the calls to powerx/2 have finished, there is still something for the caller to do. > > powerc(N, P) -> > if > P > 0 -> > N * powerc(N, P - 1); > P < 0 -> > 1 / powerc(N, -1 * P); > P == 0 -> > 1 > end. The only change of any significance here is the PAINFULLY EYE-JARRING indentation by 8 columns. The stack space required is O(N). > > > power(N, P) when is_integer(P), P >= 0 -> > ipower(N, P, 1); > power(N, P) when is_integer(P), P < 0 -> > 1/ipower(N, -P, 1). The first call here is a tail call, the second is not. > > > ipower(0, _, R) -> R; > ipower(N, P, R) -> ipower(N, P-1, R*N). This is equivalent to ipower(N, P, R) { while (P != 0) P--, R *= N; return R; } The amount of stack space required is O(1). From rapsey@REDACTED Fri Jun 20 06:35:43 2008 From: rapsey@REDACTED (Rapsey) Date: Fri, 20 Jun 2008 06:35:43 +0200 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <399E231E-E06E-4FCE-901A-24C6B7B2DFA8@udc.es> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> <399E231E-E06E-4FCE-901A-24C6B7B2DFA8@udc.es> Message-ID: <97619b170806192135r5504e3a4s80179885510971d9@mail.gmail.com> All data goes through nginx which acts as a proxy. Its CPU consumption is never over 1%. Sergej On Thu, Jun 19, 2008 at 9:35 PM, Javier Par?s Fern?ndez wrote: > > El 19/06/2008, a las 20:06, Rapsey escribi?: > > It loops from another module, that way I can update the code at any time >> without disrupting anything. >> The packets are generally a few hundred bytes big, except keyframes which >> tend to be in the kB range. I haven't tried looking with wireshark. Still >> it seems a bit odd that a large CPU consumption would be the symptom. The >> traffic is strictly one way. Either someone is sending the stream or >> receiving it. >> The transmit could of course be written with a passive receive, but the >> code would be significantly uglier. I'm sure someone here knows if setting >> {active, once} every packet is CPU intensive or not. >> It seems the workings of gen_tcp is quite platform dependent. If I run the >> code in windows, sending more than 128 bytes per gen_tcp call significantly >> decreases network output. >> Oh and I forgot to mention I use R12B-3. >> > > Hi, > > Without being an expert. > > 200-300 mb/s in small (hundreds of bytes) packets means a *lot* of system > calls if you are doing a gen_tcp:send for each one. If you buffer 3 packets, > you are reducing that by a factor of 3 :). I'd try to do an small test doing > the same thing in C and compare the results. I think it will also eat a lot > of CPU. > > About the proxy CPU... I'm a bit lost about it, but speculating wildly it > is possible that the time spent doing the system calls that gen_tcp is doing > is added to the proxy CPU process. > > Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From devdoer2@REDACTED Fri Jun 20 07:23:06 2008 From: devdoer2@REDACTED (devdoer bird) Date: Fri, 20 Jun 2008 13:23:06 +0800 Subject: [erlang-questions] Can one mnesia table fragment be further fragmented? In-Reply-To: References: <62588.1213834893@snookles.snookles.com> Message-ID: Thanks ,Paul. I decide to replace the mnesia_frag_hash with my own hash module .With this customized hash module,I hope I can explicitly pass a fragment number to make the fragment splited. I decide to use a mnesia table or other persitent-able data structure in erlang which can globally seen by all nodes to store the mapping relations between key value range and fragment(Eg. [1..999]-> frag 1,[1000-1999]->frag2), so when I need to split a specifed fragment ,I can rejust the mapping relation table. Here's my plan,any suggestions are welcome. 2008/6/19, Paul Mineiro : > > Re: picking what gets split, not with the default frag_hash. You can see > what it's doing in mnesia_frag_hash.erl ... it basically splits the > fragments in order. This means that unless there is a power-of-two number > of fragments they will be of different magnitudes. You could leverage > that for your load balancing by placing bigger fragments on less noded > nodes, etc. In fact, maybe this is why you are seeing > an uneven distribution to begin with? > > -- p > > p.z. If you are feeling your wheaties, you can make your own frag hash, > and specify the module in the frag_properties when you create the table. > > http://www.erlang.org/doc/apps/mnesia/Mnesia_chap5.html#5.3 > > Don't be tempted to just use phash2 or something like that; in order to go > from N fragments to M fragments you have to pass through all the > intermediate stages N + 1, ..., M - 1, M ; you do not want to rehash all > the data in all the buckets at each step. > > On Thu, 19 Jun 2008, devdoer bird wrote: > > > Thanks. > > > > Thant means ,I need fragment table many times not for load balance but > for > > decreasing one specific node's (the node is not as powerful as others ) > > load. > > > > Eg. Node A has 2 fragments,Node B has 2 fragments. Each fragment has > equal > > records(say 100 records each fragment). > > > > Now the Node B is heavily loaded.If I adding one or two fragments in > Node > > A , the load of Node B won't be > > > > decreased ,for the records from Node A will be splited and moved to the > new > > fragments,the records on Node A is 50+50+50+50=200 ,the total number of > > Node B's record is still 200. > > > > In order to decreas the load on Node B,I have to calcaulate carefully how > > many new fragments should be added. > > > > If there's a way for mnesia to specify which node's fragment should be > > splited when addin a new fragment,the problems above will be solved.Take > the > > example above ,when adding a new fragment,I can tell mnesia to split > Node > > B's fragment. > > > > 2008/6/19, Scott Lystig Fritchie : > > > > > > devdoer bird wrote: > > > > > > dd> I spilit one mnesia table into 3 fragments which distributed on 3 > > > dd> machines named A,B and C,but now the the machine "A" is very busy, > > > > > > dd> so I decide split the fragment on the machine "A" into small > > > dd> fragments.Does mnesia support this? > > > > > > No, that isn't possible with Mnesia, not as you wish/state above. > > > > > > If you have a table 'tab' split into fragments 'tab', 'tab_frag2', and > > > 'tab_frag3', you can add more fragments: 'tab_frag4', 'tab_frag5', etc. > > > You can choose to put only 1 fragment on node A, 4 fragments on node B, > > > and 4 fragments on node C. Assuming that access to the table is evenly > > > distributed over all fragments, then node A should be getting only 1/9 > > > of the load, and B & C will get 4/9 of the load. > > > > > > -Scott > > > > > > > In an artificial world, only extremists live naturally. > > -- Paul Graham > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang-questions_efine@REDACTED Fri Jun 20 08:04:39 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Fri, 20 Jun 2008 02:04:39 -0400 Subject: [erlang-questions] Bit syntax decscription in erlang reference manual? In-Reply-To: References: <6c2563b20806192004k18a9c45bk2e055da10edc9b55@mail.gmail.com> Message-ID: <6c2563b20806192304l7d10d4c4u6da7f4c42a05113e@mail.gmail.com> Sounds right to me. One little thing, though: the newer versions of Erlang support a type, "bytes", which is identical to binary, but a more accurate description. There is also a size_bytes() function. As I understand it, we are encouraged to phase out the use of binary in favor of bytes. So the example could have been written with a more "modern" syntax as: <> = <<1,2,3>>. S = byte_size(B). Example: 10> <> = <<1,2,3>>. <<1,2,3>> 11> S = byte_size(B). 2 12> A. <<1>> 13> B. <<2,3>> Hope this helps. On Fri, Jun 20, 2008 at 12:00 AM, ??? wrote: > yeah. I see. Thank you very much Edwin. > my opinion: in binary syntax, the integer, float can always use their > default size, but binary(bits and bytes) must declare its size specification > except the very last binary element. Is't right? > I come from china, so my english is not good, please don't mind my > expression. :) > > 2008/6/20 Edwin Fine : > > There's a subtlety here that you may have missed. The manual says >> >> All other *bit string or binary* elements in the matching must have a size >> specification. >> >> In <> = <<1,2,3>>, all elements on the left hand side are by >> default of type integer, not bitstring or binary. Therefore sizes are >> assumed to be 1 byte each. >> >> Your example is the same as writing >> >> <> = <<1,2,3>>. >> >> This below, for example, would NOT work: >> >> <> = <<1,2,3>>. >> >> You would have to write >> >> <> = <<1,2,3>>. >> >> Example: >> 1> <> = <<1,2,3>>. >> * 1: a binary field without size is only allowed at the end of a binary >> pattern >> 2> <> = <<1,2,3>>. >> <<1,2,3>> >> 3> A. >> <<1>> >> 4> B. >> <<2,3>> >> 5> >> >> Hope this helps. >> 2008/6/19 ??? : >> >>> In erlang reference mannual, the Bit Syntax Expressions section, about >>> the binary Size, the manual said: >>> "In matching, this default value is only valid for the very last element. >>> All other bit string or binary elements in the matching must have a size >>> specification." >>> we know the default size for integer is 8, float is 64, binary is the >>> whole bytes. >>> for example: >>> > <> = <<1, 2, 3>>. >>> I think in this pattern match, the variables A, B, C all use the dafault >>> Size. >>> Is it a conflict with reference manual? >>> thanks! >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang-questions_efine@REDACTED Fri Jun 20 08:24:05 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Fri, 20 Jun 2008 02:24:05 -0400 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> Message-ID: <6c2563b20806192324s250d5e35wbc6bfb518bde821e@mail.gmail.com> Which Erlang command-line options are you using? Specifically, are you using -K true and the -A flags? Does OS/X support kernel poll (-K true)? I saw benchmarks where CPU usage without kernel poll was high (60 - 80%), and without it was much lower (5 - 10%). I wouldn't necessarily agree that "the workings of gen_tcp is quite platform dependent." I would rather guess that TCP/IP stacks, and TCP/IP parameters, are very different across certain operating systems. The default values are often not even close to optimal. There are numerous registry tweaks to improve Windows TCP/IP performance, for example. I am surprised that you are forced to send only 128 bytes at a time or face lower performance in Erlang on Windows. That seems odd indeed. I would be taking looks at default buffer sizes and the registry hacks that are findable on Google, and then experiment. I was able to improve performance of an application I am working on from 3 message/sec to 70 msgs/sec simply by spawning a function (to gen_tcp:send the data) that was previously being called sequentially. This was because TCP/IP could now pack multiple packets into the same frame, which previously only had one packet in it. The RTT of the link was dreadful (290ms), so this was a bit of a special case but I think the principle remains the same. Transmitting data in fewer packets means fewer system calls, better utilization of available frame space, and less CPU. Plus using -K true and perhaps +A 128 should improve things. Give it a try (if you already haven't) and see if it improves things. Also take a look if you will at Boost socket performance on Linux, which has some interesting information on this topic. 2008/6/19 Rapsey : > It loops from another module, that way I can update the code at any time > without disrupting anything. > The packets are generally a few hundred bytes big, except keyframes which > tend to be in the kB range. I haven't tried looking with wireshark. Still > it seems a bit odd that a large CPU consumption would be the symptom. The > traffic is strictly one way. Either someone is sending the stream or > receiving it. > The transmit could of course be written with a passive receive, but the > code would be significantly uglier. I'm sure someone here knows if setting > {active, once} every packet is CPU intensive or not. > It seems the workings of gen_tcp is quite platform dependent. If I run the > code in windows, sending more than 128 bytes per gen_tcp call significantly > decreases network output. > Oh and I forgot to mention I use R12B-3. > > > Sergej > > > On Thu, Jun 19, 2008 at 7:39 PM, Edwin Fine < > erlang-questions_efine@REDACTED> wrote: > >> How large is each packet? Can multiple packets fit into one TCP window? >> Have you looked at the TCP/IP wire-level data with Wireshark/Ethereal to see >> if the packets are being combined at the TCP level? If you see that you are >> only getting one packet per TCP frame (assuming a packet is much smaller >> than the window size), you might be falling foul of the Nagle congestion >> algorithm. The fact that manually buffering your packets improves >> performance suggests this may be the case. Nagle says that send, send, send >> is OK, receive, receive, receive is ok, and even send, receive, send, >> receive is ok, but you get into trouble if sends and receives are mixed >> asymmetrically on the same socket (e.g. send, send, receive). >> >> Also, I don't understand your transmit_loop. Where is it looping (or am I >> misunderstanding something)? >> >> From what I have seen, people writing Erlang TCP/IP code do an {active, >> once} receive, and when getting the first packet, drop into another loop >> that does a passive receive until there's no data waiting, then go back into >> the {active, once} receive. Are you doing this? I am not sure, but I fear >> that if all your receives are {active, once} it will incur more CPU overhead >> than the active/passive split. It's hard to know because I can't see enough >> of your code to know what you are doing overall. Disclaimer: I'm no Erlang >> or TCP/IP expert. >> >> Hope this helps. >> >> 2008/6/19 Rapsey : >> >>> I have a streaming server written in Erlang. When it was pushing 200-300 >>> mb/s the CPU was getting completely hammered. I traced the problem to >>> gen_tcp:send. >>> So instead of sending every audio/video packet with a single gen_tcp:send >>> call, I buffer 3 packets and then send them all at once. CPU consumption >>> dropped dramatically. >>> On one of the servers I have a simple proxy, the main process that sends >>> packets between the client and some other server looks like this: >>> >>> transmit_loop({tcp, Sock, Data}, P) when P#transdat.client == Sock -> >>> gen_tcp:send(P#transdat.server, Data), >>> inet:setopts(P#transdat.client, [{active, once}]), >>> {ok, P}; >>> transmit_loop({tcp, Sock, Data}, P) when P#transdat.server == Sock -> >>> gen_tcp:send(P#transdat.client, Data), >>> inet:setopts(P#transdat.server, [{active, once}]), >>> {ok, P}; >>> transmit_loop({start, ServerPort}, P) -> >>> {ok, Sock} = gen_tcp:connect("127.0.0.1", ServerPort, [binary, >>> {active, once}, {packet, 0}]), >>> {ok, P#transdat{server = Sock}}; >>> transmit_loop({tcp_closed, _}, _) -> >>> exit(stop). >>> >>> The proxy is eating more CPU time than the streaming server. >>> Is this normal behavior? The server is running OSX 10.4 >>> >>> >>> Sergej >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From w.a.de.jong@REDACTED Fri Jun 20 08:24:11 2008 From: w.a.de.jong@REDACTED (Willem de Jong) Date: Fri, 20 Jun 2008 08:24:11 +0200 Subject: [erlang-questions] Erlsom it's so close to being SAXy In-Reply-To: <485ACE7F.901@lummie.co.uk> References: <485ACE7F.901@lummie.co.uk> Message-ID: <407d9ef80806192324x47c9800cp5dd1c853a4c6f2cd@mail.gmail.com> Hello Matt, Yes, this is possible (and documented, but maybe not clear enough - are you using the latest version? Did you get erlsom from CEAN? That version is quite old). There even is an example included in the distribution that shows how to do it. I am copying part of it below. Good luck, Willem %% Example to show how the Erlsom Sax parser can be used in combination %% with a 'continuation function'. This enables parsing of very big documents %% in a sort of streaming mode. %% %% When the sax parser reaches the end of a block of data, it calls the %% continuation function. This should return the next block of data. %% %% the continuation function is a function that takes 2 arguments: Tail and %% State. %% - Tail is the (short) list of characters that could not yet be parsed %% because it might be a special token or not. Since this still has to %% be parsed, it should be put in front of the next block of data. %% - State is information that is passed by the parser to the callback %% functions transparently. This can be used to keep track of the %% location in the file etc. %% The function returns {NewData, NewState}, where NewData is a list of %% characters/unicode code points, and NewState the new value for the State. -export([run/0]). %% 'chunk' is the number of characters that is read at a time. %% should be tuned for the best result. (109 is obviously not a good value, %% it should be bigger than that - try it out). -define(chunk, 109). run() -> F = fun count_books/2, %% the callback function that handles the sax events G = fun continue_file/2, %% the callback function that returns the next %% chunk of data %% open file {ok, Handle} = file:open(xml(), [read, raw, binary]), Position = 0, CState = {Handle, Position, ?chunk}, SaxCallbackState = undefined, %% erlsom:parse_sax() returns {ok, FinalState, TrailingBytes}, %% where TrailingBytes is the rest of the input-document %% that follows after the last closing tag of the XML, and Result %% is the value of the State after processing the last SAX event. {ok, Result, _TrailingBytes} = erlsom:parse_sax(<<>>, SaxCallbackState, F, [{continuation_function, G, CState}]), %% close file ok = file:close(Handle), %% Result is a list [{track_id, count}, ...] lists:foreach(fun({Date, Count}) -> io:format("Date: ~p - count: ~p~n", [Date, Count]) end, Result), ok. %% this is a continuation function that reads chunks of data %% from a file. continue_file(Tail, {Handle, Offset, Chunk}) -> %% read the next chunk case file:pread(Handle, Offset, Chunk) of {ok, Data} -> {<>, {Handle, Offset + Chunk, Chunk}}; eof -> {Tail, {Handle, Offset, Chunk}} end. count_books(startDocument, _) -> etc... On 6/19/08, Matt Harrison wrote: > > See what I did there with the subject title :) > > All > > Has anyone used Erlsom for sax parsing straight from a file, i.e without > using file:read_file to load a whole file. I can't seem to find an > appropriate call, and the docs don't seem to cover it. > > I have large files 6gb+ that I need to SAX parse, the main requirement > being able to parse them with about 1gb of memory. > > Erlsom works a treat with files that are a few 100Mb but requires the > whole file loaded in memory, which kinda kills the main benefit of sax > parsing in my opinion. > > I am not especially bothered about speed, (it was suggested that I look > at c parsers linked into erlang) as this is for a data import process > that will only happen rarely (mainly for development and testing purposes). > > I still don't seem to be able to find any xmerl_eventp examples so if > you have one please let me know. > > regards and thanks, > > Matt > > I haven't discarded the using a c library I'm just fairly new to erlang > and would prefer an erlang solution if possible as I don't want to > venture into ports quite yet :) > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rapsey@REDACTED Fri Jun 20 09:52:50 2008 From: rapsey@REDACTED (Rapsey) Date: Fri, 20 Jun 2008 09:52:50 +0200 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <6c2563b20806192324s250d5e35wbc6bfb518bde821e@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> <6c2563b20806192324s250d5e35wbc6bfb518bde821e@mail.gmail.com> Message-ID: <97619b170806200052g1887e1cbwaa62cb01bcb6cef@mail.gmail.com> I have kernel-poll enabled (osx supports it). Don't have async threads enabled, because I don't know what it does. I'll try it. Sergej On Fri, Jun 20, 2008 at 8:24 AM, Edwin Fine wrote: > Which Erlang command-line options are you using? Specifically, are you > using -K true and the -A flags? Does OS/X support kernel poll (-K true)? I > saw benchmarks where CPU usage without kernel poll was high (60 - 80%), and > without it was much lower (5 - 10%). > > I wouldn't necessarily agree that "the workings of gen_tcp is quite > platform dependent." I would rather guess that TCP/IP stacks, and TCP/IP > parameters, are very different across certain operating systems. The default > values are often not even close to optimal. There are numerous registry > tweaks to improve Windows TCP/IP performance, for example. I am surprised > that you are forced to send only 128 bytes at a time or face lower > performance in Erlang on Windows. That seems odd indeed. I would be taking > looks at default buffer sizes and the registry hacks that are findable on > Google, and then experiment. > > I was able to improve performance of an application I am working on from 3 > message/sec to 70 msgs/sec simply by spawning a function (to gen_tcp:send > the data) that was previously being called sequentially. This was because > TCP/IP could now pack multiple packets into the same frame, which previously > only had one packet in it. The RTT of the link was dreadful (290ms), so this > was a bit of a special case but I think the principle remains the same. > Transmitting data in fewer packets means fewer system calls, better > utilization of available frame space, and less CPU. Plus using -K true and > perhaps +A 128 should improve things. > > Give it a try (if you already haven't) and see if it improves things. Also > take a look if you will at Boost socket performance on Linux, > which has some interesting information on this topic. > > > 2008/6/19 Rapsey : > >> It loops from another module, that way I can update the code at any time >> without disrupting anything. >> The packets are generally a few hundred bytes big, except keyframes which >> tend to be in the kB range. I haven't tried looking with wireshark. Still >> it seems a bit odd that a large CPU consumption would be the symptom. The >> traffic is strictly one way. Either someone is sending the stream or >> receiving it. >> The transmit could of course be written with a passive receive, but the >> code would be significantly uglier. I'm sure someone here knows if setting >> {active, once} every packet is CPU intensive or not. >> It seems the workings of gen_tcp is quite platform dependent. If I run the >> code in windows, sending more than 128 bytes per gen_tcp call significantly >> decreases network output. >> Oh and I forgot to mention I use R12B-3. >> >> >> Sergej >> >> >> On Thu, Jun 19, 2008 at 7:39 PM, Edwin Fine < >> erlang-questions_efine@REDACTED> wrote: >> >>> How large is each packet? Can multiple packets fit into one TCP window? >>> Have you looked at the TCP/IP wire-level data with Wireshark/Ethereal to see >>> if the packets are being combined at the TCP level? If you see that you are >>> only getting one packet per TCP frame (assuming a packet is much smaller >>> than the window size), you might be falling foul of the Nagle congestion >>> algorithm. The fact that manually buffering your packets improves >>> performance suggests this may be the case. Nagle says that send, send, send >>> is OK, receive, receive, receive is ok, and even send, receive, send, >>> receive is ok, but you get into trouble if sends and receives are mixed >>> asymmetrically on the same socket (e.g. send, send, receive). >>> >>> Also, I don't understand your transmit_loop. Where is it looping (or am I >>> misunderstanding something)? >>> >>> From what I have seen, people writing Erlang TCP/IP code do an {active, >>> once} receive, and when getting the first packet, drop into another loop >>> that does a passive receive until there's no data waiting, then go back into >>> the {active, once} receive. Are you doing this? I am not sure, but I fear >>> that if all your receives are {active, once} it will incur more CPU overhead >>> than the active/passive split. It's hard to know because I can't see enough >>> of your code to know what you are doing overall. Disclaimer: I'm no Erlang >>> or TCP/IP expert. >>> >>> Hope this helps. >>> >>> 2008/6/19 Rapsey : >>> >>>> I have a streaming server written in Erlang. When it was pushing >>>> 200-300 mb/s the CPU was getting completely hammered. I traced the problem >>>> to gen_tcp:send. >>>> So instead of sending every audio/video packet with a single >>>> gen_tcp:send call, I buffer 3 packets and then send them all at once. CPU >>>> consumption dropped dramatically. >>>> On one of the servers I have a simple proxy, the main process that sends >>>> packets between the client and some other server looks like this: >>>> >>>> transmit_loop({tcp, Sock, Data}, P) when P#transdat.client == Sock -> >>>> gen_tcp:send(P#transdat.server, Data), >>>> inet:setopts(P#transdat.client, [{active, once}]), >>>> {ok, P}; >>>> transmit_loop({tcp, Sock, Data}, P) when P#transdat.server == Sock -> >>>> gen_tcp:send(P#transdat.client, Data), >>>> inet:setopts(P#transdat.server, [{active, once}]), >>>> {ok, P}; >>>> transmit_loop({start, ServerPort}, P) -> >>>> {ok, Sock} = gen_tcp:connect("127.0.0.1", ServerPort, [binary, >>>> {active, once}, {packet, 0}]), >>>> {ok, P#transdat{server = Sock}}; >>>> transmit_loop({tcp_closed, _}, _) -> >>>> exit(stop). >>>> >>>> The proxy is eating more CPU time than the streaming server. >>>> Is this normal behavior? The server is running OSX 10.4 >>>> >>>> >>>> Sergej >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Fri Jun 20 10:16:08 2008 From: erlang@REDACTED (Dominic Williams) Date: Fri, 20 Jun 2008 04:16:08 -0400 (EDT) Subject: [erlang-questions] Idiomatic Erlang, style/performance question In-Reply-To: References: <310053.70921.qm@web28314.mail.ukl.yahoo.com> Message-ID: <963b309aaaba5312158788bd42b4cab5.squirrel@www.geekisp.com> Hello, ROK wrote: > I prefer the term "Tail-Call Optimisation" because it is more accurate. > The idea (as explained in "Lambda - the Ultimate Goto") is that if the > dynamically last thing a function does (in any language) is to return > the result of another function call, then there is no point in keeping > the current stack frame. You might as well arrange for the new > function to return directly to the original caller. Think of *all* > tail calls as "argument binding + GOTO". Is the optimisation possible even when the tail call is not to the same function? E.g.: >> power(N, P) when is_integer(P), P >= 0 -> >> ipower(N, P, 1); >> power(N, P) when is_integer(P), P < 0 -> >> 1/ipower(N, -P, 1). > > The first call here is a tail call, the second is not. As I wrote in a previous post, I thought that there was no optimisation here, because it is not a /recursive/ tail-call. Am I wrong? What is the case for Erlang, in particular? Thanks, Dominic Williams http://dominicwilliams.net From torben.lehoff@REDACTED Fri Jun 20 10:25:19 2008 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Fri, 20 Jun 2008 10:25:19 +0200 Subject: [erlang-questions] good project for beginner erlanger? In-Reply-To: <222076.55508.qm@web28304.mail.ukl.yahoo.com> References: <222076.55508.qm@web28304.mail.ukl.yahoo.com> Message-ID: 1. Buy Joe's book. 2. Wait for it to arrive. 3. Run through all the examples. 4. Extend one of the examples with some cool functionality (TM). 5. See where you stand and decide on the path ahead. The intention behind this approach is to get into the thinking without having to start from scratch which I from personal experience have found slightly painful. At some point you may want to consider a training class on Erlang/OTP since that is a beast with a relatively steep learning curve. Cheers, Torben 2008/6/20 Circular Function : > Im learning Erlang like some of you have noticed perhaps by my earlier > post. > I am quite experienced with programming though, having good experience with > python and some other languages adn Im not new to functional and celarative > programming having used both lisp and prolog. > > anyone have a good recommendation for an application to write as my first a > little bigger project? > > it shouldnt be huge, but perhaps a few hundred lines of code and really > capture the spirit of erlang. > > Erlang is made for concurrency so something with that. i was thinking a > chat-program but im not sure how big that is and it is also hard to test by > myself. > > any tips? > ------------------------------ > G?r det l?ngsamt? Skaffa dig en snabbare bredbandsuppkoppling.. > S?k och j?mf?r hos Yahoo! Shopping. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlm@REDACTED Fri Jun 20 10:35:25 2008 From: vlm@REDACTED (Lev Walkin) Date: Fri, 20 Jun 2008 01:35:25 -0700 Subject: [erlang-questions] Idiomatic Erlang, style/performance question In-Reply-To: <963b309aaaba5312158788bd42b4cab5.squirrel@www.geekisp.com> References: <310053.70921.qm@web28314.mail.ukl.yahoo.com> <963b309aaaba5312158788bd42b4cab5.squirrel@www.geekisp.com> Message-ID: <485B6BCD.7070607@lionet.info> Dominic Williams wrote: > Hello, > > ROK wrote: >> I prefer the term "Tail-Call Optimisation" because it is more accurate. >> The idea (as explained in "Lambda - the Ultimate Goto") is that if the >> dynamically last thing a function does (in any language) is to return >> the result of another function call, then there is no point in keeping >> the current stack frame. You might as well arrange for the new >> function to return directly to the original caller. Think of *all* >> tail calls as "argument binding + GOTO". > > Is the optimisation possible even when the tail call is not to the same > function? Yes. > E.g.: > >>> power(N, P) when is_integer(P), P >= 0 -> >>> ipower(N, P, 1); >>> power(N, P) when is_integer(P), P < 0 -> >>> 1/ipower(N, -P, 1). >> The first call here is a tail call, the second is not. > > As I wrote in a previous post, I thought that there was no optimisation > here, because it is not a /recursive/ tail-call. Am I wrong? What is the > case for Erlang, in particular? Erlang optimizes tail calls, not necessarily recursive ones. -- Lev Walkin vlm@REDACTED From litaocheng@REDACTED Fri Jun 20 11:27:41 2008 From: litaocheng@REDACTED (=?UTF-8?B?5oiQ56uL5rab?=) Date: Fri, 20 Jun 2008 17:27:41 +0800 Subject: [erlang-questions] Bit syntax decscription in erlang reference manual? In-Reply-To: <6c2563b20806192304l7d10d4c4u6da7f4c42a05113e@mail.gmail.com> References: <6c2563b20806192004k18a9c45bk2e055da10edc9b55@mail.gmail.com> <6c2563b20806192304l7d10d4c4u6da7f4c42a05113e@mail.gmail.com> Message-ID: thank you! :) 2008/6/20 Edwin Fine : > Sounds right to me. > > One little thing, though: the newer versions of Erlang support a type, > "bytes", which is identical to binary, but a more accurate description. > There is also a size_bytes() function. As I understand it, we are encouraged > to phase out the use of binary in favor of bytes. > > So the example could have been written with a more "modern" syntax as: > > <> = <<1,2,3>>. > S = byte_size(B). > > Example: > 10> <> = <<1,2,3>>. > <<1,2,3>> > 11> S = byte_size(B). > 2 > 12> A. > <<1>> > 13> B. > <<2,3>> > > Hope this helps. > > > On Fri, Jun 20, 2008 at 12:00 AM, ??? wrote: > >> yeah. I see. Thank you very much Edwin. >> my opinion: in binary syntax, the integer, float can always use their >> default size, but binary(bits and bytes) must declare its size specification >> except the very last binary element. Is't right? >> I come from china, so my english is not good, please don't mind my >> expression. :) >> >> 2008/6/20 Edwin Fine : >> >> There's a subtlety here that you may have missed. The manual says >>> >>> All other *bit string or binary* elements in the matching must have a >>> size specification. >>> >>> In <> = <<1,2,3>>, all elements on the left hand side are by >>> default of type integer, not bitstring or binary. Therefore sizes are >>> assumed to be 1 byte each. >>> >>> Your example is the same as writing >>> >>> <> = <<1,2,3>>. >>> >>> This below, for example, would NOT work: >>> >>> <> = <<1,2,3>>. >>> >>> You would have to write >>> >>> <> = <<1,2,3>>. >>> >>> Example: >>> 1> <> = <<1,2,3>>. >>> * 1: a binary field without size is only allowed at the end of a binary >>> pattern >>> 2> <> = <<1,2,3>>. >>> <<1,2,3>> >>> 3> A. >>> <<1>> >>> 4> B. >>> <<2,3>> >>> 5> >>> >>> Hope this helps. >>> 2008/6/19 ??? : >>> >>>> In erlang reference mannual, the Bit Syntax Expressions section, about >>>> the binary Size, the manual said: >>>> "In matching, this default value is only valid for the very last >>>> element. All other bit string or binary elements in the matching must have a >>>> size specification." >>>> we know the default size for integer is 8, float is 64, binary is the >>>> whole bytes. >>>> for example: >>>> > <> = <<1, 2, 3>>. >>>> I think in this pattern match, the variables A, B, C all use the dafault >>>> Size. >>>> Is it a conflict with reference manual? >>>> thanks! >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Fri Jun 20 11:33:35 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Fri, 20 Jun 2008 11:33:35 +0200 Subject: [erlang-questions] testing a list against an ets table In-Reply-To: References: Message-ID: <4d08db370806200233l36ab8cd0sc41755d60e111956@mail.gmail.com> It depends of ets table content, what is best in "absolute" time. For short sets using lists:member/2 can be fastest. For big data where amount of tuples with same first element is low, using ets:match_object/2 should be fastest. But for worst case, where there is big amount of tuples with same first elemnt, best should be transform ets talbe from bag to set where each item will be transformed by function (Tuple) -> {Tuple}. And time will be O(N). (But with big constant factor.) check_list(List, Table) -> Set = ets:new(set, [set, private]), true = ets:foldl(fun(X, _) -> true = ets:insert(Set, {X}) end, true, Table), try lists:foreach(fun(X) -> case ets:member(Set, {X}) of true -> ok; false-> throw(not_found) end end, List) of _ -> true catch throw:not_found -> false after ets:delete(Set) end. 2008/6/20 Doug Edmunds : > Scenario: > The goal is determine if any tuples in the List are not > in the Table (and if so, identify which are missing). > > I wrote two functions, one which uses ets:match_object/2 and > the other which uses lists:member/2. > > Function parameters: > List is a list of tuples [{a,b}, {c,d,e}, {a,d}, ...]. > Table is an open ets table of type 'bag', which means > there can be more than one item with the same key in > the table. > > Which of these methods is preferable, and why? > > %% a quick display utility function > io(X) -> io:format("~p~n",[X]). > > findlist1(List,Table) -> > R1 = [E || E <- List, length(ets:match_object(Table, E)) == 0], > io(R1), %% elements not found in table > io(length(R1)). %% if length(R1) == 0, all are in list > > findlist2(List, Table) -> > AsList = ets:tab2list(Table), > R1 = [E || E <- List, not lists:member(E,AsList)], > io(R1), %% same as above > io(length(R1)). > > length(R1) could be turned into true/false thus: > > case length(R1) of > > 0 -> true; % all in table > _Other -> false > end. > > What are some other alternatives? > > Also, I would appreciate suggestions on > how to short-circuit the evaluation > (stopping as soon as one element of the > list is not found in the table). > > Thanks. > > -dae > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From cthulahoops@REDACTED Fri Jun 20 12:04:13 2008 From: cthulahoops@REDACTED (Adam Kelly) Date: Fri, 20 Jun 2008 11:04:13 +0100 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> Message-ID: <8d1798e90806200304s78e12efbt2096eaa54ab3a7a9@mail.gmail.com> 2008/6/19 Rapsey : > The transmit could of course be written with a passive receive, but the code > would be significantly uglier. I'm sure someone here knows if setting > {active, once} every packet is CPU intensive or not. Is the streaming process building up a message queue (a back log of messages waiting to be sent?). gen_tcp:send does a selective receive in order to acknowledge that the packet has been sent, so if the process has a message queue it has to do a linear search of the message queue on each send and the streaming becomes O(N^2). Adam. From rapsey@REDACTED Fri Jun 20 12:38:55 2008 From: rapsey@REDACTED (Rapsey) Date: Fri, 20 Jun 2008 12:38:55 +0200 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <8d1798e90806200304s78e12efbt2096eaa54ab3a7a9@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> <8d1798e90806200304s78e12efbt2096eaa54ab3a7a9@mail.gmail.com> Message-ID: <97619b170806200338g2092af8aiabf16b00c888153a@mail.gmail.com> The proxy is not building up a queue, because {active, once} is set after every packet has been sent to the other side. Streaming server keeps a close eye on the message queue and it does not allow it to grow by much. On Fri, Jun 20, 2008 at 12:04 PM, Adam Kelly wrote: > 2008/6/19 Rapsey : > > The transmit could of course be written with a passive receive, but the > code > > would be significantly uglier. I'm sure someone here knows if setting > > {active, once} every packet is CPU intensive or not. > > Is the streaming process building up a message queue (a back log of > messages waiting > to be sent?). gen_tcp:send does a selective receive in order to > acknowledge that the > packet has been sent, so if the process has a message queue it has to > do a linear search > of the message queue on each send and the streaming becomes O(N^2). > > Adam. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mazen@REDACTED Fri Jun 20 16:13:21 2008 From: mazen@REDACTED (Mazen Harake) Date: Fri, 20 Jun 2008 15:13:21 +0100 Subject: [erlang-questions] good project for beginner erlanger? In-Reply-To: <222076.55508.qm@web28304.mail.ukl.yahoo.com> References: <222076.55508.qm@web28304.mail.ukl.yahoo.com> Message-ID: <485BBB01.7050706@erlang-consulting.com> (re)write yet another web server... :) /Mazen Circular Function wrote: > Im learning Erlang like some of you have noticed perhaps by my earlier > post. > I am quite experienced with programming though, having good experience > with python and some other languages adn Im not new to functional and > celarative programming having used both lisp and prolog. > > anyone have a good recommendation for an application to write as my > first a little bigger project? > > it shouldnt be huge, but perhaps a few hundred lines of code and > really capture the spirit of erlang. > > Erlang is made for concurrency so something with that. i was thinking > a chat-program but im not sure how big that is and it is also hard to > test by myself. > > any tips? > > > ------------------------------------------------------------------------ > G?r det l?ngsamt? Skaffa dig en snabbare bredbandsuppkoppling.. > S?k och j?mf?r hos Yahoo! Shopping. > > ------------------------------------------------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions -- Mazen Harake Erlang Software Developer and Consultant, Erlang Training & Consulting, Ltd Mobile Phone: +44 (0)795 13 26 317 Office Phone: +44 (0)207 45 61 020 Office Address: 401 London Fruit & Wool Exchange Brushfield St, London, E1 6EL United Kingdom This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of "Erlang Training & Consulting, Ltd". If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone. Please contact the sender if you believe you have received this email in error. From gleber.p@REDACTED Fri Jun 20 16:38:55 2008 From: gleber.p@REDACTED (Gleb Peregud) Date: Fri, 20 Jun 2008 16:38:55 +0200 Subject: [erlang-questions] cacherl memcached app startup problem question In-Reply-To: <1218d6a50805301550u50ddc883l4e07607c2e70f760@mail.gmail.com> References: <1218d6a50805301550u50ddc883l4e07607c2e70f760@mail.gmail.com> Message-ID: <14f0e3620806200738y65a7439fwde1ed4d4fb98e63a@mail.gmail.com> On Sat, May 31, 2008 at 12:50 AM, db wrote: > I am not sure what the problem is. Mnesia directory actually gets > created at /var/mensia/cacherl@REDACTED but there is no content. > Is this something to do with disc_copies and any reason for using > disc_copies rather than ram_copies? Another question, does cacherl > work with mysql memcached engine? > > [a@REDACTED cacherl]# ./start.sh > Erlang (BEAM) emulator version 5.6.1 [source] [smp:2] > [async-threads:30] [hipe] [kernel-poll:true] > > Eshell V5.6.1 (abort with ^G) > (cacherl@REDACTED)1> > =INFO REPORT==== 30-May-2008::17:16:51 === > Creating mnesia table ns_default > > =SUPERVISOR REPORT==== 30-May-2008::17:16:51 === > Supervisor: {local,memcached} > Context: start_error > Reason: {{badmatch, > {aborted, > {bad_type,ns_default,disc_copies, > 'cacherl@REDACTED'}}}, > [{memcached_mnesia,init_mnesia_table,2}, > {memcached_mnesia,init,1}, > {gen_server,init_it,6}, > {proc_lib,init_p,5}]} > Offender: [{pid,undefined}, > {name,memcached_mnesia}, > {mfa,{memcached_mnesia,start_link,[]}}, > {restart_type,permanent}, > {shutdown,2000}, > {child_type,worker}] > > > =CRASH REPORT==== 30-May-2008::17:16:51 === > crasher: > pid: <0.69.0> > registered_name: [] > exception exit: {shutdown,{memcached,start,[normal,[]]}} > in function application_master:init/4 > initial call: application_master:init(<0.6.0>,<0.68.0>, > {appl_data,memcached, > [memcached,memcached_mnesia], > undefined, > {memcached,[]}, > [memcached,memcached_mnesia, > memcached_stats,memcached_util, > memcached_purge,memcached_reader], > [],infinity,infinity}, > normal) > ancestors: [<0.68.0>] > messages: [{'EXIT',<0.70.0>,normal}] > links: [<0.68.0>,<0.6.0>] > dictionary: [] > trap_exit: true > status: running > heap_size: 377 > stack_size: 23 > reductions: 87 > neighbours: > > =INFO REPORT==== 30-May-2008::17:16:51 === > application: memcached > exited: {shutdown,{memcached,start,[normal,[]]}} > type: temporary > > =INFO REPORT==== 30-May-2008::17:16:51 === > started TCP listener on 0.0.0.0:11311 > > -- > rk > > That which we persist in doing becomes easier for us to do; not that > the nature of the thing itself is changed, but that our power to do is > increased. > -Ralph Waldo Emerson > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > After power failure i've stumbled upon exacly the same problem. This problem occures because of mnesia schema is created with mnesia:start() (though it is ram-based schema) and not with mnesia:create_schema([node()]). The solution is quite simple, though was not easy to find for me (i'm far from expert in mnesia). To successfully start cacherl one have to do the following: cd path/to/cacherl make make create-schema ./start.sh Hope this helps :) -- Gleb Peregud http://gleber.pl/ Every minute is to be grasped. Time waits for nobody. -- Inscription on a Zen Gong From patrickdlogan@REDACTED Fri Jun 20 19:49:37 2008 From: patrickdlogan@REDACTED (Patrick Logan) Date: Fri, 20 Jun 2008 10:49:37 -0700 Subject: [erlang-questions] Erlang at OSCON 2008 Message-ID: There are several Erlang-related activities at OSCON in a few weeks. There is no BoF posted on the web site though. Is anyone interested in a BoF or some kind of a social setting outside of the specific tutorials and presentations? I'm amazed at the increased awareness of Erlang over the last year, even. At OSCON 2007 I held an Erlang BoF which had 20-30 or so people mostly curious lurkers, so the BoF essentially turned into an hour mini-tutorial running in my Emacs. Anyway... I imagine this year could bring a good population of experienced folks. -Patrick From francesco@REDACTED Fri Jun 20 20:22:39 2008 From: francesco@REDACTED (Francesco Cesarini) Date: Fri, 20 Jun 2008 19:22:39 +0100 Subject: [erlang-questions] Erlang at OSCON 2008 (Portland ErlLounge?) In-Reply-To: References: Message-ID: <485BF56F.7030407@erlang-consulting.com> Hi Patrick, great idea, whatever a BOF is (Definitions I found include Board of Fisheries, A pear shaped furnace & Birds of Feather)! But an ErlLounge sounds great. There are quite a few Erlang developers in Portland, several of which were at the workshop back in 2006. There is also a user group, but I am not sure how active it is. I'll send out a few emails. I will be arriving in Portland on the 27th of July, leaving on the 30th. The 29th would probably be a good day, as it is on the day of the tutorial. It should also drum up interest for Kevin and Jan's talks. For those of you thinking of OSCON, the following are the Erlang talks / events I am aware of: Jan Lehnardt on Couch DB: http://en.oreilly.com/oscon2008/public/schedule/detail/2725 (He is also speaking at the Erlang eXchange next week) Kevin Scaldaferri on Erlang Concurrency: http://en.oreilly.com/oscon2008/public/schedule/detail/3065 and me giving a Practical Erlang Programming Tutorial: http://en.oreilly.com/oscon2008/public/schedule/detail/3373 Francesco -- http://www.erlang-consulting.com Patrick Logan wrote: > There are several Erlang-related activities at OSCON in a few weeks. > There is no BoF posted on the web site though. > > Is anyone interested in a BoF or some kind of a social setting outside > of the specific tutorials and presentations? > > I'm amazed at the increased awareness of Erlang over the last year, > even. At OSCON 2007 I held an Erlang BoF which had 20-30 or so people > mostly curious lurkers, so the BoF essentially turned into an hour > mini-tutorial running in my Emacs. > > Anyway... I imagine this year could bring a good population of > experienced folks. > > -Patrick > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > From matt@REDACTED Fri Jun 20 21:42:12 2008 From: matt@REDACTED (Matt Harrison) Date: Fri, 20 Jun 2008 20:42:12 +0100 Subject: [erlang-questions] Erlsom it's so close to being SAXy In-Reply-To: <407d9ef80806192324x47c9800cp5dd1c853a4c6f2cd@mail.gmail.com> References: <485ACE7F.901@lummie.co.uk> <407d9ef80806192324x47c9800cp5dd1c853a4c6f2cd@mail.gmail.com> Message-ID: <485C0814.9090502@lummie.co.uk> It's official, Erlsom is Saxy... I have the latest version of sf.net, and have successfully parsed the 6.6Gb file in about 22 minutes. With an average of 5.5Mb/s parsing time which on my T2500 laptop is pretty damn good. It was nice to note that it was CPU bound, so when knowing the XML structure I can potentially split the file processing into blocks and multi-thread it. Either way 22 minutes is an acceptable time in my book. Thanks for your help Willem. Willem de Jong wrote: > Hello Matt, > > Yes, this is possible (and documented, but maybe not clear enough - > are you using the latest version? Did you get erlsom from CEAN? That > version is quite old). > > There even is an example included in the distribution that shows how > to do it. I am copying part of it below. > > Good luck, > Willem > > > %% Example to show how the Erlsom Sax parser can be used in combination > %% with a 'continuation function'. This enables parsing of very big > documents > %% in a sort of streaming mode. > %% > %% When the sax parser reaches the end of a block of data, it calls the > %% continuation function. This should return the next block of data. > %% > %% the continuation function is a function that takes 2 arguments: > Tail and > %% State. > %% - Tail is the (short) list of characters that could not yet be > parsed > %% because it might be a special token or not. Since this still > has to > %% be parsed, it should be put in front of the next block of data. > %% - State is information that is passed by the parser to the callback > %% functions transparently. This can be used to keep track of the > %% location in the file etc. > %% The function returns {NewData, NewState}, where NewData is a list of > %% characters/unicode code points, and NewState the new value for the > State. > > -export([run/0]). > > %% 'chunk' is the number of characters that is read at a time. > %% should be tuned for the best result. (109 is obviously not a good > value, > %% it should be bigger than that - try it out). > -define(chunk, 109). > > run() -> > F = fun count_books/2, %% the callback function that handles the > sax events > G = fun continue_file/2, %% the callback function that returns the next > %% chunk of data > %% open file > {ok, Handle} = file:open(xml(), [read, raw, binary]), > Position = 0, > CState = {Handle, Position, ?chunk}, > SaxCallbackState = undefined, > %% erlsom:parse_sax() returns {ok, FinalState, TrailingBytes}, > %% where TrailingBytes is the rest of the input-document > %% that follows after the last closing tag of the XML, and Result > %% is the value of the State after processing the last SAX event. > {ok, Result, _TrailingBytes} = > erlsom:parse_sax(<<>>, SaxCallbackState, F, > [{continuation_function, G, CState}]), > %% close file > ok = file:close(Handle), > > %% Result is a list [{track_id, count}, ...] > lists:foreach(fun({Date, Count}) -> > io:format("Date: ~p - count: ~p~n", [Date, Count]) > end, Result), > ok. > > %% this is a continuation function that reads chunks of data > %% from a file. > continue_file(Tail, {Handle, Offset, Chunk}) -> > %% read the next chunk > case file:pread(Handle, Offset, Chunk) of > {ok, Data} -> > {<>, {Handle, Offset + Chunk, Chunk}}; > eof -> > {Tail, {Handle, Offset, Chunk}} > end. > > count_books(startDocument, _) -> > etc... > > > > On 6/19/08, *Matt Harrison* > wrote: > > See what I did there with the subject title :) > > All > > Has anyone used Erlsom for sax parsing straight from a file, i.e > without > using file:read_file to load a whole file. I can't seem to find an > appropriate call, and the docs don't seem to cover it. > > I have large files 6gb+ that I need to SAX parse, the main requirement > being able to parse them with about 1gb of memory. > > Erlsom works a treat with files that are a few 100Mb but requires the > whole file loaded in memory, which kinda kills the main benefit of sax > parsing in my opinion. > > I am not especially bothered about speed, (it was suggested that I > look > at c parsers linked into erlang) as this is for a data import process > that will only happen rarely (mainly for development and testing > purposes). > > I still don't seem to be able to find any xmerl_eventp examples so if > you have one please let me know. > > regards and thanks, > > Matt > > I haven't discarded the using a c library I'm just fairly new to > erlang > and would prefer an erlang solution if possible as I don't want to > venture into ports quite yet :) > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pub.address@REDACTED Fri Jun 20 21:25:51 2008 From: pub.address@REDACTED (public address) Date: Fri, 20 Jun 2008 12:25:51 -0700 (PDT) Subject: [erlang-questions] Linking problems to libei.a Message-ID: <850952.7778.qm@web59902.mail.ac4.yahoo.com> Hi all, I'm trying to link a c program to the ei library. This seems not to work, I get linking errors: undefined reference to XXXXX. Platform is linux, erlang is compiled from otp_src_R12B-3 the test program: #include #include int main(int argc, char** argv) { ei_term et; char buf [255]; int i = 0; ei_decode_ei_term(buf, &i, &et); return 0; } the compile commands: # no problems here: gcc -Wall -ggdb -I/home/steffen/usr/lib/erlang/lib/erl_interface-3.5.7/include -c ltest.c #linker problems: gcc -L/home/steffen/usr/lib/erlang/lib/erl_interface-3.5.7/lib -lei -lerl_interface ltest.o -o ltest >>/home/steffen/hacking/erl_link_tst/ltest.c:9: undefined reference to `ei_decode_ei_term'<< the content of home/steffen/usr/lib/erlang/lib/erl_interface-3.5.7/lib: ls -l /home/steffen/usr/lib/erlang/lib/erl_interface-3.5.7/lib total 1856 -rw-r--r-- 1 steffen steffen 724286 2008-06-19 02:15 libei.a -rw-r--r-- 1 steffen steffen 714258 2008-06-19 02:15 libei_st.a -rw-r--r-- 1 steffen steffen 220266 2008-06-19 02:15 liberl_interface.a -rw-r--r-- 1 steffen steffen 219330 2008-06-19 02:15 liberl_interface_st.a thanks in advance for helpful hints greetings steffen ps: I hope yahoo does not send this in html "format" -------------- next part -------------- An HTML attachment was scrubbed... URL: From smparkes@REDACTED Fri Jun 20 23:19:44 2008 From: smparkes@REDACTED (Steven Parkes) Date: Fri, 20 Jun 2008 14:19:44 -0700 Subject: [erlang-questions] Actor BoF at OSCON? (was RE: Erlang at OSCON 2008) In-Reply-To: References: Message-ID: <003301c8d31b$5cab0a30$16011e90$@net> [Originated on the Erlang list ...] I've been toying with the idea of submitting a proposal for an Actor BoF (birds-of-a-feather), sharing experiences among the different actor/actor-like languages and libraries out there ... -----Original Message----- From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Patrick Logan Sent: Friday, June 20, 2008 10:50 AM To: erlang-questions@REDACTED Subject: [erlang-questions] Erlang at OSCON 2008 There are several Erlang-related activities at OSCON in a few weeks. There is no BoF posted on the web site though. Is anyone interested in a BoF or some kind of a social setting outside of the specific tutorials and presentations? I'm amazed at the increased awareness of Erlang over the last year, even. At OSCON 2007 I held an Erlang BoF which had 20-30 or so people mostly curious lurkers, so the BoF essentially turned into an hour mini-tutorial running in my Emacs. Anyway... I imagine this year could bring a good population of experienced folks. -Patrick _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://www.erlang.org/mailman/listinfo/erlang-questions From rvirding@REDACTED Sat Jun 21 00:10:46 2008 From: rvirding@REDACTED (Robert Virding) Date: Sat, 21 Jun 2008 00:10:46 +0200 Subject: [erlang-questions] Idiomatic Erlang, style/performance question In-Reply-To: <963b309aaaba5312158788bd42b4cab5.squirrel@www.geekisp.com> References: <310053.70921.qm@web28314.mail.ukl.yahoo.com> <963b309aaaba5312158788bd42b4cab5.squirrel@www.geekisp.com> Message-ID: <3dbc6d1c0806201510k593f9d90x6e9bb4f67416e9c@mail.gmail.com> On 20/06/2008, Dominic Williams wrote: > Hello, > > ROK wrote: >> I prefer the term "Tail-Call Optimisation" because it is more accurate. >> The idea (as explained in "Lambda - the Ultimate Goto") is that if the >> dynamically last thing a function does (in any language) is to return >> the result of another function call, then there is no point in keeping >> the current stack frame. You might as well arrange for the new >> function to return directly to the original caller. Think of *all* >> tail calls as "argument binding + GOTO". > > Is the optimisation possible even when the tail call is not to the same > function? Erlang always optimises the last call irrespective which function is called. Apart from implementing loops as recursive functions this feature makes some very pleasant types of programming possible, for example finite-state machines where changing state is implemented by the function for one state calling the function for the next state as the last thing it does. Robert From ahmed.nawras@REDACTED Sat Jun 21 15:00:29 2008 From: ahmed.nawras@REDACTED (Ahmed Ali) Date: Sat, 21 Jun 2008 17:00:29 +0400 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: References: Message-ID: Hi, Sure. Let me clean up my code and then post it somewhere in the internet. I'll do that within the coming week. Best regards, Ahmed Al-Issaei On Tue, Jun 17, 2008 at 5:49 PM, devdoer bird wrote: > yes, that's the one I need. > > 2008/6/14, Ahmed Ali : >> >> Hi, >> >> I understand what you want. Having used log4j myself, I've tried to >> find something similar for my own. Eventually, I had to write it >> myself. This module only supports file logs but it also have support >> for multiple log levels, including critical (fatal) level, different >> loggers for different modules, size-based rotation...etc. >> >> I haven't shared the project yet, as it has some shortcomings I need >> to address before releasing it, however, I'll announce it soon. I hope >> this is what you're look for. I'll keep you updated. >> >> Best regards, >> >> Ahmed Al-Issaei >> >> 2008/6/12 devdoer bird : >> > Hi: >> > >> > I wonder whether any logging module for erlang exists,like python's >> > logging >> > module which support :critical level setting ,filtering ,diffrent >> > handlers >> > (stream handler,socket handler,ram handler,file handler...). >> > >> > Thanks a lot! >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://www.erlang.org/mailman/listinfo/erlang-questions >> > > > From anders.nygren@REDACTED Sat Jun 21 16:50:30 2008 From: anders.nygren@REDACTED (Anders Nygren) Date: Sat, 21 Jun 2008 09:50:30 -0500 Subject: [erlang-questions] erlsom:write encode ">" Message-ID: Hi erlsom:write does not encode ">" to ">" this patch fixes that. /Anders Index: lib/erlsom/src/erlsom_write.erl =================================================================== --- lib/erlsom/src/erlsom_write.erl (revision 1051) +++ lib/erlsom/src/erlsom_write.erl (working copy) @@ -648,6 +648,7 @@ escapeChar(38) -> "&"; escapeChar(34) -> """; escapeChar(60) -> "<"; +escapeChar(62) -> ">"; escapeChar(Char) -> Char. decodeIfRequired(Text) when is_binary(Text) -> From patrickdlogan@REDACTED Sat Jun 21 20:17:35 2008 From: patrickdlogan@REDACTED (Patrick Logan) Date: Sat, 21 Jun 2008 11:17:35 -0700 Subject: [erlang-questions] Actor BoF at OSCON? (was RE: Erlang at OSCON 2008) Message-ID: "I've been toying with the idea of submitting a proposal for an Actor BoF (birds-of-a-feather), sharing experiences among the different actor/actor-like languages and libraries out there ..." I would be interested in something like this, or even concurrent and distributed programming generally. -Patrick From matt@REDACTED Sun Jun 22 12:03:48 2008 From: matt@REDACTED (Matt Harrison) Date: Sun, 22 Jun 2008 11:03:48 +0100 Subject: [erlang-questions] Erlsom it's so close to being SAXy In-Reply-To: <485C9CB7.3060704@cs.ntua.gr> References: <485ACE7F.901@lummie.co.uk> <407d9ef80806192324x47c9800cp5dd1c853a4c6f2cd@mail.gmail.com> <485C0814.9090502@lummie.co.uk> <485C9CB7.3060704@cs.ntua.gr> Message-ID: <485E2384.3050101@lummie.co.uk> Kostis, Excellent you learn something new every day....(well I do) Compiling Erlsom with +native brings the erlsom SAX parsing performance up to a whopping 14.1 MByte/s and processes my 6.6Gb file in about 8 minutes. Result ! Thanks for your help everyone. Kostis Sagonas wrote: > Matt Harrison wrote: >> It's official, Erlsom is Saxy... >> >> I have the latest version of sf.net, and have successfully parsed the >> 6.6Gb file in about 22 minutes. With an average of 5.5Mb/s parsing >> time which on my T2500 laptop is pretty damn good. It was nice to >> note that it was CPU bound, so when knowing the XML structure I can >> potentially split the file processing into blocks and multi-thread >> it. Either way 22 minutes is an acceptable time in my book. > > Since this is CPU bound, you might also want to native compile Erlsom > -- this can be done by adding a +native option to the Erlang compiler > -- and see how much this improves things. > > Kostis > From alpar@REDACTED Sun Jun 22 12:56:15 2008 From: alpar@REDACTED (=?ISO-8859-1?Q?Alp=E1r_J=FCttner?=) Date: Sun, 22 Jun 2008 11:56:15 +0100 Subject: [erlang-questions] or? In-Reply-To: References: <997099.93066.qm@web28315.mail.ukl.yahoo.com> Message-ID: <1214132175.15083.20.camel@piko.site> > In this case, it would be better to do > > fib(N) when is_integer(N), N >= 0 -> > fib_aux(N). > > fib_aux(N) when N > 1 -> fib_aux(N-1) + fib_aux(N-2); > fib_aux(N) -> N. > > Better still, of course, would be to use the standard > linear-time tail-recursive version, or even better, the > less well known logarithmic time version (an approach > that works for other recurrences as well). Does erlang implement some kind of fast multiplication algorithm for bignums? I'm just wondering because if the multiplication is done in O(n^2) time, then this 'logaritmic time' Fibonacci computation is actually not faster then the 'linear time' one. (It does O(log n) operation, however not only does it use additions but also multiplication.) Regards, Alpar From w.a.de.jong@REDACTED Sun Jun 22 19:05:34 2008 From: w.a.de.jong@REDACTED (Willem de Jong) Date: Sun, 22 Jun 2008 19:05:34 +0200 Subject: [erlang-questions] erlsom:write encode ">" In-Reply-To: References: Message-ID: <407d9ef80806221005u2b34a17do3a3675ed6ca0597c@mail.gmail.com> I thought that that was not required. Just to be sure I had another look at the spec, and found out that it is indeed not required, *except* if the previous characters are "]]" ("for compatibility reasons"). In practice I can't imagine that any parser would have problems with ">", but I'll change it anyway to be fully conformant even in this special case. Regards, Willem On Sat, Jun 21, 2008 at 4:50 PM, Anders Nygren wrote: > Hi > erlsom:write does not encode ">" to ">" > > this patch fixes that. > > /Anders > > Index: lib/erlsom/src/erlsom_write.erl > =================================================================== > --- lib/erlsom/src/erlsom_write.erl (revision 1051) > +++ lib/erlsom/src/erlsom_write.erl (working copy) > @@ -648,6 +648,7 @@ > escapeChar(38) -> "&"; > escapeChar(34) -> """; > escapeChar(60) -> "<"; > +escapeChar(62) -> ">"; > escapeChar(Char) -> Char. > > decodeIfRequired(Text) when is_binary(Text) -> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dnew@REDACTED Sun Jun 22 20:25:31 2008 From: dnew@REDACTED (Darren New) Date: Sun, 22 Jun 2008 11:25:31 -0700 Subject: [erlang-questions] An interesting Erlang-like data structure Message-ID: <485E991B.3050109@san.rr.com> Just B-Trees, but it's interesting how they did some of the distribution stuff. In particular, the client does most of the work (like allocating blocks during a split), and the server just stores and does commit protocols. http://www.hpl.hp.com/techreports/2007/HPL-2007-193.html Thinking on it, I'm kind of surprised a bit there is gen_server and gen_fsm and such, but no "gen_reliable" or "gen_replicated" or "gen_transactional" or something like that. Mnesia has such things, but it's not broken out into a behavior. It sounds like it would be difficult, but doable, to come up with a reasonable API for such. -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From vladdu55@REDACTED Sun Jun 22 21:59:37 2008 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Sun, 22 Jun 2008 21:59:37 +0200 Subject: [erlang-questions] Use of Records in the Java API In-Reply-To: <14f0e3620806110358td345347j506b8ac5787fd401@mail.gmail.com> References: <59F222DE-8A8B-4011-91EE-DBEB1920C14C@rodanotech.ch> <14f0e3620806110358td345347j506b8ac5787fd401@mail.gmail.com> Message-ID: <95be1d3b0806221259r52e51d81oc0cb7179b9a48a2a@mail.gmail.com> Hi, On Wed, Jun 11, 2008 at 10:23 AM, Alexander Lamb > wrote: > > My Erlang modules are mainly a server to Java Web applications (for > > the time being). > > This means that I am making a big use of the Erlang Java API which > > works fine. > > Since my Erlang modules use Mnesia to store data coming from the Java > > side (and vice versa), I need to implement functions with many > > parameters for example to insert new data in a Mnesia table. > > I also need to convert tuples into actual Java objects when reading > > data. > > This would be simpler if the Java API had an OtpErlangRecord. It would > > make it possible to subclass it and could even be the prefered way of > > bridging the Erlang world with the Java world. > As Gleb pointed out, records are module local, and their name alone isn't enough to tell them apart. There is nothing that stops you to create your Java objects with a constructor (or factory method) that takes an OtpErlangTuple as a parameter and converts it to its Java counterpart. Conversely, a method similar to toString, named for example toErlang, would make the opposite conversion. Compared to an automatic conversion, this allows mappings that are more complex that one-field-to-one-field. The downside is that in most cases, this is boilerplate code. best regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From anders.nygren@REDACTED Sun Jun 22 22:13:27 2008 From: anders.nygren@REDACTED (Anders Nygren) Date: Sun, 22 Jun 2008 15:13:27 -0500 Subject: [erlang-questions] erlsom:write encode ">" In-Reply-To: <407d9ef80806221005u2b34a17do3a3675ed6ca0597c@mail.gmail.com> References: <407d9ef80806221005u2b34a17do3a3675ed6ca0597c@mail.gmail.com> Message-ID: On Sun, Jun 22, 2008 at 12:05 PM, Willem de Jong wrote: > I thought that that was not required. Just to be sure I had another look at > the spec, and found out that it is indeed not required, *except* if the > previous characters are "]]" ("for compatibility reasons"). In practice I > can't imagine that any parser would have problems with ">", but I'll change > it anyway to be fully conformant even in this special case. > Sorry, I didn't know that. I just saw that Axis is sending > and we were not, so I assumed that it was an error. /Anders > Regards, > Willem > > On Sat, Jun 21, 2008 at 4:50 PM, Anders Nygren > wrote: >> >> Hi >> erlsom:write does not encode ">" to ">" >> >> this patch fixes that. >> >> /Anders >> >> Index: lib/erlsom/src/erlsom_write.erl >> =================================================================== >> --- lib/erlsom/src/erlsom_write.erl (revision 1051) >> +++ lib/erlsom/src/erlsom_write.erl (working copy) >> @@ -648,6 +648,7 @@ >> escapeChar(38) -> "&"; >> escapeChar(34) -> """; >> escapeChar(60) -> "<"; >> +escapeChar(62) -> ">"; >> escapeChar(Char) -> Char. >> >> decodeIfRequired(Text) when is_binary(Text) -> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions > > From ok@REDACTED Mon Jun 23 04:16:30 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 23 Jun 2008 14:16:30 +1200 Subject: [erlang-questions] Idiomatic Erlang, style/performance question In-Reply-To: <963b309aaaba5312158788bd42b4cab5.squirrel@www.geekisp.com> References: <310053.70921.qm@web28314.mail.ukl.yahoo.com> <963b309aaaba5312158788bd42b4cab5.squirrel@www.geekisp.com> Message-ID: <23A48D07-BE76-40D0-B073-355F6AA9B5D7@cs.otago.ac.nz> On 20 Jun 2008, at 8:16 pm, Dominic Williams wrote: > Hello, > > ROK wrote: >> I prefer the term "Tail-Call Optimisation" because it is more >> accurate. >> The idea (as explained in "Lambda - the Ultimate Goto") is that if >> the >> dynamically last thing a function does (in any language) is to return >> the result of another function call, then there is no point in >> keeping >> the current stack frame. You might as well arrange for the new >> function to return directly to the original caller. Think of *all* >> tail calls as "argument binding + GOTO". > > Is the optimisation possible even when the tail call is not to the > same > function? That's precisely why I prefer to call it tail CALL optimisation. It is not in any way limited to recursion. >>> >>> power(N, P) when is_integer(P), P >= 0 -> >>> ipower(N, P, 1); >>> power(N, P) when is_integer(P), P < 0 -> >>> 1/ipower(N, -P, 1). >> >> The first call here is a tail call, the second is not. > > As I wrote in a previous post, I thought that there was no > optimisation > here, because it is not a /recursive/ tail-call. Am I wrong? What is > the > case for Erlang, in particular? The first clause ends with a call to ipower/3. The fact that it is not a *recursive* call doesn't matter a whit; what counts is that there is nothing left to do in power/2 when ipower/3 finishes. The dynamically last action in the caller is the call. That clause IS subject to tail call optimisation. The last thing that happens in the second clause is the division. So that one isn't a tail call. As for what Erlang actually does, there's one way to tell for sure. % erlc -"'S'" foobar.erl % view foobar.S Given power(X, P) when is_integer(P), P >= 0 -> ipower(X, P, 1); power(X, P) when is_integer(P), P < 0 -> 1/ipower(X, -P, 1). ipower(_, 0, R) -> R; ipower(X, P, R) -> ipower(X, P-1, R*X). > you get {function, power, 2, 2}. {label,1}. {func_info,{atom,fib},{atom,power},2}. {label,2}. {test,is_integer,{f,3},[{x,1}]}. {test,is_ge,{f,3},[{x,1},{integer,0}]}. {move,{integer,1},{x,2}}. {'%live',3}. {call_only,3,{f,5}}. %% In the code for clause 1, no stack frame is %% allocated or deallocated and there is no 'return'. {label,3}. {test,is_integer,{f,1},[{x,1}]}. {test,is_lt,{f,1},[{x,1},{integer,0}]}. {allocate,0,2}. {gc_bif,'-',{f,0},2,[{x,1}],{x,1}}. {move,{integer,1},{x,2}}. {'%live',3}. {call,3,{f,5}}. {test_heap,{alloc,[{words,0},{floats,1}]},1}. {fconv,{integer,1},{fr,0}}. {fconv,{x,0},{fr,1}}. fclearerror. {bif,fdiv,{f,0},[{fr,0},{fr,1}],{fr,1}}. {fcheckerror,{f,0}}. {fmove,{fr,1},{x,0}}. {'%live',1}. {deallocate,0}. return. %% In the code for clause 2, a stack frame IS %% allocated, and deallocated, and there is a 'return'. {function, ipower, 3, 5}. {label,4}. {func_info,{atom,fib},{atom,ipower},3}. {label,5}. {test,is_eq_exact,{f,6},[{x,1},{integer,0}]}. {move,{x,2},{x,0}}. return. %% This is a simple return from a leaf call. {label,6}. {gc_bif,'-',{f,0},3,[{x,1},{integer,1}],{x,1}}. {gc_bif,'*',{f,0},3,[{x,2},{x,0}],{x,2}}. {'%live',3}. {call_only,3,{f,5}}. %% This is a tail call with no frame. Basically, if the code for a clause doesn't end with 'return', tail call optimisation happened. From ok@REDACTED Mon Jun 23 05:03:15 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 23 Jun 2008 15:03:15 +1200 Subject: [erlang-questions] or? In-Reply-To: <1214132175.15083.20.camel@piko.site> References: <997099.93066.qm@web28315.mail.ukl.yahoo.com> <1214132175.15083.20.camel@piko.site> Message-ID: On 22 Jun 2008, at 10:56 pm, Alp?r J?ttner wrote: > Does erlang implement some kind of fast multiplication algorithm for > bignums? Looking at emulator/beam/big.c, it does not appear so. However, that's something that could be changed without affecting anything else in Erlang. I note that the Open Solaris sources include a fast integer multiply, though oddly enough, not in libmp. Of course, I was thinking of floating point operations, not large integer ones. Arguably I _should_ have been thinking of large integer ones. I note that the performance of the large integer operations in Squeak Smalltalk was substantially improved in response to the needs of cryptography, and suspect that long term, there might be a similar need for Erlang to have and to be documented as having efficient large integer arithmetic. From jachym.holecek@REDACTED Mon Jun 23 09:37:46 2008 From: jachym.holecek@REDACTED (Jachym Holecek) Date: Mon, 23 Jun 2008 09:37:46 +0200 Subject: [erlang-questions] Linking problems to libei.a In-Reply-To: <850952.7778.qm@web59902.mail.ac4.yahoo.com> References: <850952.7778.qm@web59902.mail.ac4.yahoo.com> Message-ID: On Fri, 20 Jun 2008 21:25:51 +0200, public address wrote: > the compile commands: > # no problems here: > gcc -Wall -ggdb > -I/home/steffen/usr/lib/erlang/lib/erl_interface-3.5.7/include -c ltest.c > #linker problems: > gcc -L/home/steffen/usr/lib/erlang/lib/erl_interface-3.5.7/lib -lei > -lerl_interface ltest.o -o ltest The order of object files (including libraries) does matter, see '-l' option in gcc manual for more details. This should work: $ gcc -L/home/steffen/usr/lib/erlang/lib/erl_interface-3.5.7/lib \ -o ltest ltest.o -lei -lerl_interface HTH, -- Jachym From pub.address@REDACTED Mon Jun 23 09:58:49 2008 From: pub.address@REDACTED (public address) Date: Mon, 23 Jun 2008 00:58:49 -0700 (PDT) Subject: [erlang-questions] Linking problems to libei.a Message-ID: <824666.3871.qm@web59909.mail.ac4.yahoo.com> > The order of object files (including libraries) does matter, see '-l' > option in gcc manual for more details. I was to blind to see. [...] Thanks, that helped Steffen -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Mon Jun 23 10:53:24 2008 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Mon, 23 Jun 2008 10:53:24 +0200 Subject: [erlang-questions] Erlide tutorial? In-Reply-To: <9754549.225131213038009752.JavaMail.root@zimbra> References: <9754549.225131213038009752.JavaMail.root@zimbra> Message-ID: <95be1d3b0806230153v6126e899ib871d7a13a8de391@mail.gmail.com> Hi! On Mon, Jun 9, 2008 at 9:00 PM, Michael FIG wrote: > Hi, > > Could anybody out there please point me at a very simple "hello world" > tutorial in using Erlide? I'm an Emacs user with absolutely no experience > with Eclipse, and I can't for the life of me figure out how to run my > newly-created Erlang projects, evaluate something in the console, or debug a > project. I don't know if that's because my Erlide installation is broken, > or if it's just because I don't know where to find these features. Sorry for the delay, I was on vacation. There are no tutorials yet, sorry. Your main point of interaction with your erlang programs is the erlang console view. It works as usual, except that the input is in a separate field. If you press Enter and the expression in that field is complete and correct, it will be sent to the shell. If auto-build is enabled, you don't need to do anything after editing your code. It will be compiled and loaded into the runtime, ready to use from the console. Debugging isn't working yet, but it's the next big thing to come. Is this what you are after? If you need more detailed information, please try to structure the questions so that we can reuse the answers in a future tutorial. best regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Mon Jun 23 11:55:24 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 23 Jun 2008 11:55:24 +0200 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <6c2563b20806192324s250d5e35wbc6bfb518bde821e@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> <6c2563b20806192324s250d5e35wbc6bfb518bde821e@mail.gmail.com> Message-ID: "Edwin Fine" writes: > Plus using -K true and > perhaps +A 128 should improve things. > Async threads (+A 128) can only speed up file operations (the inet driver driver does not use the async thread pool). /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From bjorn@REDACTED Mon Jun 23 12:00:29 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 23 Jun 2008 12:00:29 +0200 Subject: [erlang-questions] or? In-Reply-To: <1214132175.15083.20.camel@piko.site> References: <997099.93066.qm@web28315.mail.ukl.yahoo.com> <1214132175.15083.20.camel@piko.site> Message-ID: Alp?r J?ttner writes: > Does erlang implement some kind of fast multiplication algorithm for > bignums? No. /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From ignatios@REDACTED Mon Jun 23 14:46:16 2008 From: ignatios@REDACTED (Ignatios Souvatzis) Date: Mon, 23 Jun 2008 14:46:16 +0200 Subject: [erlang-questions] shared library avoidance code harmful on certain architectures Message-ID: <20080623124616.GG6701@cs.uni-bonn.de> Hi, erlang 12 fails to build on NetBSD/amd64. I was told to test the patches below. Basically - use of -lfoo will do the right thing on all (sane) systems, while the complicated stuff ended up using libfoo.a, which happened to work on NetBSD/i386, but fails on NetBSD/amd64 due to the created text relocations. should be able to give you more details. I'm only a user of Erlang, most of the time. Regards, -is Allow the system linker to use the shared library for dynamic modules. --- lib/crypto/c_src/Makefile.in.orig 2007-11-26 20:01:59.000000000 +0100 +++ lib/crypto/c_src/Makefile.in @@ -91,11 +91,7 @@ endif endif DYNAMIC_CRYPTO_LIB=@SSL_DYNAMIC_ONLY@ -ifeq ($(DYNAMIC_CRYPTO_LIB),yes) CRYPTO_LINK_LIB=-L$(SSL_LIBDIR) -lcrypto -else -CRYPTO_LINK_LIB=$(SSL_LIBDIR)/libcrypto.a -endif llow the system linker to use the shared library for dynamic modules. --- lib/ssl/c_src/Makefile.in.orig 2008-06-20 19:33:40.000000000 +0200 +++ lib/ssl/c_src/Makefile.in @@ -87,11 +87,7 @@ PORT_PROGRAM = $(BINDIR)/ssl_esock$(exe) SKIP_BUILDING_BINARIES := false # Try to be BC for R10 -ifeq ($(findstring @SSL_,@REDACTED@),@REDACTED) DYNAMIC_CRYPTO_LIB=yes -else -DYNAMIC_CRYPTO_LIB=@SSL_DYNAMIC_ONLY@ -endif ifeq ($(DYNAMIC_CRYPTO_LIB),yes) From mats.cronqvist@REDACTED Mon Jun 23 15:01:26 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Mon, 23 Jun 2008 15:01:26 +0200 Subject: [erlang-questions] Idiomatic Erlang, style/performance question In-Reply-To: <23A48D07-BE76-40D0-B073-355F6AA9B5D7@cs.otago.ac.nz> References: <310053.70921.qm@web28314.mail.ukl.yahoo.com> <963b309aaaba5312158788bd42b4cab5.squirrel@www.geekisp.com> <23A48D07-BE76-40D0-B073-355F6AA9B5D7@cs.otago.ac.nz> Message-ID: <485F9EA6.8090604@gmail.com> Richard A. O'Keefe wrote: > On 20 Jun 2008, at 8:16 pm, Dominic Williams wrote: > > >> Hello, >> >> ROK wrote: >> >>> I prefer the term "Tail-Call Optimisation" because it is more >>> accurate. >>> The idea (as explained in "Lambda - the Ultimate Goto") is that if >>> the >>> dynamically last thing a function does (in any language) is to return >>> the result of another function call, then there is no point in >>> keeping >>> the current stack frame. You might as well arrange for the new >>> function to return directly to the original caller. Think of *all* >>> tail calls as "argument binding + GOTO". >>> >> Is the optimisation possible even when the tail call is not to the >> same >> function? >> > > That's precisely why I prefer to call it tail CALL optimisation. > It is not in any way limited to recursion. > for illustration, consider this wonderful code(somewhat obfuscated to prevent inlining); -module('foo'). -author('Mats Cronqvist'). -export([a/1]). a(X) -> c(b(X)). b(X) -> X/element(3,now()). c(X) -> float_to_list(X)/element(2,now()). when it crashes in c/1, the stack does not contain a/1. when it crashes in b/1, the stack will show that b/1 was called from a/1. (erl@REDACTED)19> foo:a(1). ** exited: {badarith,[{foo,c,1}, {erl_eval,do_apply,5}, {shell,exprs,6}, {shell,eval_loop,3}]} ** (erl@REDACTED)20> foo:a(x). ** exited: {badarith,[{foo,b,1}, {foo,a,1}, {erl_eval,do_apply,5}, {shell,exprs,6}, {shell,eval_loop,3}]} ** the stack only contains frames for functions that we are supposed to return to. the call to b/1 will return to a/1, but the call to c/1 won't. mats From joelr1@REDACTED Mon Jun 23 15:33:27 2008 From: joelr1@REDACTED (Joel Reymont) Date: Mon, 23 Jun 2008 14:33:27 +0100 Subject: [erlang-questions] Mnesia Unlimited Message-ID: http://www.wagerlabs.com/blog/2008/06/mnesia-unlimited.html With the support from the Dukes of Erl I was able to add a generic external table mechanism to Mnesia 4.3.5 (R11B5). Details above. The project is open source and I'm the maintainer. Feedback and questions are welcome! Thanks, Joel -- wagerlabs.com From erlang-questions_efine@REDACTED Mon Jun 23 17:12:48 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Mon, 23 Jun 2008 11:12:48 -0400 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> <6c2563b20806192324s250d5e35wbc6bfb518bde821e@mail.gmail.com> Message-ID: <6c2563b20806230812w3178576iba8950bbaf43930c@mail.gmail.com> That's good to know. It was given as a general recommendation to speed up communications (I forget where). Thanks. On Mon, Jun 23, 2008 at 5:55 AM, Bjorn Gustavsson wrote: > "Edwin Fine" writes: > > > Plus using -K true and > > perhaps +A 128 should improve things. > > > > Async threads (+A 128) can only speed up file operations (the inet driver > driver does not use the async thread pool). > > /Bjorn > -- > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From circularfunc@REDACTED Mon Jun 23 17:51:15 2008 From: circularfunc@REDACTED (Circular Function) Date: Mon, 23 Jun 2008 15:51:15 +0000 (GMT) Subject: [erlang-questions] running file from win prompt? Message-ID: <918518.59424.qm@web28313.mail.ukl.yahoo.com> if i have mymath.erl in usr and want to run it from the windows command prompt and not the shell. how do i compile and run? just calling mymath.erl doesnt work. does it have to be run through the shell? __________________________________________________________ Ta semester! - s?k efter resor hos Yahoo! Shopping. J?mf?r pris p? flygbiljetter och hotellrum h?r: http://shopping.yahoo.se/c-169901-resor-biljetter.html?partnerId=96914052 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hayeah@REDACTED Mon Jun 23 19:03:49 2008 From: hayeah@REDACTED (Howard Yeh) Date: Mon, 23 Jun 2008 10:03:49 -0700 Subject: [erlang-questions] Mnesia Unlimited In-Reply-To: References: Message-ID: nice! looks pretty cool. On 6/23/08, Joel Reymont wrote: > http://www.wagerlabs.com/blog/2008/06/mnesia-unlimited.html > > With the support from the Dukes of Erl I was able to add a generic > external table mechanism to Mnesia 4.3.5 (R11B5). Details above. > > The project is open source and I'm the maintainer. > > Feedback and questions are welcome! > > Thanks, Joel > > > -- > wagerlabs.com > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- hayeah.wordpress.com --metacircular thinking From martin.gignac@REDACTED Mon Jun 23 19:16:08 2008 From: martin.gignac@REDACTED (Martin Gignac) Date: Mon, 23 Jun 2008 13:16:08 -0400 Subject: [erlang-questions] running file from win prompt? In-Reply-To: <918518.59424.qm@web28313.mail.ukl.yahoo.com> References: <918518.59424.qm@web28313.mail.ukl.yahoo.com> Message-ID: > if i have mymath.erl in usr and want to run it from the windows command > prompt and not the shell. > how do i compile and run? There's two choices: 1. You must first compile (unlike Perl, Python or Ruby) it with erlc, and then run it by feeding it as an argument to erl: http://www.erlang.org/doc/man/erl.html 2. However, if it's a small script you might want to use the escript method instead (you don't have to compile for that): http://www.erlang.org/doc/man/escript.html That second method is more like people are used to with Perl, Python and Ruby. -Martin From erlang-questions_efine@REDACTED Tue Jun 24 03:08:57 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Mon, 23 Jun 2008 21:08:57 -0400 Subject: [erlang-questions] http:request/5 returns econnrefused for localhost Message-ID: <6c2563b20806231808r18fffaabu8120134b4ba1da30@mail.gmail.com> I have a really bizarre situation here. I have code using http:request that used to work, apparently until I installed R12B-3 (or did something stupid to break something somewhere). Now I get this bizarre situation where it will work with a URL containing a numeric IP, but not "localhost". Whenever I use localhost, I get {error,econnrefused}, but if I use 0.0.0.0 or 127.0.0.1 it works. It always used to work with localhost. I even replaced localhost in /etc/hosts with 0.0.0.0 instead of 127.0.0.1 but to no avail. If I post to localhost via browser it works fine, it's only http:request that is giving problems. I can ping localhost no problem. I'm using Yaws 1.75 on Ubuntu 8.04 (Hardy Heron) x86_64 and Erlang OTP R12B-3. Yaws is running in embedded mode on port 8888 of {0,0,0,0}. Can anyone suggest what might have gone wrong? (xhg_test@REDACTED)13> http:request(post,{"http://localhost:8888/deliver/002",[],"text/xml; charset='utf-8'",""}, [], [], 'XHG'). {error,econnrefused} (xhg_test@REDACTED)14> http:request(post,{"http://127.0.0.1:8888/deliver/002",[],"text/xml; charset='utf-8'",""}, [], [], 'XHG'). {ok,{{"HTTP/1.1",200,"OK"}, [{"date","Tue, 24 Jun 2008 00:58:58 GMT"}, {"server","Yaws/1.75 Yet Another Web Server"}, {"content-length","22"}, {"content-type","text/xml"}], ""}} (xhg_test@REDACTED)15> http:request(post,{"http://0.0.0.0:8888/deliver/002",[],"text/xml; charset='utf-8'",""}, [], [], 'XHG'). {ok,{{"HTTP/1.1",200,"OK"}, [{"date","Tue, 24 Jun 2008 00:59:16 GMT"}, {"server","Yaws/1.75 Yet Another Web Server"}, {"content-length","22"}, {"content-type","text/xml"}], ""}} -------------- next part -------------- An HTML attachment was scrubbed... URL: From javierparis@REDACTED Thu Jun 19 21:35:40 2008 From: javierparis@REDACTED (=?ISO-8859-1?Q?Javier_Par=EDs_Fern=E1ndez?=) Date: Thu, 19 Jun 2008 21:35:40 +0200 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> Message-ID: <399E231E-E06E-4FCE-901A-24C6B7B2DFA8@udc.es> El 19/06/2008, a las 20:06, Rapsey escribi?: > It loops from another module, that way I can update the code at any > time without disrupting anything. > The packets are generally a few hundred bytes big, except keyframes > which tend to be in the kB range. I haven't tried looking with > wireshark. Still it seems a bit odd that a large CPU consumption > would be the symptom. The traffic is strictly one way. Either > someone is sending the stream or receiving it. > The transmit could of course be written with a passive receive, but > the code would be significantly uglier. I'm sure someone here knows > if setting {active, once} every packet is CPU intensive or not. > It seems the workings of gen_tcp is quite platform dependent. If I > run the code in windows, sending more than 128 bytes per gen_tcp > call significantly decreases network output. > Oh and I forgot to mention I use R12B-3. Hi, Without being an expert. 200-300 mb/s in small (hundreds of bytes) packets means a *lot* of system calls if you are doing a gen_tcp:send for each one. If you buffer 3 packets, you are reducing that by a factor of 3 :). I'd try to do an small test doing the same thing in C and compare the results. I think it will also eat a lot of CPU. About the proxy CPU... I'm a bit lost about it, but speculating wildly it is possible that the time spent doing the system calls that gen_tcp is doing is added to the proxy CPU process. Regards. From jan@REDACTED Tue Jun 24 08:43:35 2008 From: jan@REDACTED (Jan Lehnardt) Date: Tue, 24 Jun 2008 08:43:35 +0200 Subject: [erlang-questions] http:request/5 returns econnrefused for localhost In-Reply-To: <6c2563b20806231808r18fffaabu8120134b4ba1da30@mail.gmail.com> References: <6c2563b20806231808r18fffaabu8120134b4ba1da30@mail.gmail.com> Message-ID: <5CA9B9A5-C9EB-4E17-9D63-8C2D18834DEE@apache.org> Heya, On Jun 24, 2008, at 03:08, Edwin Fine wrote: > I have a really bizarre situation here. > > I have code using http:request that used to work, apparently until I > installed R12B-3 (or did something stupid to break something > somewhere). Now I get this bizarre situation where it will work with > a URL containing a numeric IP, but not "localhost". Whenever I use > localhost, I get {error,econnrefused}, but if I use 0.0.0.0 or > 127.0.0.1 it works. It always used to work with localhost. I even > replaced localhost in /etc/hosts with 0.0.0.0 instead of 127.0.0.1 > but to no avail. If I post to localhost via browser it works fine, > it's only http:request that is giving problems. I can ping localhost > no problem. > running in embedded mode on port 8888 of {0,0,0,0}. > > Can anyone suggest what might have gone wrong? We have seen the same issue with CouchDB. What we found out is that in our case localhost was not only resolving to 127.0.0.1 (IPv4) but also ::1 (IPv6) and that http:request() would try to connect to ::1 where no service was listening. Try disabling IPv6 networking to verify this. Cheers Jan -- From alpar@REDACTED Tue Jun 24 10:57:25 2008 From: alpar@REDACTED (=?ISO-8859-1?Q?Alp=E1r_J=FCttner?=) Date: Tue, 24 Jun 2008 09:57:25 +0100 Subject: [erlang-questions] R12B-3 doesn't compile with glibc-2.8 Message-ID: <1214297845.4085.10.camel@piko.site> Hi, There is a line #if __GLIBC__ == 2 && (__GLIBC_MINOR__ >= 3 && __GLIBC_MINOR__ <= 7) in otp_src_R12B-3/erts/emulator/hipe/hipe_x86_signal.c, which causes a compilation error with glibc-2.8. Can I safely change the 7 to 8 in this line? Regards, Alpar From holger@REDACTED Tue Jun 24 11:24:57 2008 From: holger@REDACTED (Holger Hoffstaette) Date: Tue, 24 Jun 2008 11:24:57 +0200 Subject: [erlang-questions] R12B-3 doesn't compile with glibc-2.8 References: <1214297845.4085.10.camel@piko.site> Message-ID: On Tue, 24 Jun 2008 09:57:25 +0100, Alp?r J?ttner wrote: > There is a line > > #if __GLIBC__ == 2 && (__GLIBC_MINOR__ >= 3 && __GLIBC_MINOR__ <= 7) > > in otp_src_R12B-3/erts/emulator/hipe/hipe_x86_signal.c, which causes a > compilation error with glibc-2.8. > Can I safely change the 7 to 8 in this line? I reported this to the bugs list a few days ago and the official resolution was to simply remove the minor version check. Works fine so far, see http://bugs.gentoo.org/show_bug.cgi?id=226063 regards, Holger From mikpe@REDACTED Tue Jun 24 11:42:46 2008 From: mikpe@REDACTED (Mikael Pettersson) Date: Tue, 24 Jun 2008 11:42:46 +0200 Subject: [erlang-questions] R12B-3 doesn't compile with glibc-2.8 In-Reply-To: <1214297845.4085.10.camel@piko.site> References: <1214297845.4085.10.camel@piko.site> Message-ID: <18528.49558.25252.107132@harpo.it.uu.se> Alp?r J?ttner writes: > Hi, > > There is a line > > #if __GLIBC__ == 2 && (__GLIBC_MINOR__ >= 3 && __GLIBC_MINOR__ <= 7) > > in otp_src_R12B-3/erts/emulator/hipe/hipe_x86_signal.c, which causes a > compilation error with glibc-2.8. > Can I safely change the 7 to 8 in this line? Yes. /Mikael From ignatios@REDACTED Tue Jun 24 11:46:13 2008 From: ignatios@REDACTED (Ignatios Souvatzis) Date: Tue, 24 Jun 2008 11:46:13 +0200 Subject: [erlang-questions] http:request/5 returns econnrefused for localhost In-Reply-To: <6c2563b20806231808r18fffaabu8120134b4ba1da30@mail.gmail.com> References: <6c2563b20806231808r18fffaabu8120134b4ba1da30@mail.gmail.com> Message-ID: <20080624094613.GA4827@cs.uni-bonn.de> On Mon, Jun 23, 2008 at 09:08:57PM -0400, Edwin Fine wrote: > I have a really bizarre situation here. > > I have code using http:request that used to work, apparently until I > installed R12B-3 (or did something stupid to break something somewhere). Now > I get this bizarre situation where it will work with a URL containing a > numeric IP, but not "localhost". Whenever I use localhost, I get > {error,econnrefused}, but if I use 0.0.0.0 or 127.0.0.1 it works. It always > used to work with localhost. I even replaced localhost in /etc/hosts with > 0.0.0.0 instead of 127.0.0.1 but to no avail. If I post to localhost via > browser it works fine, it's only http:request that is giving problems. I can > ping localhost no problem. You should teach Yaws to listen also on IPv6 - "localhost" resolves not only to IPv4 127.0.0.1, but also to IPv6 ::1. Depending on operating system, "ping" only works with IPv4 and you have to use "ping6" for IPv6 (that's because you need dirty tricks to write a ping program for IPv4, and less tricks to write one for IPv6, so it's easier to have a seperate one...) Regards, Ignatios Souvatzis From ignatios@REDACTED Tue Jun 24 11:48:11 2008 From: ignatios@REDACTED (Ignatios Souvatzis) Date: Tue, 24 Jun 2008 11:48:11 +0200 Subject: [erlang-questions] http:request/5 returns econnrefused for localhost In-Reply-To: <5CA9B9A5-C9EB-4E17-9D63-8C2D18834DEE@apache.org> References: <6c2563b20806231808r18fffaabu8120134b4ba1da30@mail.gmail.com> <5CA9B9A5-C9EB-4E17-9D63-8C2D18834DEE@apache.org> Message-ID: <20080624094811.GB4827@cs.uni-bonn.de> On Tue, Jun 24, 2008 at 08:43:35AM +0200, Jan Lehnardt wrote: > We have seen the same issue with CouchDB. What we found out > is that in our case localhost was not only resolving to 127.0.0.1 (IPv4) > but also ::1 (IPv6) and that http:request() would try to connect to > ::1 where no service was listening. > > Try disabling IPv6 networking to verify this. Uhm... the fix would be to enable IPv6 in his Yaws. After all, it's easy; the IPv6 support in Erlang is pretty good. -is From jan@REDACTED Tue Jun 24 11:58:04 2008 From: jan@REDACTED (Jan Lehnardt) Date: Tue, 24 Jun 2008 11:58:04 +0200 Subject: [erlang-questions] http:request/5 returns econnrefused for localhost In-Reply-To: <20080624094811.GB4827@cs.uni-bonn.de> References: <6c2563b20806231808r18fffaabu8120134b4ba1da30@mail.gmail.com> <5CA9B9A5-C9EB-4E17-9D63-8C2D18834DEE@apache.org> <20080624094811.GB4827@cs.uni-bonn.de> Message-ID: On Jun 24, 2008, at 11:48, Ignatios Souvatzis wrote: > On Tue, Jun 24, 2008 at 08:43:35AM +0200, Jan Lehnardt wrote: > >> We have seen the same issue with CouchDB. What we found out >> is that in our case localhost was not only resolving to 127.0.0.1 >> (IPv4) >> but also ::1 (IPv6) and that http:request() would try to connect to >> ::1 where no service was listening. >> >> Try disabling IPv6 networking to verify this. > > Uhm... the fix would be to enable IPv6 in his Yaws. After all, it's > easy; > the IPv6 support in Erlang is pretty good. Right, that's the other solution :) My previous mail was sent pre- coffee :) Cheers Jan -- From saleyn@REDACTED Tue Jun 24 13:28:23 2008 From: saleyn@REDACTED (Serge Aleynikov) Date: Tue, 24 Jun 2008 07:28:23 -0400 Subject: [erlang-questions] OTP Release Handling Tutorial Message-ID: <4860DA57.5020808@gmail.com> I noticed that the OTP Release Handling Tutorial at trapexit (http://www.trapexit.org/OTP_Release_Handling_Tutorial) is missing all files referenced in the links. I believe this wasn't the case when the project was hosted on the original trapexit server. Serge From nicolas@REDACTED Tue Jun 24 13:53:38 2008 From: nicolas@REDACTED (Nicolas Niclausse) Date: Tue, 24 Jun 2008 13:53:38 +0200 Subject: [erlang-questions] Slave nodes without proxied I/O? In-Reply-To: <73AF1508-8DE3-437F-8208-521C84A35683@ketralnis.com> References: <73AF1508-8DE3-437F-8208-521C84A35683@ketralnis.com> Message-ID: <4860E042.8050803@niclux.org> David King ecrivait le 01.05.2008 20:34: >> By default, nodes started as slave nodes with slave:start/1 have their >> I/O proxied through the master node that started them. (That is, I/O >> operations done with the 'file' module are done on the master's >> filesystem, not the slaves'). >> Is there any way to turn this off? > > Given the lack of response here, I'll assume that there isn't an easy > to to turn it off. I'm having the same problem as you: i want to start a slave without proxied I/O. After some digging, i discover that you only have to remove the "-master node()" argument in the mk_cmd fun to do that (slave.erl module). It would be nice to add an extra argument to slave:start/start_link : something like: start(Host, Name, Args, ExtraArgs) where ExtraArgs can be:[noproxy] I can write a patch if necessary. -- Nicolas From twoggle@REDACTED Tue Jun 24 13:57:57 2008 From: twoggle@REDACTED (Tim Fletcher) Date: Tue, 24 Jun 2008 04:57:57 -0700 (PDT) Subject: [erlang-questions] how: info on how to traverse abstract format trees? Message-ID: The ERTS User's Guide describes the abstract format, but is there any info (official or otherwise) on how best to traverse these trees? It's possible to get most of the way by just matching on the terms and guessing which parts to traverse next, but it would be easier if there was a clearer definition of what should be handled. From gleber.p@REDACTED Tue Jun 24 14:10:57 2008 From: gleber.p@REDACTED (Gleb Peregud) Date: Tue, 24 Jun 2008 14:10:57 +0200 Subject: [erlang-questions] how: info on how to traverse abstract format trees? In-Reply-To: References: Message-ID: <14f0e3620806240510sb9430c0k31ca96375217dda5@mail.gmail.com> On Tue, Jun 24, 2008 at 1:57 PM, Tim Fletcher wrote: > The ERTS User's Guide describes the abstract format, but is there any > info (official or otherwise) on how best to traverse these trees? It's > possible to get most of the way by just matching on the terms and > guessing which parts to traverse next, but it would be easier if there > was a clearer definition of what should be handled. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions You may take a look at newly created "frabjous" project by Kevin Smith Here it is: http://weblog.hypotheticalabs.com/?p=275 First, it interfaces directly with the compiler and handles all of the low-level details around iterating over the AST and other parse transform muck. Frabjous knows all about multi-pass transforms and provides a persistent state mechanism for parse transformers so they can remember where they've been and what they're doing. -- Gleb Peregud http://gleber.pl/ Every minute is to be grasped. Time waits for nobody. -- Inscription on a Zen Gong From twoggle@REDACTED Tue Jun 24 15:46:03 2008 From: twoggle@REDACTED (Tim Fletcher) Date: Tue, 24 Jun 2008 06:46:03 -0700 (PDT) Subject: [erlang-questions] how: info on how to traverse abstract format trees? In-Reply-To: <14f0e3620806240510sb9430c0k31ca96375217dda5@mail.gmail.com> References: <14f0e3620806240510sb9430c0k31ca96375217dda5@mail.gmail.com> Message-ID: <79854fe0-10a0-4cd6-990f-d9f7d38af651@27g2000hsf.googlegroups.com> > You may take a look at newly created "frabjous" project by Kevin Smith That's where my question came from :) AFAIK frabjous doesn't yet handle deep structures like function definitions. So if I wanted to transform expressions that can be contained in function definitions (e.g. function calls), I would have to write the additional traversal code myself. I'm just checking to see if there might be some recommended/documented way of doing such a traversal. Trying to work it out from parse transform code directly is a bit intimidating (which is why frabjous is very appealing). From vladdu55@REDACTED Tue Jun 24 15:56:38 2008 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Tue, 24 Jun 2008 15:56:38 +0200 Subject: [erlang-questions] how: info on how to traverse abstract format trees? In-Reply-To: <79854fe0-10a0-4cd6-990f-d9f7d38af651@27g2000hsf.googlegroups.com> References: <14f0e3620806240510sb9430c0k31ca96375217dda5@mail.gmail.com> <79854fe0-10a0-4cd6-990f-d9f7d38af651@27g2000hsf.googlegroups.com> Message-ID: <95be1d3b0806240656p655ebd87i7d70e55183c9ab05@mail.gmail.com> Hi, On Tue, Jun 24, 2008 at 3:46 PM, Tim Fletcher wrote: > I'm just checking to see if there might be some recommended/documented > way of doing such a traversal. Trying to work it out from parse > transform code directly is a bit intimidating (which is why frabjous > is very appealing). > You can use syntax_tools. They let you construct and take apart ASTs without having to know the exact representation. regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From svenolof@REDACTED Tue Jun 24 16:23:47 2008 From: svenolof@REDACTED (Sven-Olof Nystr|m) Date: Tue, 24 Jun 2008 16:23:47 +0200 Subject: [erlang-questions] Ideas for a new Erlang Message-ID: <18529.883.586609.71953@hamberg.it.uu.se> Hello everyone, I have written down some thoughts and ideas on the future development of Erlang. Main topics: - some notes on implementation and backward compatibility - an alternative to Erlang's selective receive - a simple language mechanism to allow function in-lining across module boundaries - a new mechanism for introducing local variables with a more cleanly defined semantics - a mini-language to allow the efficent implementation of low-level algorithms The paper can be downloaded here: Sven-Olof Nystr?m From nicholassm@REDACTED Tue Jun 24 16:24:45 2008 From: nicholassm@REDACTED (=?ISO-8859-1?Q?Nicholas_Schultz-M=F8ller?=) Date: Tue, 24 Jun 2008 16:24:45 +0200 Subject: [erlang-questions] (how) Can a tcp socket get "overloaded"? Message-ID: Hi, I am using tcp sockets in a distributed system and I have run into a strange error (or perhaps I'm using sockets the wrong way): I am trying to send small binary packets across the socket with small intervals in between. The binary packets are Erlang terms marshalled with term_to_binary/1. The problem is that sometimes only some of the packets arrive in the other end even though gen_tcp:send/2 returns 'ok' for all packets. I have tried setting all kinds of properties for the sockets e.g. 'nodelay', {sndbuf, 0}, {recbuf, 0} but the only thing that works is if I "sleep" (timer:sleep/1) 1 millisec between each gen_tcp_send/2 call. The error occurs both in Windows XP and in Linux and when running the system on a network or on localhost. I know that gen_tcp:send/2 returns ok even though the socket in the other end is closed, but I have checked and it isn't closed. Can the sockets get overloaded? Not that I have ever heard about that... Or is it a buffer problem? What am I doing wrong? (Sorry for not providing any code, but it is difficult to extract a useful snippet.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From anupam.kapoor@REDACTED Tue Jun 24 16:40:08 2008 From: anupam.kapoor@REDACTED (anupamk) Date: Tue, 24 Jun 2008 07:40:08 -0700 (PDT) Subject: [erlang-questions] re moving nth element from a list Message-ID: <18092530.post@talk.nabble.com> hi all, can you please let me know what would be a more efficient approach to removing nth element from a list. here are 2 versions that i have come up with: ,---- | %% remove nth element from a list | remove_nth(L, N) -> | do_remove_nth(L, N, 1, []). | | do_remove_nth([_ | Rest], N, Start, Result) when N =:= Start -> | do_remove_nth(Rest, N, Start+1, Result); | | do_remove_nth([First | Rest], N, Start, Result) -> | do_remove_nth(Rest, N, Start+1, [First | Result]); | | do_remove_nth([], _, _, Result) -> | lists:reverse(Result). | | remove_nth_2(L, N) -> | {Split_left, Split_right} = lists:split(N, L), | | lists:append(lists:sublist(Split_left, length(Split_left)-1), | lists:sublist(Split_right, length(Split_right))). `---- i will be more than happy to know of other approaches to do the same too. thanks kind regards anupam ps: i posted this message earlier, but i didn't see it appear on the list. apologies if there are multiple copies of it. -- View this message in context: http://www.nabble.com/removing-nth-element-from-a-list-tp18092530p18092530.html Sent from the Erlang Questions mailing list archive at Nabble.com. From jesper.louis.andersen@REDACTED Tue Jun 24 16:51:12 2008 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 24 Jun 2008 16:51:12 +0200 Subject: [erlang-questions] re moving nth element from a list In-Reply-To: <18092530.post@talk.nabble.com> References: <18092530.post@talk.nabble.com> Message-ID: <56a0a2840806240751h4336cd26ga09fd9e267d04aba@mail.gmail.com> On Tue, Jun 24, 2008 at 4:40 PM, anupamk wrote: > > hi all, > > can you please let me know what would be a more efficient approach to > removing nth element from a list. If your list is rather small, say N < 20, then I would suggest you use something along the lines of your first implementation. If your list is extremely big and performance matters, then I suggest you look for another data structure. With a list there is only so much you can do: You have to dig into the list in order to get to the Nth element and that will cost you O(n). From erlang-questions_efine@REDACTED Tue Jun 24 19:05:03 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Tue, 24 Jun 2008 13:05:03 -0400 Subject: [erlang-questions] http:request/5 returns econnrefused for localhost In-Reply-To: References: <6c2563b20806231808r18fffaabu8120134b4ba1da30@mail.gmail.com> <5CA9B9A5-C9EB-4E17-9D63-8C2D18834DEE@apache.org> <20080624094811.GB4827@cs.uni-bonn.de> Message-ID: <6c2563b20806241005k6c7be2c5y518e27176019a993@mail.gmail.com> Thanks! I modified my /etc/hosts file as follows: #::1 localhost ip6-localhost ip6-loopback ::1 ip6-localhost ip6-loopback Everything then worked correctly after that. That's probably not the best way to do things, so I looked everywhere to try to find how to enable IPv6 in Yaws and could find no reference to it. I even looked in the Yaws source code. Can you clue me in please? On Tue, Jun 24, 2008 at 5:58 AM, Jan Lehnardt wrote: > > On Jun 24, 2008, at 11:48, Ignatios Souvatzis wrote: > > > On Tue, Jun 24, 2008 at 08:43:35AM +0200, Jan Lehnardt wrote: > > > >> We have seen the same issue with CouchDB. What we found out > >> is that in our case localhost was not only resolving to 127.0.0.1 > >> (IPv4) > >> but also ::1 (IPv6) and that http:request() would try to connect to > >> ::1 where no service was listening. > >> > >> Try disabling IPv6 networking to verify this. > > > > Uhm... the fix would be to enable IPv6 in his Yaws. After all, it's > > easy; > > the IPv6 support in Erlang is pretty good. > > Right, that's the other solution :) My previous mail was sent pre- > coffee :) > > Cheers > Jan > -- > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nick@REDACTED Tue Jun 24 19:45:24 2008 From: nick@REDACTED (Nick Gerakines) Date: Tue, 24 Jun 2008 10:45:24 -0700 Subject: [erlang-questions] re moving nth element from a list In-Reply-To: <56a0a2840806240751h4336cd26ga09fd9e267d04aba@mail.gmail.com> References: <18092530.post@talk.nabble.com> <56a0a2840806240751h4336cd26ga09fd9e267d04aba@mail.gmail.com> Message-ID: There might be a better way to do it, but this is what I use: -module(remelem). -compile(export_all). remove_element(1, List) -> [_ | TheRest] = List, TheRest; remove_element(ElemPos, List) when length(List) == ElemPos -> [_ | TheRest] = lists:reverse(List), lists:reverse(TheRest); remove_element(ElemPos, List) -> {ListA, ListB} = lists:split(ElemPos - 1, List), [_, ElemB | ListC] = ListB, ListResB = [ElemB | ListC], ListA ++ ListResB. # Nick Gerakines On Tue, Jun 24, 2008 at 7:51 AM, Jesper Louis Andersen wrote: > On Tue, Jun 24, 2008 at 4:40 PM, anupamk wrote: >> >> hi all, >> >> can you please let me know what would be a more efficient approach to >> removing nth element from a list. > > If your list is rather small, say N < 20, then I would suggest you use > something along the > lines of your first implementation. If your list is extremely big and > performance matters, then > I suggest you look for another data structure. With a list there is > only so much you can do: You > have to dig into the list in order to get to the Nth element and that > will cost you O(n). > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From twoggle@REDACTED Tue Jun 24 19:58:59 2008 From: twoggle@REDACTED (Tim Fletcher) Date: Tue, 24 Jun 2008 10:58:59 -0700 (PDT) Subject: [erlang-questions] how: info on how to traverse abstract format trees? In-Reply-To: <95be1d3b0806240656p655ebd87i7d70e55183c9ab05@mail.gmail.com> References: <14f0e3620806240510sb9430c0k31ca96375217dda5@mail.gmail.com> <79854fe0-10a0-4cd6-990f-d9f7d38af651@27g2000hsf.googlegroups.com> <95be1d3b0806240656p655ebd87i7d70e55183c9ab05@mail.gmail.com> Message-ID: <18c525f4-08a1-4665-84bb-4211775ac8db@p25g2000hsf.googlegroups.com> > You can use syntax_tools. They let you construct and take apart ASTs without > having to know the exact representation. That makes my code considerably clearer. Thanks for the pointer. From scott.gregory@REDACTED Tue Jun 24 19:09:04 2008 From: scott.gregory@REDACTED (Scott Gregory) Date: Tue, 24 Jun 2008 13:09:04 -0400 Subject: [erlang-questions] Beginner: Windows UDP multicast receive Message-ID: <000c01c8d61d$01922610$04b67230$@gregory@infiniumgroup.com> Hello (I also posted this to the "Advanced Erlang/OTP" forum - apologies if either is the wrong place). Absolute Erlang newbie here, trying to do a UDP multicast receive on Windows as a useful/interesting test case for my purposes. Version is: Erlang (BEAM) emulator version 5.6.3 [smp:2] [async-threads:0] Below is my code - What I am trying to do is listen on "address" 224.0.66.66 "port" 6666. My computer's regular IP address is 192.168.6.49 on our internal network. The code "runs" ok, but never seems to receive anything, I always get the "huh" output, and I know from other handlers that there are all sorts of packets flying by on this multicast address. Help?! Scott Code: -module(listen). -export([init/0, loop/1]). loop(Socket) -> io:format("in-loop~n"), receive {udp, Socket, Host, Port, Bin} -> io:format("host",Host), io:format("Port: ", Port), io:format("BinSize: ", size(Bin)), loop(Socket); {udp, _ } -> io:format("got one") after 2000 -> io:format("huh?~n") end, gen_udp:close(Socket). init() -> {ok, Socket} = gen_udp:open(6666, [binary, {reuseaddr,true}, {multicast_loop,true}, {ip,{192,168,6,49}}, {add_membership,{{224,0,66,66},{192,168,6,49}}}]), loop(Socket). From nc@REDACTED Tue Jun 24 20:24:42 2008 From: nc@REDACTED (Nicolas Charpentier) Date: Tue, 24 Jun 2008 20:24:42 +0200 Subject: [erlang-questions] Slave nodes without proxied I/O? In-Reply-To: <4860E042.8050803@niclux.org> References: <73AF1508-8DE3-437F-8208-521C84A35683@ketralnis.com> <4860E042.8050803@niclux.org> Message-ID: <48613BEA.7060301@charpi.net> Nicolas Niclausse wrote: > David King ecrivait le 01.05.2008 20:34: >>> By default, nodes started as slave nodes with slave:start/1 have their >>> I/O proxied through the master node that started them. (That is, I/O >>> operations done with the 'file' module are done on the master's >>> filesystem, not the slaves'). >>> Is there any way to turn this off? > I'm having the same problem as you: i want to start a slave without > proxied I/O. > > After some digging, i discover that you only have to remove the "-master > node()" argument in the mk_cmd fun to do that (slave.erl module). > Hi, Just a question. I might be wrong but I have always thought that I/O was proxied because they (master and slave nodes) was sharing the same group_leader. If I'm right, we might be able to change the group_leader of processes on the slave node. Regards, Nicolas Charpentier From erlang-questions_efine@REDACTED Tue Jun 24 20:43:05 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Tue, 24 Jun 2008 14:43:05 -0400 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <97619b170806192135r5504e3a4s80179885510971d9@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> <399E231E-E06E-4FCE-901A-24C6B7B2DFA8@udc.es> <97619b170806192135r5504e3a4s80179885510971d9@mail.gmail.com> Message-ID: <6c2563b20806241143x1fd06acdq34b14b04058428c1@mail.gmail.com> I wrote a small benchmark in Erlang to see how fast I could get socket communications to go. All the benchmark does is pump the same buffer to a socket for (by default) 10 seconds. It uses {active, once} each time, just like you do. Server TCP options: {active, once}, {reuseaddr, true}, {packet, 0}, {packet_size, 65536}, {recbuf, 1000000} Client TCP options: {packet, raw}, {packet_size, 65536}, {sndbuf, 1024 * 1024}, {send_timeout, 3000} Here are some results using Erlang R12B-3 (erl +K true in the Linux version): Linux (Ubuntu 8.10 x86_64, Intel Core 2 Q6600, 8 GB): - Using localhost (127.0.0.1): 7474.14 MB in 10.01 secs (746.66 MB/sec) - Using 192.168.x.x IP address: 8064.94 MB in 10.00 secs (806.22 MB/sec) [Don't ask me why it's faster than using loopback, I repeated the tests and got the same result] Windows XP SP3 (32 bits), Intel Core 2 Duo E6600: - Using loopback: 2166.97 MB in 10.02 secs (216.35 MB/sec) - Using 192.168.x.x IP address: 2140.72 MB in 10.02 secs (213.75 MB/sec) - On Gigabit Ethernet to the Q6600 Linux box: 1063.61 MB in 10.02 secs (106.17 MB/sec) using non-jumbo frames. I don't think my router supports jumbo frames. There's undoubtedly a huge discrepancy between the two systems, whether because of kernel poll in Linux, or that it's 64 bits, or unoptimized Windows TCP/IP flags, I don't know. I don't believe it's the number of CPUs (there's only 1 process sending and one receiving), or the CPU speed (they are both 2.4 GHz Core 2s). Maybe some Erlang TCP/IP gurus could comment. I've attached the code for interest. It's not supposed to be production quality, so please don't beat me up :) although I am always open to suggestions for improvement. If you do improve it, I'd like to see what you've done. Maybe there is another simple Erlang tcp benchmark program out there (i.e. not Tsung), but I couldn't find one in a cursory Google search. To run: VM1: tb_server:start(Port, Opts). tb_server:stop() to stop. Port = integer() Opts = []|[opt()] opt() = {atom(), term()} (Accepts inet setopts options, too) The server prints out the transfer rate (for simplicity). VM2: tb_client(Host, Port, Opts). Host = atom()|string() hostname or IP address Port, Opts as in tb_server Runs for 10 seconds, sending a 64K buffer as fast as possible to Host/Port. You can change this to 20 seconds (e.g.) by adding the tupls {time_limit, 20000} to Opts. You can change buffer size by adding the tuple {blksize, Bytes} to Opts. 2008/6/20 Rapsey : > All data goes through nginx which acts as a proxy. Its CPU consumption is > never over 1%. > > > Sergej > > > On Thu, Jun 19, 2008 at 9:35 PM, Javier Par?s Fern?ndez < > javierparis@REDACTED> wrote: > >> >> El 19/06/2008, a las 20:06, Rapsey escribi?: >> >> It loops from another module, that way I can update the code at any time >>> without disrupting anything. >>> The packets are generally a few hundred bytes big, except keyframes which >>> tend to be in the kB range. I haven't tried looking with wireshark. Still >>> it seems a bit odd that a large CPU consumption would be the symptom. The >>> traffic is strictly one way. Either someone is sending the stream or >>> receiving it. >>> The transmit could of course be written with a passive receive, but the >>> code would be significantly uglier. I'm sure someone here knows if setting >>> {active, once} every packet is CPU intensive or not. >>> It seems the workings of gen_tcp is quite platform dependent. If I run >>> the code in windows, sending more than 128 bytes per gen_tcp call >>> significantly decreases network output. >>> Oh and I forgot to mention I use R12B-3. >>> >> >> Hi, >> >> Without being an expert. >> >> 200-300 mb/s in small (hundreds of bytes) packets means a *lot* of system >> calls if you are doing a gen_tcp:send for each one. If you buffer 3 packets, >> you are reducing that by a factor of 3 :). I'd try to do an small test doing >> the same thing in C and compare the results. I think it will also eat a lot >> of CPU. >> >> About the proxy CPU... I'm a bit lost about it, but speculating wildly it >> is possible that the time spent doing the system calls that gen_tcp is doing >> is added to the proxy CPU process. >> >> Regards. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tcp_bench.tgz Type: application/x-gzip Size: 4553 bytes Desc: not available URL: From mog-lists@REDACTED Tue Jun 24 20:50:43 2008 From: mog-lists@REDACTED (mog) Date: Tue, 24 Jun 2008 13:50:43 -0500 Subject: [erlang-questions] Beginner: Windows UDP multicast receive In-Reply-To: <000c01c8d61d$01922610$04b67230$@gregory@infiniumgroup.com> References: <000c01c8d61d$01922610$04b67230$@gregory@infiniumgroup.com> Message-ID: <48614203.8080009@rldn.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I believe you are missing {active, false}. On my gnu/linux box the following works ~ {ok, Socket} = gen_udp:open(Port, [binary, {active, false}, {reuseaddr, true}, {ip, Addr}, {add_membership, {Addr, LAddr}}]), hope this helps mog -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIYUICeq+tARrxhnsRAgkUAKCd0xQtNPk8Qix/U4ExFKo59VwizwCeLRIw aBEvP9W8M8cpJNjmI8gG2gE= =t8VN -----END PGP SIGNATURE----- From rapsey@REDACTED Tue Jun 24 21:00:42 2008 From: rapsey@REDACTED (Rapsey) Date: Tue, 24 Jun 2008 21:00:42 +0200 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <6c2563b20806241143x1fd06acdq34b14b04058428c1@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> <399E231E-E06E-4FCE-901A-24C6B7B2DFA8@udc.es> <97619b170806192135r5504e3a4s80179885510971d9@mail.gmail.com> <6c2563b20806241143x1fd06acdq34b14b04058428c1@mail.gmail.com> Message-ID: <97619b170806241200t6a1e8c35vd53494ad9f3806ff@mail.gmail.com> You're using very large packets. I think the results would be much more telling if the packets would be a few kB at most. That is closer to most real life situations. Sergej On Tue, Jun 24, 2008 at 8:43 PM, Edwin Fine wrote: > I wrote a small benchmark in Erlang to see how fast I could get socket > communications to go. All the benchmark does is pump the same buffer to a > socket for (by default) 10 seconds. It uses {active, once} each time, just > like you do. > > Server TCP options: > {active, once}, > {reuseaddr, true}, > {packet, 0}, > {packet_size, 65536}, > {recbuf, 1000000} > > Client TCP options: > {packet, raw}, > {packet_size, 65536}, > {sndbuf, 1024 * 1024}, > {send_timeout, 3000} > > Here are some results using Erlang R12B-3 (erl +K true in the Linux > version): > > Linux (Ubuntu 8.10 x86_64, Intel Core 2 Q6600, 8 GB): > - Using localhost (127.0.0.1): 7474.14 MB in 10.01 secs (746.66 MB/sec) > - Using 192.168.x.x IP address: 8064.94 MB in 10.00 secs (806.22 MB/sec) > [Don't ask me why it's faster than using loopback, I repeated the tests and > got the same result] > > Windows XP SP3 (32 bits), Intel Core 2 Duo E6600: > - Using loopback: 2166.97 MB in 10.02 secs (216.35 MB/sec) > - Using 192.168.x.x IP address: 2140.72 MB in 10.02 secs (213.75 MB/sec) > - On Gigabit Ethernet to the Q6600 Linux box: 1063.61 MB in 10.02 secs > (106.17 MB/sec) using non-jumbo frames. I don't think my router supports > jumbo frames. > > There's undoubtedly a huge discrepancy between the two systems, whether > because of kernel poll in Linux, or that it's 64 bits, or unoptimized > Windows TCP/IP flags, I don't know. I don't believe it's the number of CPUs > (there's only 1 process sending and one receiving), or the CPU speed (they > are both 2.4 GHz Core 2s). > > Maybe some Erlang TCP/IP gurus could comment. > > I've attached the code for interest. It's not supposed to be production > quality, so please don't beat me up :) although I am always open to > suggestions for improvement. If you do improve it, I'd like to see what > you've done. Maybe there is another simple Erlang tcp benchmark program out > there (i.e. not Tsung), but I couldn't find one in a cursory Google search. > > To run: > > VM1: > > tb_server:start(Port, Opts). > tb_server:stop() to stop. > > Port = integer() > Opts = []|[opt()] > opt() = {atom(), term()} (Accepts inet setopts options, too) > > The server prints out the transfer rate (for simplicity). > > VM2: > tb_client(Host, Port, Opts). > > Host = atom()|string() hostname or IP address > Port, Opts as in tb_server > > Runs for 10 seconds, sending a 64K buffer as fast as possible to Host/Port. > You can change this to 20 seconds (e.g.) by adding the tupls {time_limit, > 20000} to Opts. > You can change buffer size by adding the tuple {blksize, Bytes} to Opts. > > 2008/6/20 Rapsey : > >> All data goes through nginx which acts as a proxy. Its CPU consumption is >> never over 1%. >> >> >> Sergej >> >> >> On Thu, Jun 19, 2008 at 9:35 PM, Javier Par?s Fern?ndez < >> javierparis@REDACTED> wrote: >> >>> >>> El 19/06/2008, a las 20:06, Rapsey escribi?: >>> >>> It loops from another module, that way I can update the code at any time >>>> without disrupting anything. >>>> The packets are generally a few hundred bytes big, except keyframes >>>> which tend to be in the kB range. I haven't tried looking with wireshark. >>>> Still it seems a bit odd that a large CPU consumption would be the symptom. >>>> The traffic is strictly one way. Either someone is sending the stream or >>>> receiving it. >>>> The transmit could of course be written with a passive receive, but the >>>> code would be significantly uglier. I'm sure someone here knows if setting >>>> {active, once} every packet is CPU intensive or not. >>>> It seems the workings of gen_tcp is quite platform dependent. If I run >>>> the code in windows, sending more than 128 bytes per gen_tcp call >>>> significantly decreases network output. >>>> Oh and I forgot to mention I use R12B-3. >>>> >>> >>> Hi, >>> >>> Without being an expert. >>> >>> 200-300 mb/s in small (hundreds of bytes) packets means a *lot* of >>> system calls if you are doing a gen_tcp:send for each one. If you buffer 3 >>> packets, you are reducing that by a factor of 3 :). I'd try to do an small >>> test doing the same thing in C and compare the results. I think it will also >>> eat a lot of CPU. >>> >>> About the proxy CPU... I'm a bit lost about it, but speculating wildly it >>> is possible that the time spent doing the system calls that gen_tcp is doing >>> is added to the proxy CPU process. >>> >>> Regards. >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmercer@REDACTED Tue Jun 24 22:02:11 2008 From: dmercer@REDACTED (David Mercer) Date: Tue, 24 Jun 2008 15:02:11 -0500 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <97619b170806241200t6a1e8c35vd53494ad9f3806ff@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com><6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com><97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com><399E231E-E06E-4FCE-901A-24C6B7B2DFA8@udc.es><97619b170806192135r5504e3a4s80179885510971d9@mail.gmail.com><6c2563b20806241143x1fd06acdq34b14b04058428c1@mail.gmail.com> <97619b170806241200t6a1e8c35vd53494ad9f3806ff@mail.gmail.com> Message-ID: <019501c8d635$31455fe0$f21ea8c0@SSI.CORP> I tried some alternative block sizes (using the blksize option). I found that from 1 to somewhere around??maybe a bit short of??1000 bytes, the test was able to send about 300,000 blocks in 10 seconds regardless of size. (That means, 0.03 MB/sec for block size of 1, 0.3 MB/sec for block size of 10, 3 MB/sec for block size of 100, etc.) I suspect the system was CPU bound at those levels. Above 1000, the number of blocks sent seemed to decrease, though this was more than offset by the increased size of the blocks. Above about 10,000 byte blocks (may have been less, I didn?t check any value between 4,000 and 10,000), however, performance peaked and block size no longer mattered: it always sent between 70 and 80 MB/sec. My machine is clearly slower than Edwin?s DBM _____ From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Rapsey Sent: Tuesday, June 24, 2008 14:01 To: erlang-questions@REDACTED Subject: Re: [erlang-questions] why is gen_tcp:send slow? You're using very large packets. I think the results would be much more telling if the packets would be a few kB at most. That is closer to most real life situations. Sergej On Tue, Jun 24, 2008 at 8:43 PM, Edwin Fine wrote: I wrote a small benchmark in Erlang to see how fast I could get socket communications to go. All the benchmark does is pump the same buffer to a socket for (by default) 10 seconds. It uses {active, once} each time, just like you do. Server TCP options: {active, once}, {reuseaddr, true}, {packet, 0}, {packet_size, 65536}, {recbuf, 1000000} Client TCP options: {packet, raw}, {packet_size, 65536}, {sndbuf, 1024 * 1024}, {send_timeout, 3000} Here are some results using Erlang R12B-3 (erl +K true in the Linux version): Linux (Ubuntu 8.10 x86_64, Intel Core 2 Q6600, 8 GB): - Using localhost (127.0.0.1): 7474.14 MB in 10.01 secs (746.66 MB/sec) - Using 192.168.x.x IP address: 8064.94 MB in 10.00 secs (806.22 MB/sec) [Don't ask me why it's faster than using loopback, I repeated the tests and got the same result] Windows XP SP3 (32 bits), Intel Core 2 Duo E6600: - Using loopback: 2166.97 MB in 10.02 secs (216.35 MB/sec) - Using 192.168.x.x IP address: 2140.72 MB in 10.02 secs (213.75 MB/sec) - On Gigabit Ethernet to the Q6600 Linux box: 1063.61 MB in 10.02 secs (106.17 MB/sec) using non-jumbo frames. I don't think my router supports jumbo frames. There's undoubtedly a huge discrepancy between the two systems, whether because of kernel poll in Linux, or that it's 64 bits, or unoptimized Windows TCP/IP flags, I don't know. I don't believe it's the number of CPUs (there's only 1 process sending and one receiving), or the CPU speed (they are both 2.4 GHz Core 2s). Maybe some Erlang TCP/IP gurus could comment. I've attached the code for interest. It's not supposed to be production quality, so please don't beat me up :) although I am always open to suggestions for improvement. If you do improve it, I'd like to see what you've done. Maybe there is another simple Erlang tcp benchmark program out there (i.e. not Tsung), but I couldn't find one in a cursory Google search. To run: VM1: tb_server:start(Port, Opts). tb_server:stop() to stop. Port = integer() Opts = []|[opt()] opt() = {atom(), term()} (Accepts inet setopts options, too) The server prints out the transfer rate (for simplicity). VM2: tb_client(Host, Port, Opts). Host = atom()|string() hostname or IP address Port, Opts as in tb_server Runs for 10 seconds, sending a 64K buffer as fast as possible to Host/Port. You can change this to 20 seconds (e.g.) by adding the tupls {time_limit, 20000} to Opts. You can change buffer size by adding the tuple {blksize, Bytes} to Opts. 2008/6/20 Rapsey : All data goes through nginx which acts as a proxy. Its CPU consumption is never over 1%. Sergej On Thu, Jun 19, 2008 at 9:35 PM, Javier Par?s Fern?ndez wrote: El 19/06/2008, a las 20:06, Rapsey escribi?: It loops from another module, that way I can update the code at any time without disrupting anything. The packets are generally a few hundred bytes big, except keyframes which tend to be in the kB range. I haven't tried looking with wireshark. Still it seems a bit odd that a large CPU consumption would be the symptom. The traffic is strictly one way. Either someone is sending the stream or receiving it. The transmit could of course be written with a passive receive, but the code would be significantly uglier. I'm sure someone here knows if setting {active, once} every packet is CPU intensive or not. It seems the workings of gen_tcp is quite platform dependent. If I run the code in windows, sending more than 128 bytes per gen_tcp call significantly decreases network output. Oh and I forgot to mention I use R12B-3. Hi, Without being an expert. 200-300 mb/s in small (hundreds of bytes) packets means a *lot* of system calls if you are doing a gen_tcp:send for each one. If you buffer 3 packets, you are reducing that by a factor of 3 :). I'd try to do an small test doing the same thing in C and compare the results. I think it will also eat a lot of CPU. About the proxy CPU... I'm a bit lost about it, but speculating wildly it is possible that the time spent doing the system calls that gen_tcp is doing is added to the proxy CPU process. Regards. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://www.erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From fritchie@REDACTED Tue Jun 24 22:31:21 2008 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Tue, 24 Jun 2008 15:31:21 -0500 Subject: [erlang-questions] (how) Can a tcp socket get "overloaded"? In-Reply-To: Message of "Tue, 24 Jun 2008 16:24:45 +0200." Message-ID: <69852.1214339481@snookles.snookles.com> Nicholas, are you assuming that if you use: %% Client side Size0 = size(Bin), gen_tcp:send(Socket, Bin), And: %% Server side {ok, Bin1} = gen_tcp:recv(ConnectedSocket, 0), Size1 = size(Bin1), %% or perhaps we're using active mode? Size2 = receive {tcp, ConnectedSocket, B} -> size(B) end, ... that Size0 is exactly equal to Size1 or Size2? If yes, then you're mis-using TCP, because TCP communication is a pure byte steam and does not have a notion of message boundaries. Any delays on the sender's side to try to keep hunks-of-the-stream's-bytes separate will eventually fail when the sender's load increases, or if the OS on either side feels like rebuffering, or when the network in the middle drops a packet and a retransmission juggles the receiver's buffering efforts, or ... -Scott From psa@REDACTED Tue Jun 24 21:45:24 2008 From: psa@REDACTED (=?ISO-8859-1?Q?Paulo_S=E9rgio_Almeida?=) Date: Tue, 24 Jun 2008 20:45:24 +0100 Subject: [erlang-questions] re moving nth element from a list In-Reply-To: References: <18092530.post@talk.nabble.com><56a0a2840806240751h4336cd26ga09f d9e267d04aba@mail.gmail.com> Message-ID: <48614ED4.1080909@di.uminho.pt> This is one of those cases where it is not necessarily better to use tail calls; one must measure what happens. Here are two versions: Body recursive: remove(_, []) -> []; remove(1, [_|T]) -> T; remove(N, [H|T]) -> [H | remove(N-1, T)]. Tail recursive: remove2(N, L) -> remove2(N, L, []). remove2(_, [], Acc) -> lists:reverse(Acc); remove2(1, [_|T], Acc) -> lists:reverse(Acc, T); remove2(N, [H|T], Acc) -> remove2(N-1, T, [H|Acc]). The second one seems to be better for removing near the end of a large list, due to lists:reverse being builtin. Regards, Paulo Nick Gerakines wrote: > There might be a better way to do it, but this is what I use: > > -module(remelem). > -compile(export_all). > > remove_element(1, List) -> > [_ | TheRest] = List, TheRest; > remove_element(ElemPos, List) when length(List) == ElemPos -> > [_ | TheRest] = lists:reverse(List), lists:reverse(TheRest); > remove_element(ElemPos, List) -> > {ListA, ListB} = lists:split(ElemPos - 1, List), > [_, ElemB | ListC] = ListB, > ListResB = [ElemB | ListC], > ListA ++ ListResB. > > # Nick Gerakines > > On Tue, Jun 24, 2008 at 7:51 AM, Jesper Louis Andersen > wrote: >> On Tue, Jun 24, 2008 at 4:40 PM, anupamk wrote: >>> hi all, >>> >>> can you please let me know what would be a more efficient approach to >>> removing nth element from a list. >> If your list is rather small, say N < 20, then I would suggest you use >> something along the >> lines of your first implementation. If your list is extremely big and >> performance matters, then >> I suggest you look for another data structure. With a list there is >> only so much you can do: You >> have to dig into the list in order to get to the Nth element and that >> will cost you O(n). >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From erlang-questions_efine@REDACTED Tue Jun 24 23:16:18 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Tue, 24 Jun 2008 17:16:18 -0400 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <019501c8d635$31455fe0$f21ea8c0@SSI.CORP> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> <399E231E-E06E-4FCE-901A-24C6B7B2DFA8@udc.es> <97619b170806192135r5504e3a4s80179885510971d9@mail.gmail.com> <6c2563b20806241143x1fd06acdq34b14b04058428c1@mail.gmail.com> <97619b170806241200t6a1e8c35vd53494ad9f3806ff@mail.gmail.com> <019501c8d635$31455fe0$f21ea8c0@SSI.CORP> Message-ID: <6c2563b20806241416r71fdb225v90930bead9c95d65@mail.gmail.com> David, Thanks for trying out the benchmark. With my limited knowledge of TCP/IP, I believe you are seeing the 300,000 limit because TCP/IP requires acknowledgements to each packet, and although it can batch up multiple acknowledgements in one packet, there is a theoretical limit of packets per seconds beyond which it cannot go due to the laws of physics. I understand that limit is determined by the Round-Trip Time (RTT), which can be shown by ping. On my system, pinging 127.0.0.1gives a minimum RTT of 0.018 ms (out of 16 pings). That means that the maximum number of packets that can make it to and dest and back per second is 1/0.000018 seconds, or 55555 packets per second. The TCP/IP stack is evidently packing 5 or 6 blocks into each packet to get the 300K blocks/sec you are seeing. Using Wireshark or Ethereal would confirm this. I am guessing that this means that the TCP window is about 6 * 1000 bytes or 6KB. What I neglected to tell this group is that I have modified the Linux sysctl.conf as follows, which might have had an effect (like I said, I am not an expert): # increase Linux autotuning TCP buffer limits # min, default, and max number of bytes to use # set max to at least 4MB, or higher if you use very high BDP paths net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 32768 16777216 When I have more time, I will vary a number of different Erlang TCP/IP parameters and get a data set together that gives a broader picture of the effect of the parameters. Thanks again for taking the time. 2008/6/24 David Mercer : > I tried some alternative block sizes (using the blksize option). I found > that from 1 to somewhere around??maybe a bit short of??1000 bytes, the test > was able to send about 300,000 blocks in 10 seconds regardless of size. > (That means, 0.03 MB/sec for block size of 1, 0.3 MB/sec for block size of > 10, 3 MB/sec for block size of 100, etc.) I suspect the system was CPU > bound at those levels. > > > > Above 1000, the number of blocks sent seemed to decrease, though this was > more than offset by the increased size of the blocks. Above about 10,000 > byte blocks (may have been less, I didn't check any value between 4,000 and > 10,000), however, performance peaked and block size no longer mattered: it > always sent between 70 and 80 MB/sec. My machine is clearly slower than > Edwin's? > > > > DBM > > > ------------------------------ > > *From:* erlang-questions-bounces@REDACTED [mailto: > erlang-questions-bounces@REDACTED] *On Behalf Of *Rapsey > *Sent:* Tuesday, June 24, 2008 14:01 > *To:* erlang-questions@REDACTED > *Subject:* Re: [erlang-questions] why is gen_tcp:send slow? > > > > You're using very large packets. I think the results would be much more > telling if the packets would be a few kB at most. That is closer to most > real life situations. > > > Sergej > > On Tue, Jun 24, 2008 at 8:43 PM, Edwin Fine < > erlang-questions_efine@REDACTED> wrote: > > I wrote a small benchmark in Erlang to see how fast I could get socket > communications to go. All the benchmark does is pump the same buffer to a > socket for (by default) 10 seconds. It uses {active, once} each time, just > like you do. > > Server TCP options: > {active, once}, > {reuseaddr, true}, > {packet, 0}, > {packet_size, 65536}, > {recbuf, 1000000} > > Client TCP options: > {packet, raw}, > {packet_size, 65536}, > {sndbuf, 1024 * 1024}, > {send_timeout, 3000} > > Here are some results using Erlang R12B-3 (erl +K true in the Linux > version): > > Linux (Ubuntu 8.10 x86_64, Intel Core 2 Q6600, 8 GB): > - Using localhost (127.0.0.1): 7474.14 MB in 10.01 secs (746.66 MB/sec) > - Using 192.168.x.x IP address: 8064.94 MB in 10.00 secs (806.22 MB/sec) > [Don't ask me why it's faster than using loopback, I repeated the tests and > got the same result] > > Windows XP SP3 (32 bits), Intel Core 2 Duo E6600: > - Using loopback: 2166.97 MB in 10.02 secs (216.35 MB/sec) > - Using 192.168.x.x IP address: 2140.72 MB in 10.02 secs (213.75 MB/sec) > - On Gigabit Ethernet to the Q6600 Linux box: 1063.61 MB in 10.02 secs > (106.17 MB/sec) using non-jumbo frames. I don't think my router supports > jumbo frames. > > There's undoubtedly a huge discrepancy between the two systems, whether > because of kernel poll in Linux, or that it's 64 bits, or unoptimized > Windows TCP/IP flags, I don't know. I don't believe it's the number of CPUs > (there's only 1 process sending and one receiving), or the CPU speed (they > are both 2.4 GHz Core 2s). > > Maybe some Erlang TCP/IP gurus could comment. > > I've attached the code for interest. It's not supposed to be production > quality, so please don't beat me up :) although I am always open to > suggestions for improvement. If you do improve it, I'd like to see what > you've done. Maybe there is another simple Erlang tcp benchmark program out > there (i.e. not Tsung), but I couldn't find one in a cursory Google search. > > To run: > > VM1: > > tb_server:start(Port, Opts). > tb_server:stop() to stop. > > Port = integer() > Opts = []|[opt()] > opt() = {atom(), term()} (Accepts inet setopts options, too) > > The server prints out the transfer rate (for simplicity). > > VM2: > tb_client(Host, Port, Opts). > > Host = atom()|string() hostname or IP address > Port, Opts as in tb_server > > Runs for 10 seconds, sending a 64K buffer as fast as possible to Host/Port. > You can change this to 20 seconds (e.g.) by adding the tupls {time_limit, > 20000} to Opts. > You can change buffer size by adding the tuple {blksize, Bytes} to Opts. > > 2008/6/20 Rapsey : > > All data goes through nginx which acts as a proxy. Its CPU consumption is > never over 1%. > > > Sergej > > > > On Thu, Jun 19, 2008 at 9:35 PM, Javier Par?s Fern?ndez < > javierparis@REDACTED> wrote: > > > El 19/06/2008, a las 20:06, Rapsey escribi?: > > > > It loops from another module, that way I can update the code at any time > without disrupting anything. > The packets are generally a few hundred bytes big, except keyframes which > tend to be in the kB range. I haven't tried looking with wireshark. Still > it seems a bit odd that a large CPU consumption would be the symptom. The > traffic is strictly one way. Either someone is sending the stream or > receiving it. > The transmit could of course be written with a passive receive, but the > code would be significantly uglier. I'm sure someone here knows if setting > {active, once} every packet is CPU intensive or not. > It seems the workings of gen_tcp is quite platform dependent. If I run the > code in windows, sending more than 128 bytes per gen_tcp call significantly > decreases network output. > Oh and I forgot to mention I use R12B-3. > > > > Hi, > > Without being an expert. > > 200-300 mb/s in small (hundreds of bytes) packets means a *lot* of system > calls if you are doing a gen_tcp:send for each one. If you buffer 3 packets, > you are reducing that by a factor of 3 :). I'd try to do an small test doing > the same thing in C and compare the results. I think it will also eat a lot > of CPU. > > About the proxy CPU... I'm a bit lost about it, but speculating wildly it > is possible that the time spent doing the system calls that gen_tcp is doing > is added to the proxy CPU process. > > Regards. > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dougedmunds@REDACTED Tue Jun 24 23:34:31 2008 From: dougedmunds@REDACTED (Doug Edmunds) Date: Tue, 24 Jun 2008 14:34:31 -0700 Subject: [erlang-questions] My favorite syntax error: semicolon before end Message-ID: In the docs: at /doc/design_principles/fsm.html AKA OTP Design Principles 3.2 Version 5.6.3 Example: Section 3.2 1> c(code_lock). ./code_lock.erl:26: syntax error before: 'end' ./code_lock.erl:6: function locked/2 undefined ./code_lock.erl:29: function do_lock/0 undefined ./code_lock.erl:2: Warning: undefined callback function code_change/4 (behaviour 'gen_fsm') ./code_lock.erl:2: Warning: undefined callback function handle_event/3 (behaviour 'gen_fsm') ./code_lock.erl:2: Warning: undefined callback function handle_info/3 (behaviour 'gen_fsm') ./code_lock.erl:2: Warning: undefined callback function handle_sync_event/4 (behaviour 'gen_fsm') ./code_lock.erl:2: Warning: undefined callback function terminate/3 (behaviour 'gen_fsm') error -dae From mog-lists@REDACTED Wed Jun 25 00:00:06 2008 From: mog-lists@REDACTED (mog) Date: Tue, 24 Jun 2008 17:00:06 -0500 Subject: [erlang-questions] My favorite syntax error: semicolon before end In-Reply-To: References: Message-ID: <48616E66.10705@rldn.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I fixed this error a year ago.... http://www.erlang.org/pipermail/erlang-bugs/2007-September/000443.html I guess they just are to busy to look at patches from community.... Mog -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIYW5leq+tARrxhnsRAkmNAJ9ihAwlI93SJnRQn6EOqs+8mtpb2ACeNavE AyZ0iz3upXqp646fL8iYjC8= =uwHt -----END PGP SIGNATURE----- From dougedmunds@REDACTED Wed Jun 25 00:03:58 2008 From: dougedmunds@REDACTED (Doug Edmunds) Date: Tue, 24 Jun 2008 15:03:58 -0700 Subject: [erlang-questions] My favorite syntax error: semicolon before end In-Reply-To: <48616E66.10705@rldn.net> References: <48616E66.10705@rldn.net> Message-ID: Actually it's there twice. (3.2 and 3.4). Maybe someone can get the RE module to check for semicolons before end and clean up all the docs. :-) From nicholassm@REDACTED Wed Jun 25 00:17:58 2008 From: nicholassm@REDACTED (=?ISO-8859-1?Q?Nicholas_Schultz-M=F8ller?=) Date: Wed, 25 Jun 2008 00:17:58 +0200 Subject: [erlang-questions] (how) Can a tcp socket get "overloaded"? In-Reply-To: <69852.1214339481@snookles.snookles.com> References: <69852.1214339481@snookles.snookles.com> Message-ID: Hi Scott, Thanks - yeah you're right. I forgot to set the 'packet' property which results in Erlang handling any fragmentation and reassembling logical packets (application specific units of data) in the other end. /Nicholas 2008/6/24 Scott Lystig Fritchie : > Nicholas, are you assuming that if you use: > > %% Client side > Size0 = size(Bin), > gen_tcp:send(Socket, Bin), > > And: > > %% Server side > {ok, Bin1} = gen_tcp:recv(ConnectedSocket, 0), > Size1 = size(Bin1), > %% or perhaps we're using active mode? > Size2 = receive > {tcp, ConnectedSocket, B} -> size(B) > end, > > ... that Size0 is exactly equal to Size1 or Size2? If yes, then you're > mis-using TCP, because TCP communication is a pure byte steam and does > not have a notion of message boundaries. Any delays on the sender's > side to try to keep hunks-of-the-stream's-bytes separate will eventually > fail when the sender's load increases, or if the OS on either side feels > like rebuffering, or when the network in the middle drops a packet and a > retransmission juggles the receiver's buffering efforts, or ... > > -Scott > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang-questions_efine@REDACTED Wed Jun 25 00:55:44 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Tue, 24 Jun 2008 18:55:44 -0400 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <48616E93.2010300@softjar.se> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> <399E231E-E06E-4FCE-901A-24C6B7B2DFA8@udc.es> <97619b170806192135r5504e3a4s80179885510971d9@mail.gmail.com> <6c2563b20806241143x1fd06acdq34b14b04058428c1@mail.gmail.com> <97619b170806241200t6a1e8c35vd53494ad9f3806ff@mail.gmail.com> <019501c8d635$31455fe0$f21ea8c0@SSI.CORP> <6c2563b20806241416r71fdb225v90930bead9c95d65@mail.gmail.com> <48616E93.2010300@softjar.se> Message-ID: <6c2563b20806241555k269a0848n302fc7335deb0c23@mail.gmail.com> Johnny, Thanks for the lesson! I am always happy to learn. Like I said, I am not an expert in TCP/IP. What I was writing about when I said that packets are acknowledged is what I saw in Wireshark while trying to understand performance issues. I perhaps should have said "TCP/IP" instead of just "TCP". There were definitely acknowledgements, but I guess they were at the IP level. I wonder what the MSS is for loopback? I think it's about 1536 on my eth0 interface, but not sure. As for RTT, I sent data over a link that had a very long (290ms) RTT, and that definitely limited the rate at which packets could be sent. Can RTT be used to calculate the theoretical maximum traffic that a link can carry? For example, a satellite link with a 400ms RTT but 2 Mbps bandwidth? Ed On Tue, Jun 24, 2008 at 6:00 PM, Johnny Billquist wrote: > No. TCP don't acknowledge every packet. In fact, TCP don't acknowledge > packets as such at all. TCP is not packet based. It's just that if you use > IP as the carrier, IP itself it packet based. > TCP can in theory generate any number of packets per second. However, the > amount of unacknowledged data that can be outstanding at any time is limited > by the transmit window. Each packet carries a window size, which is how much > more data that can be accepted by the reciever. TCP can (is allowed to) send > that much data and no more. > > The RTT calculations are used for figuring out how long to wait before > doing retransmissions. You also normally have a slow start transmission > algorithm which prevents the sender from even using the full window size > from the start, as a way of avoiding congestions. That is used in > combination with a backoff algorithm when retransmissions are needed to > further decrease congestions, but all of this only really comes into effect > if you start loosing data, and TCP actually needs to do retransmissions. > > Another thing you have is an algorithm called Nagle, which tries to collect > small amount of data sent into larger packets before sending it, so that you > don't flood the net with silly small packets. > > One addisional detail is that receivers normally, when the receive buffers > becomes full, don't announce newly freed space immediately, since that is > normally rather small amounts, but instead wait a while, until a larger part > of the receive buffer is free, so that the sender actually can send some > full sized packets once it starts sending again. > > In addition to all this, you also have a max segment size which is > negotiated between the TCP ends, which limit the size of a single IP packet > sent by the TCP protocol. This is done in order to try to avoid packet > fragmentation. > > So the window size is actually a flow control mechanism, and is in reality > limiting the amount of data that can be sent. And it varies all the time. > And the number of packets that will be used for sending that much data is > determined by the MSS (Max Segment Size). > > Sorry for the long text on how TCP works. :-) > > Johnny > > Edwin Fine wrote: > >> David, >> >> Thanks for trying out the benchmark. >> >> With my limited knowledge of TCP/IP, I believe you are seeing the 300,000 >> limit because TCP/IP requires acknowledgements to each packet, and although >> it can batch up multiple acknowledgements in one packet, there is a >> theoretical limit of packets per seconds beyond which it cannot go due to >> the laws of physics. I understand that limit is determined by the Round-Trip >> Time (RTT), which can be shown by ping. On my system, pinging 127.0.0.1 < >> http://127.0.0.1> gives a minimum RTT of 0.018 ms (out of 16 pings). That >> means that the maximum number of packets that can make it to and dest and >> back per second is 1/0.000018 seconds, or 55555 packets per second. The >> TCP/IP stack is evidently packing 5 or 6 blocks into each packet to get the >> 300K blocks/sec you are seeing. Using Wireshark or Ethereal would confirm >> this. I am guessing that this means that the TCP window is about 6 * 1000 >> bytes or 6KB. >> >> What I neglected to tell this group is that I have modified the Linux >> sysctl.conf as follows, which might have had an effect (like I said, I am >> not an expert): >> >> # increase Linux autotuning TCP buffer limits >> # min, default, and max number of bytes to use >> # set max to at least 4MB, or higher if you use very high BDP paths >> net.ipv4.tcp_rmem = 4096 87380 16777216 >> net.ipv4.tcp_wmem = 4096 32768 16777216 >> >> When I have more time, I will vary a number of different Erlang TCP/IP >> parameters and get a data set together that gives a broader picture of the >> effect of the parameters. >> >> Thanks again for taking the time. >> >> 2008/6/24 David Mercer >: >> >> I tried some alternative block sizes (using the blksize option). I >> found that from 1 to somewhere around??maybe a bit short of??1000 >> bytes, the test was able to send about 300,000 blocks in 10 seconds >> regardless of size. (That means, 0.03 MB/sec for block size of 1, >> 0.3 MB/sec for block size of 10, 3 MB/sec for block size of 100, >> etc.) I suspect the system was CPU bound at those levels. >> >> >> Above 1000, the number of blocks sent seemed to decrease, though >> this was more than offset by the increased size of the blocks. Above >> about 10,000 byte blocks (may have been less, I didn't check >> any value between 4,000 and 10,000), however, performance peaked and >> block size no longer mattered: it always sent between 70 and 80 >> MB/sec. My machine is clearly slower than Edwin's? >> >> >> DBM >> >> >> >> ------------------------------------------------------------------------ >> >> *From:* erlang-questions-bounces@REDACTED >> >> [mailto:erlang-questions-bounces@REDACTED >> ] *On Behalf Of *Rapsey >> *Sent:* Tuesday, June 24, 2008 14:01 >> *To:* erlang-questions@REDACTED >> *Subject:* Re: [erlang-questions] why is gen_tcp:send slow? >> >> >> You're using very large packets. I think the results would be much >> more telling if the packets would be a few kB at most. That is >> closer to most real life situations. >> >> >> Sergej >> >> On Tue, Jun 24, 2008 at 8:43 PM, Edwin Fine >> > > wrote: >> >> I wrote a small benchmark in Erlang to see how fast I could get >> socket communications to go. All the benchmark does is pump the same >> buffer to a socket for (by default) 10 seconds. It uses {active, >> once} each time, just like you do. >> >> Server TCP options: >> {active, once}, >> {reuseaddr, true}, >> {packet, 0}, >> {packet_size, 65536}, >> {recbuf, 1000000} >> >> Client TCP options: >> {packet, raw}, >> {packet_size, 65536}, >> {sndbuf, 1024 * 1024}, >> {send_timeout, 3000} >> >> Here are some results using Erlang R12B-3 (erl +K true in the Linux >> version): >> >> Linux (Ubuntu 8.10 x86_64, Intel Core 2 Q6600, 8 GB): >> - Using localhost (127.0.0.1 ): 7474.14 MB in >> >> 10.01 secs (746.66 MB/sec) >> - Using 192.168.x.x IP address: 8064.94 MB in 10.00 secs (806.22 >> MB/sec) [Don't ask me why it's faster than using loopback, I >> repeated the tests and got the same result] >> >> Windows XP SP3 (32 bits), Intel Core 2 Duo E6600: >> - Using loopback: 2166.97 MB in 10.02 secs (216.35 MB/sec) >> - Using 192.168.x.x IP address: 2140.72 MB in 10.02 secs (213.75 >> MB/sec) >> - On Gigabit Ethernet to the Q6600 Linux box: 1063.61 MB in 10.02 >> secs (106.17 MB/sec) using non-jumbo frames. I don't think my router >> supports jumbo frames. >> >> There's undoubtedly a huge discrepancy between the two systems, >> whether because of kernel poll in Linux, or that it's 64 bits, or >> unoptimized Windows TCP/IP flags, I don't know. I don't believe it's >> the number of CPUs (there's only 1 process sending and one >> receiving), or the CPU speed (they are both 2.4 GHz Core 2s). >> >> Maybe some Erlang TCP/IP gurus could comment. >> >> I've attached the code for interest. It's not supposed to be >> production quality, so please don't beat me up :) although I am >> always open to suggestions for improvement. If you do improve it, >> I'd like to see what you've done. Maybe there is another simple >> Erlang tcp benchmark program out there (i.e. not Tsung), but I >> couldn't find one in a cursory Google search. >> >> To run: >> >> VM1: >> >> tb_server:start(Port, Opts). >> tb_server:stop() to stop. >> >> Port = integer() >> Opts = []|[opt()] >> opt() = {atom(), term()} (Accepts inet setopts options, too) >> >> The server prints out the transfer rate (for simplicity). >> >> VM2: >> tb_client(Host, Port, Opts). >> >> Host = atom()|string() hostname or IP address >> Port, Opts as in tb_server >> >> Runs for 10 seconds, sending a 64K buffer as fast as possible to >> Host/Port. >> You can change this to 20 seconds (e.g.) by adding the tupls >> {time_limit, 20000} to Opts. >> You can change buffer size by adding the tuple {blksize, Bytes} to >> Opts. >> >> 2008/6/20 Rapsey >: >> >> All data goes through nginx which acts as a proxy. Its CPU >> consumption is never over 1%. >> >> >> Sergej >> >> >> On Thu, Jun 19, 2008 at 9:35 PM, Javier Par?s Fern?ndez >> > wrote: >> >> >> El 19/06/2008, a las 20:06, Rapsey escribi?: >> >> >> It loops from another module, that way I can update the code at >> any time without disrupting anything. >> The packets are generally a few hundred bytes big, except >> keyframes which tend to be in the kB range. I haven't tried >> looking with wireshark. Still it seems a bit odd that a large >> CPU consumption would be the symptom. The traffic is strictly >> one way. Either someone is sending the stream or receiving it. >> The transmit could of course be written with a passive receive, >> but the code would be significantly uglier. I'm sure someone >> here knows if setting {active, once} every packet is CPU >> intensive or not. >> It seems the workings of gen_tcp is quite platform dependent. If >> I run the code in windows, sending more than 128 bytes per >> gen_tcp call significantly decreases network output. >> Oh and I forgot to mention I use R12B-3. >> >> >> Hi, >> >> Without being an expert. >> >> 200-300 mb/s in small (hundreds of bytes) packets means a *lot* of >> system calls if you are doing a gen_tcp:send for each one. If you >> buffer 3 packets, you are reducing that by a factor of 3 :). I'd try >> to do an small test doing the same thing in C and compare the >> results. I think it will also eat a lot of CPU. >> >> About the proxy CPU... I'm a bit lost about it, but speculating >> wildly it is possible that the time spent doing the system calls >> that gen_tcp is doing is added to the proxy CPU process. >> >> Regards. >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bqt@REDACTED Wed Jun 25 00:00:51 2008 From: bqt@REDACTED (Johnny Billquist) Date: Wed, 25 Jun 2008 00:00:51 +0200 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <6c2563b20806241416r71fdb225v90930bead9c95d65@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> <399E231E-E06E-4FCE-901A-24C6B7B2DFA8@udc.es> <97619b170806192135r5504e3a4s80179885510971d9@mail.gmail.com> <6c2563b20806241143x1fd06acdq34b14b04058428c1@mail.gmail.com> <97619b170806241200t6a1e8c35vd53494ad9f3806ff@mail.gmail.com> <019501c8d635$31455fe0$f21ea8c0@SSI.CORP> <6c2563b20806241416r71fdb225v90930bead9c95d65@mail.gmail.com> Message-ID: <48616E93.2010300@softjar.se> No. TCP don't acknowledge every packet. In fact, TCP don't acknowledge packets as such at all. TCP is not packet based. It's just that if you use IP as the carrier, IP itself it packet based. TCP can in theory generate any number of packets per second. However, the amount of unacknowledged data that can be outstanding at any time is limited by the transmit window. Each packet carries a window size, which is how much more data that can be accepted by the reciever. TCP can (is allowed to) send that much data and no more. The RTT calculations are used for figuring out how long to wait before doing retransmissions. You also normally have a slow start transmission algorithm which prevents the sender from even using the full window size from the start, as a way of avoiding congestions. That is used in combination with a backoff algorithm when retransmissions are needed to further decrease congestions, but all of this only really comes into effect if you start loosing data, and TCP actually needs to do retransmissions. Another thing you have is an algorithm called Nagle, which tries to collect small amount of data sent into larger packets before sending it, so that you don't flood the net with silly small packets. One addisional detail is that receivers normally, when the receive buffers becomes full, don't announce newly freed space immediately, since that is normally rather small amounts, but instead wait a while, until a larger part of the receive buffer is free, so that the sender actually can send some full sized packets once it starts sending again. In addition to all this, you also have a max segment size which is negotiated between the TCP ends, which limit the size of a single IP packet sent by the TCP protocol. This is done in order to try to avoid packet fragmentation. So the window size is actually a flow control mechanism, and is in reality limiting the amount of data that can be sent. And it varies all the time. And the number of packets that will be used for sending that much data is determined by the MSS (Max Segment Size). Sorry for the long text on how TCP works. :-) Johnny Edwin Fine wrote: > David, > > Thanks for trying out the benchmark. > > With my limited knowledge of TCP/IP, I believe you are seeing the > 300,000 limit because TCP/IP requires acknowledgements to each packet, > and although it can batch up multiple acknowledgements in one packet, > there is a theoretical limit of packets per seconds beyond which it > cannot go due to the laws of physics. I understand that limit is > determined by the Round-Trip Time (RTT), which can be shown by ping. On > my system, pinging 127.0.0.1 gives a minimum RTT of > 0.018 ms (out of 16 pings). That means that the maximum number of > packets that can make it to and dest and back per second is 1/0.000018 > seconds, or 55555 packets per second. The TCP/IP stack is evidently > packing 5 or 6 blocks into each packet to get the 300K blocks/sec you > are seeing. Using Wireshark or Ethereal would confirm this. I am > guessing that this means that the TCP window is about 6 * 1000 bytes or 6KB. > > What I neglected to tell this group is that I have modified the Linux > sysctl.conf as follows, which might have had an effect (like I said, I > am not an expert): > > # increase Linux autotuning TCP buffer limits > # min, default, and max number of bytes to use > # set max to at least 4MB, or higher if you use very high BDP paths > net.ipv4.tcp_rmem = 4096 87380 16777216 > net.ipv4.tcp_wmem = 4096 32768 16777216 > > When I have more time, I will vary a number of different Erlang TCP/IP > parameters and get a data set together that gives a broader picture of > the effect of the parameters. > > Thanks again for taking the time. > > 2008/6/24 David Mercer >: > > I tried some alternative block sizes (using the blksize option). I > found that from 1 to somewhere around??maybe a bit short of??1000 > bytes, the test was able to send about 300,000 blocks in 10 seconds > regardless of size. (That means, 0.03 MB/sec for block size of 1, > 0.3 MB/sec for block size of 10, 3 MB/sec for block size of 100, > etc.) I suspect the system was CPU bound at those levels. > > > > Above 1000, the number of blocks sent seemed to decrease, though > this was more than offset by the increased size of the blocks. > Above about 10,000 byte blocks (may have been less, I didn't check > any value between 4,000 and 10,000), however, performance peaked and > block size no longer mattered: it always sent between 70 and 80 > MB/sec. My machine is clearly slower than Edwin's? > > > > DBM > > > > ------------------------------------------------------------------------ > > *From:* erlang-questions-bounces@REDACTED > > [mailto:erlang-questions-bounces@REDACTED > ] *On Behalf Of *Rapsey > *Sent:* Tuesday, June 24, 2008 14:01 > *To:* erlang-questions@REDACTED > *Subject:* Re: [erlang-questions] why is gen_tcp:send slow? > > > > You're using very large packets. I think the results would be much > more telling if the packets would be a few kB at most. That is > closer to most real life situations. > > > Sergej > > On Tue, Jun 24, 2008 at 8:43 PM, Edwin Fine > > wrote: > > I wrote a small benchmark in Erlang to see how fast I could get > socket communications to go. All the benchmark does is pump the same > buffer to a socket for (by default) 10 seconds. It uses {active, > once} each time, just like you do. > > Server TCP options: > {active, once}, > {reuseaddr, true}, > {packet, 0}, > {packet_size, 65536}, > {recbuf, 1000000} > > Client TCP options: > {packet, raw}, > {packet_size, 65536}, > {sndbuf, 1024 * 1024}, > {send_timeout, 3000} > > Here are some results using Erlang R12B-3 (erl +K true in the Linux > version): > > Linux (Ubuntu 8.10 x86_64, Intel Core 2 Q6600, 8 GB): > - Using localhost (127.0.0.1 ): 7474.14 MB in > 10.01 secs (746.66 MB/sec) > - Using 192.168.x.x IP address: 8064.94 MB in 10.00 secs (806.22 > MB/sec) [Don't ask me why it's faster than using loopback, I > repeated the tests and got the same result] > > Windows XP SP3 (32 bits), Intel Core 2 Duo E6600: > - Using loopback: 2166.97 MB in 10.02 secs (216.35 MB/sec) > - Using 192.168.x.x IP address: 2140.72 MB in 10.02 secs (213.75 MB/sec) > - On Gigabit Ethernet to the Q6600 Linux box: 1063.61 MB in 10.02 > secs (106.17 MB/sec) using non-jumbo frames. I don't think my router > supports jumbo frames. > > There's undoubtedly a huge discrepancy between the two systems, > whether because of kernel poll in Linux, or that it's 64 bits, or > unoptimized Windows TCP/IP flags, I don't know. I don't believe it's > the number of CPUs (there's only 1 process sending and one > receiving), or the CPU speed (they are both 2.4 GHz Core 2s). > > Maybe some Erlang TCP/IP gurus could comment. > > I've attached the code for interest. It's not supposed to be > production quality, so please don't beat me up :) although I am > always open to suggestions for improvement. If you do improve it, > I'd like to see what you've done. Maybe there is another simple > Erlang tcp benchmark program out there (i.e. not Tsung), but I > couldn't find one in a cursory Google search. > > To run: > > VM1: > > tb_server:start(Port, Opts). > tb_server:stop() to stop. > > Port = integer() > Opts = []|[opt()] > opt() = {atom(), term()} (Accepts inet setopts options, too) > > The server prints out the transfer rate (for simplicity). > > VM2: > tb_client(Host, Port, Opts). > > Host = atom()|string() hostname or IP address > Port, Opts as in tb_server > > Runs for 10 seconds, sending a 64K buffer as fast as possible to > Host/Port. > You can change this to 20 seconds (e.g.) by adding the tupls > {time_limit, 20000} to Opts. > You can change buffer size by adding the tuple {blksize, Bytes} to Opts. > > 2008/6/20 Rapsey >: > > All data goes through nginx which acts as a proxy. Its CPU > consumption is never over 1%. > > > Sergej > > > > On Thu, Jun 19, 2008 at 9:35 PM, Javier Par?s Fern?ndez > > wrote: > > > El 19/06/2008, a las 20:06, Rapsey escribi?: > > > > It loops from another module, that way I can update the code at > any time without disrupting anything. > The packets are generally a few hundred bytes big, except > keyframes which tend to be in the kB range. I haven't tried > looking with wireshark. Still it seems a bit odd that a large > CPU consumption would be the symptom. The traffic is strictly > one way. Either someone is sending the stream or receiving it. > The transmit could of course be written with a passive receive, > but the code would be significantly uglier. I'm sure someone > here knows if setting {active, once} every packet is CPU > intensive or not. > It seems the workings of gen_tcp is quite platform dependent. If > I run the code in windows, sending more than 128 bytes per > gen_tcp call significantly decreases network output. > Oh and I forgot to mention I use R12B-3. > > > > Hi, > > Without being an expert. > > 200-300 mb/s in small (hundreds of bytes) packets means a *lot* of > system calls if you are doing a gen_tcp:send for each one. If you > buffer 3 packets, you are reducing that by a factor of 3 :). I'd try > to do an small test doing the same thing in C and compare the > results. I think it will also eat a lot of CPU. > > About the proxy CPU... I'm a bit lost about it, but speculating > wildly it is possible that the time spent doing the system calls > that gen_tcp is doing is added to the proxy CPU process. > > Regards. > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > > > ------------------------------------------------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From bqt@REDACTED Wed Jun 25 01:15:15 2008 From: bqt@REDACTED (Johnny Billquist) Date: Wed, 25 Jun 2008 01:15:15 +0200 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <6c2563b20806241555k269a0848n302fc7335deb0c23@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <6c2563b20806191039w4f82d6d9l8740a5bcc6b8e598@mail.gmail.com> <97619b170806191106j133e76du71ce879469e9830c@mail.gmail.com> <399E231E-E06E-4FCE-901A-24C6B7B2DFA8@udc.es> <97619b170806192135r5504e3a4s80179885510971d9@mail.gmail.com> <6c2563b20806241143x1fd06acdq34b14b04058428c1@mail.gmail.com> <97619b170806241200t6a1e8c35vd53494ad9f3806ff@mail.gmail.com> <019501c8d635$31455fe0$f21ea8c0@SSI.CORP> <6c2563b20806241416r71fdb225v90930bead9c95d65@mail.gmail.com> <48616E93.2010300@softjar.se> <6c2563b20806241555k269a0848n302fc7335deb0c23@mail.gmail.com> Message-ID: <48618003.3030506@softjar.se> Edwin, always happy to help out... Edwin Fine skrev: > Johnny, > > Thanks for the lesson! I am always happy to learn. Like I said, I am not an > expert in TCP/IP. > > What I was writing about when I said that packets are acknowledged is what I > saw in Wireshark while trying to understand performance issues. I perhaps > should have said "TCP/IP" instead of just "TCP". There were definitely > acknowledgements, but I guess they were at the IP level. No. IP don't have any acknowledgements. IP (as well as UDP) is basically just sending packets without any guarantee that they will ever reach the other end. What you saw was TCP acknowledgements, but you misunderstood how they work. Think of a TCP connection as an eternal length stream of bytes. Each byte in this stream have a sequence number. TCP sends bytes in this stream, packed into IP packets. Each IP packet will have one or several bytes from that stream. TCP at the other end will acknowledge the highest ordered byte that is has received. How many packets it took to get to that byte is irrelevant, as is any retransmissions, and so on... The window size tells how many additional bytes from this stream can be sent, which is further on based in the point which the acknowledgement points at. (In reality, the sequence numbers are not infitite, but are actually a 32-bit number, which wraps. But since window sizes normally fits in a 16-bit quantity, there is no chance of ever getting back to the same sequence number again before it has long been passed by the time before, so no risk of confusion or errors there.) > I wonder what the MSS is for loopback? I think it's about 1536 on my eth0 > interface, but not sure. Smart implementations use the MTU - 40 of the interface as the MSS for a loopback connection. Otherwise the thumb of rule is that if it's on the same network, MSS is usually set to 1460 and 536 for destinations on other networks. This comes from the fact that the local network (usually ethernet) have an MTU of 1500, and the IP header is normally 20 bytes, and so is the standard TCP header, leaving 1460 bytes of data in an ethernet frame. For non-local destinations, IP requires that atleast 576 byte packets can go through unfragmented. The rest follows. :-) > As for RTT, I sent data over a link that had a very long (290ms) RTT, and > that definitely limited the rate at which packets could be sent. Can RTT be > used to calculate the theoretical maximum traffic that a link can carry? > For example, a satellite link with a 400ms RTT but 2 Mbps bandwidth? No. RTT can not be used to calculate anything regarding traffic bandwidth. You can keep sending packets until the window is exhausted, no matter what the RTT says. The RTT is only used to calculate when to do retransmissions if you haven't received an ACK. The only other thing that affects packet rates are the slow start algorithm. That will be affected by the round trip delays, since it adds a throttling effect on the window, in addition to what the received says. The reason for it being affected by the rount trip delay is that the slow start window size is only increased when you get ACK packets back. But, assuming the link can take the load, and you don't loose a lot of packets, the slow start algorithm will pretty quickly stop being a factor. Johnny > > Ed > > On Tue, Jun 24, 2008 at 6:00 PM, Johnny Billquist wrote: > >> No. TCP don't acknowledge every packet. In fact, TCP don't acknowledge >> packets as such at all. TCP is not packet based. It's just that if you use >> IP as the carrier, IP itself it packet based. >> TCP can in theory generate any number of packets per second. However, the >> amount of unacknowledged data that can be outstanding at any time is limited >> by the transmit window. Each packet carries a window size, which is how much >> more data that can be accepted by the reciever. TCP can (is allowed to) send >> that much data and no more. >> >> The RTT calculations are used for figuring out how long to wait before >> doing retransmissions. You also normally have a slow start transmission >> algorithm which prevents the sender from even using the full window size >> from the start, as a way of avoiding congestions. That is used in >> combination with a backoff algorithm when retransmissions are needed to >> further decrease congestions, but all of this only really comes into effect >> if you start loosing data, and TCP actually needs to do retransmissions. >> >> Another thing you have is an algorithm called Nagle, which tries to collect >> small amount of data sent into larger packets before sending it, so that you >> don't flood the net with silly small packets. >> >> One addisional detail is that receivers normally, when the receive buffers >> becomes full, don't announce newly freed space immediately, since that is >> normally rather small amounts, but instead wait a while, until a larger part >> of the receive buffer is free, so that the sender actually can send some >> full sized packets once it starts sending again. >> >> In addition to all this, you also have a max segment size which is >> negotiated between the TCP ends, which limit the size of a single IP packet >> sent by the TCP protocol. This is done in order to try to avoid packet >> fragmentation. >> >> So the window size is actually a flow control mechanism, and is in reality >> limiting the amount of data that can be sent. And it varies all the time. >> And the number of packets that will be used for sending that much data is >> determined by the MSS (Max Segment Size). >> >> Sorry for the long text on how TCP works. :-) >> >> Johnny >> >> Edwin Fine wrote: >> >>> David, >>> >>> Thanks for trying out the benchmark. >>> >>> With my limited knowledge of TCP/IP, I believe you are seeing the 300,000 >>> limit because TCP/IP requires acknowledgements to each packet, and although >>> it can batch up multiple acknowledgements in one packet, there is a >>> theoretical limit of packets per seconds beyond which it cannot go due to >>> the laws of physics. I understand that limit is determined by the Round-Trip >>> Time (RTT), which can be shown by ping. On my system, pinging 127.0.0.1 < >>> http://127.0.0.1> gives a minimum RTT of 0.018 ms (out of 16 pings). That >>> means that the maximum number of packets that can make it to and dest and >>> back per second is 1/0.000018 seconds, or 55555 packets per second. The >>> TCP/IP stack is evidently packing 5 or 6 blocks into each packet to get the >>> 300K blocks/sec you are seeing. Using Wireshark or Ethereal would confirm >>> this. I am guessing that this means that the TCP window is about 6 * 1000 >>> bytes or 6KB. >>> >>> What I neglected to tell this group is that I have modified the Linux >>> sysctl.conf as follows, which might have had an effect (like I said, I am >>> not an expert): >>> >>> # increase Linux autotuning TCP buffer limits >>> # min, default, and max number of bytes to use >>> # set max to at least 4MB, or higher if you use very high BDP paths >>> net.ipv4.tcp_rmem = 4096 87380 16777216 >>> net.ipv4.tcp_wmem = 4096 32768 16777216 >>> >>> When I have more time, I will vary a number of different Erlang TCP/IP >>> parameters and get a data set together that gives a broader picture of the >>> effect of the parameters. >>> >>> Thanks again for taking the time. >>> >>> 2008/6/24 David Mercer >: >>> >>> I tried some alternative block sizes (using the blksize option). I >>> found that from 1 to somewhere around??maybe a bit short of??1000 >>> bytes, the test was able to send about 300,000 blocks in 10 seconds >>> regardless of size. (That means, 0.03 MB/sec for block size of 1, >>> 0.3 MB/sec for block size of 10, 3 MB/sec for block size of 100, >>> etc.) I suspect the system was CPU bound at those levels. >>> >>> >>> Above 1000, the number of blocks sent seemed to decrease, though >>> this was more than offset by the increased size of the blocks. Above >>> about 10,000 byte blocks (may have been less, I didn't check >>> any value between 4,000 and 10,000), however, performance peaked and >>> block size no longer mattered: it always sent between 70 and 80 >>> MB/sec. My machine is clearly slower than Edwin's? >>> >>> >>> DBM >>> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> *From:* erlang-questions-bounces@REDACTED >>> >>> [mailto:erlang-questions-bounces@REDACTED >>> ] *On Behalf Of *Rapsey >>> *Sent:* Tuesday, June 24, 2008 14:01 >>> *To:* erlang-questions@REDACTED >>> *Subject:* Re: [erlang-questions] why is gen_tcp:send slow? >>> >>> >>> You're using very large packets. I think the results would be much >>> more telling if the packets would be a few kB at most. That is >>> closer to most real life situations. >>> >>> >>> Sergej >>> >>> On Tue, Jun 24, 2008 at 8:43 PM, Edwin Fine >>> >> > wrote: >>> >>> I wrote a small benchmark in Erlang to see how fast I could get >>> socket communications to go. All the benchmark does is pump the same >>> buffer to a socket for (by default) 10 seconds. It uses {active, >>> once} each time, just like you do. >>> >>> Server TCP options: >>> {active, once}, >>> {reuseaddr, true}, >>> {packet, 0}, >>> {packet_size, 65536}, >>> {recbuf, 1000000} >>> >>> Client TCP options: >>> {packet, raw}, >>> {packet_size, 65536}, >>> {sndbuf, 1024 * 1024}, >>> {send_timeout, 3000} >>> >>> Here are some results using Erlang R12B-3 (erl +K true in the Linux >>> version): >>> >>> Linux (Ubuntu 8.10 x86_64, Intel Core 2 Q6600, 8 GB): >>> - Using localhost (127.0.0.1 ): 7474.14 MB in >>> >>> 10.01 secs (746.66 MB/sec) >>> - Using 192.168.x.x IP address: 8064.94 MB in 10.00 secs (806.22 >>> MB/sec) [Don't ask me why it's faster than using loopback, I >>> repeated the tests and got the same result] >>> >>> Windows XP SP3 (32 bits), Intel Core 2 Duo E6600: >>> - Using loopback: 2166.97 MB in 10.02 secs (216.35 MB/sec) >>> - Using 192.168.x.x IP address: 2140.72 MB in 10.02 secs (213.75 >>> MB/sec) >>> - On Gigabit Ethernet to the Q6600 Linux box: 1063.61 MB in 10.02 >>> secs (106.17 MB/sec) using non-jumbo frames. I don't think my router >>> supports jumbo frames. >>> >>> There's undoubtedly a huge discrepancy between the two systems, >>> whether because of kernel poll in Linux, or that it's 64 bits, or >>> unoptimized Windows TCP/IP flags, I don't know. I don't believe it's >>> the number of CPUs (there's only 1 process sending and one >>> receiving), or the CPU speed (they are both 2.4 GHz Core 2s). >>> >>> Maybe some Erlang TCP/IP gurus could comment. >>> >>> I've attached the code for interest. It's not supposed to be >>> production quality, so please don't beat me up :) although I am >>> always open to suggestions for improvement. If you do improve it, >>> I'd like to see what you've done. Maybe there is another simple >>> Erlang tcp benchmark program out there (i.e. not Tsung), but I >>> couldn't find one in a cursory Google search. >>> >>> To run: >>> >>> VM1: >>> >>> tb_server:start(Port, Opts). >>> tb_server:stop() to stop. >>> >>> Port = integer() >>> Opts = []|[opt()] >>> opt() = {atom(), term()} (Accepts inet setopts options, too) >>> >>> The server prints out the transfer rate (for simplicity). >>> >>> VM2: >>> tb_client(Host, Port, Opts). >>> >>> Host = atom()|string() hostname or IP address >>> Port, Opts as in tb_server >>> >>> Runs for 10 seconds, sending a 64K buffer as fast as possible to >>> Host/Port. >>> You can change this to 20 seconds (e.g.) by adding the tupls >>> {time_limit, 20000} to Opts. >>> You can change buffer size by adding the tuple {blksize, Bytes} to >>> Opts. >>> >>> 2008/6/20 Rapsey >: >>> >>> All data goes through nginx which acts as a proxy. Its CPU >>> consumption is never over 1%. >>> >>> >>> Sergej >>> >>> >>> On Thu, Jun 19, 2008 at 9:35 PM, Javier Par?s Fern?ndez >>> > wrote: >>> >>> >>> El 19/06/2008, a las 20:06, Rapsey escribi?: >>> >>> >>> It loops from another module, that way I can update the code at >>> any time without disrupting anything. >>> The packets are generally a few hundred bytes big, except >>> keyframes which tend to be in the kB range. I haven't tried >>> looking with wireshark. Still it seems a bit odd that a large >>> CPU consumption would be the symptom. The traffic is strictly >>> one way. Either someone is sending the stream or receiving it. >>> The transmit could of course be written with a passive receive, >>> but the code would be significantly uglier. I'm sure someone >>> here knows if setting {active, once} every packet is CPU >>> intensive or not. >>> It seems the workings of gen_tcp is quite platform dependent. If >>> I run the code in windows, sending more than 128 bytes per >>> gen_tcp call significantly decreases network output. >>> Oh and I forgot to mention I use R12B-3. >>> >>> >>> Hi, >>> >>> Without being an expert. >>> >>> 200-300 mb/s in small (hundreds of bytes) packets means a *lot* of >>> system calls if you are doing a gen_tcp:send for each one. If you >>> buffer 3 packets, you are reducing that by a factor of 3 :). I'd try >>> to do an small test doing the same thing in C and compare the >>> results. I think it will also eat a lot of CPU. >>> >>> About the proxy CPU... I'm a bit lost about it, but speculating >>> wildly it is possible that the time spent doing the system calls >>> that gen_tcp is doing is added to the proxy CPU process. >>> >>> Regards. >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >> >> > -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: bqt@REDACTED || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol From erlang-questions_efine@REDACTED Wed Jun 25 02:38:15 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Tue, 24 Jun 2008 20:38:15 -0400 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <48618003.3030506@softjar.se> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <399E231E-E06E-4FCE-901A-24C6B7B2DFA8@udc.es> <97619b170806192135r5504e3a4s80179885510971d9@mail.gmail.com> <6c2563b20806241143x1fd06acdq34b14b04058428c1@mail.gmail.com> <97619b170806241200t6a1e8c35vd53494ad9f3806ff@mail.gmail.com> <019501c8d635$31455fe0$f21ea8c0@SSI.CORP> <6c2563b20806241416r71fdb225v90930bead9c95d65@mail.gmail.com> <48616E93.2010300@softjar.se> <6c2563b20806241555k269a0848n302fc7335deb0c23@mail.gmail.com> <48618003.3030506@softjar.se> Message-ID: <6c2563b20806241738o64987690i10c89933c6c9070d@mail.gmail.com> Ok Johnny, I think I get it now. Thanks for the detailed explanation. I wonder why the original poster (Sergej/Rapsey) is seeing such poor TCP/IP performance? In any case, I am still going to do some more benchmarks to see if I can understand how the different components of TCP/IP communication in Erlang (inet:setopts() and gen_tcp) affect performance, CPU overhead and so on. The reason I got into all this is because I was seeing very good performance between two systems on a LAN, and terrible performance over a non-local overseas link that had an RTT of about 290ms. Through various measurements and Wireshark usage I found the link was carrying only 3.4 packets per second, with only about 56 data bytes in each packet. When I investigated further, I found that a function I thought was running asynchronously was actually running synchronously inside a gen_server:call(). When I spawned the function, I still only saw 3.4 packets per second (using Wireshark timestamps) but each packet was now full of multiple blocks of data, not just 56 bytes, so the actual throughput went up hugely. Nothing else changed. When I tried to find out where the 3.4 was coming from, I calculated 1/3.4 = 0.294ms which was (coincidentally?) the exact RTT. That's why I thought there was a relationship between RTT and the number of packets/second a link could carry. Now I have to go back and try to figure it all all over again :( unless you can explain it to me (he said hopefully). Thanks Ed On Tue, Jun 24, 2008 at 7:15 PM, Johnny Billquist wrote: > Edwin, always happy to help out... > > Edwin Fine skrev: > >> Johnny, >> >> Thanks for the lesson! I am always happy to learn. Like I said, I am not >> an >> expert in TCP/IP. >> >> What I was writing about when I said that packets are acknowledged is what >> I >> saw in Wireshark while trying to understand performance issues. I perhaps >> should have said "TCP/IP" instead of just "TCP". There were definitely >> acknowledgements, but I guess they were at the IP level. >> > > No. IP don't have any acknowledgements. IP (as well as UDP) is basically > just sending packets without any guarantee that they will ever reach the > other end. > What you saw was TCP acknowledgements, but you misunderstood how they work. > > Think of a TCP connection as an eternal length stream of bytes. Each byte > in this stream have a sequence number. TCP sends bytes in this stream, > packed into IP packets. Each IP packet will have one or several bytes from > that stream. > TCP at the other end will acknowledge the highest ordered byte that is has > received. How many packets it took to get to that byte is irrelevant, as is > any retransmissions, and so on... The window size tells how many additional > bytes from this stream can be sent, which is further on based in the point > which the acknowledgement points at. > > (In reality, the sequence numbers are not infitite, but are actually a > 32-bit number, which wraps. But since window sizes normally fits in a 16-bit > quantity, there is no chance of ever getting back to the same sequence > number again before it has long been passed by the time before, so no risk > of confusion or errors there.) > > I wonder what the MSS is for loopback? I think it's about 1536 on my eth0 >> interface, but not sure. >> > > Smart implementations use the MTU - 40 of the interface as the MSS for a > loopback connection. Otherwise the thumb of rule is that if it's on the same > network, MSS is usually set to 1460 and 536 for destinations on other > networks. > This comes from the fact that the local network (usually ethernet) have an > MTU of 1500, and the IP header is normally 20 bytes, and so is the standard > TCP header, leaving 1460 bytes of data in an ethernet frame. > For non-local destinations, IP requires that atleast 576 byte packets can > go through unfragmented. The rest follows. :-) > > As for RTT, I sent data over a link that had a very long (290ms) RTT, and >> that definitely limited the rate at which packets could be sent. Can RTT >> be >> used to calculate the theoretical maximum traffic that a link can carry? >> For example, a satellite link with a 400ms RTT but 2 Mbps bandwidth? >> > > No. RTT can not be used to calculate anything regarding traffic bandwidth. > You can keep sending packets until the window is exhausted, no matter what > the RTT says. The RTT is only used to calculate when to do retransmissions > if you haven't received an ACK. > The only other thing that affects packet rates are the slow start > algorithm. That will be affected by the round trip delays, since it adds a > throttling effect on the window, in addition to what the received says. The > reason for it being affected by the rount trip delay is that the slow start > window size is only increased when you get ACK packets back. > But, assuming the link can take the load, and you don't loose a lot of > packets, the slow start algorithm will pretty quickly stop being a factor. > > Johnny > > > >> Ed >> >> On Tue, Jun 24, 2008 at 6:00 PM, Johnny Billquist wrote: >> >> No. TCP don't acknowledge every packet. In fact, TCP don't acknowledge >>> packets as such at all. TCP is not packet based. It's just that if you >>> use >>> IP as the carrier, IP itself it packet based. >>> TCP can in theory generate any number of packets per second. However, the >>> amount of unacknowledged data that can be outstanding at any time is >>> limited >>> by the transmit window. Each packet carries a window size, which is how >>> much >>> more data that can be accepted by the reciever. TCP can (is allowed to) >>> send >>> that much data and no more. >>> >>> The RTT calculations are used for figuring out how long to wait before >>> doing retransmissions. You also normally have a slow start transmission >>> algorithm which prevents the sender from even using the full window size >>> from the start, as a way of avoiding congestions. That is used in >>> combination with a backoff algorithm when retransmissions are needed to >>> further decrease congestions, but all of this only really comes into >>> effect >>> if you start loosing data, and TCP actually needs to do retransmissions. >>> >>> Another thing you have is an algorithm called Nagle, which tries to >>> collect >>> small amount of data sent into larger packets before sending it, so that >>> you >>> don't flood the net with silly small packets. >>> >>> One addisional detail is that receivers normally, when the receive >>> buffers >>> becomes full, don't announce newly freed space immediately, since that is >>> normally rather small amounts, but instead wait a while, until a larger >>> part >>> of the receive buffer is free, so that the sender actually can send some >>> full sized packets once it starts sending again. >>> >>> In addition to all this, you also have a max segment size which is >>> negotiated between the TCP ends, which limit the size of a single IP >>> packet >>> sent by the TCP protocol. This is done in order to try to avoid packet >>> fragmentation. >>> >>> So the window size is actually a flow control mechanism, and is in >>> reality >>> limiting the amount of data that can be sent. And it varies all the time. >>> And the number of packets that will be used for sending that much data is >>> determined by the MSS (Max Segment Size). >>> >>> Sorry for the long text on how TCP works. :-) >>> >>> Johnny >>> >>> Edwin Fine wrote: >>> >>> David, >>>> >>>> Thanks for trying out the benchmark. >>>> >>>> With my limited knowledge of TCP/IP, I believe you are seeing the >>>> 300,000 >>>> limit because TCP/IP requires acknowledgements to each packet, and >>>> although >>>> it can batch up multiple acknowledgements in one packet, there is a >>>> theoretical limit of packets per seconds beyond which it cannot go due >>>> to >>>> the laws of physics. I understand that limit is determined by the >>>> Round-Trip >>>> Time (RTT), which can be shown by ping. On my system, pinging 127.0.0.1< >>>> http://127.0.0.1> gives a minimum RTT of 0.018 ms (out of 16 pings). >>>> That >>>> means that the maximum number of packets that can make it to and dest >>>> and >>>> back per second is 1/0.000018 seconds, or 55555 packets per second. The >>>> TCP/IP stack is evidently packing 5 or 6 blocks into each packet to get >>>> the >>>> 300K blocks/sec you are seeing. Using Wireshark or Ethereal would >>>> confirm >>>> this. I am guessing that this means that the TCP window is about 6 * >>>> 1000 >>>> bytes or 6KB. >>>> >>>> What I neglected to tell this group is that I have modified the Linux >>>> sysctl.conf as follows, which might have had an effect (like I said, I >>>> am >>>> not an expert): >>>> >>>> # increase Linux autotuning TCP buffer limits >>>> # min, default, and max number of bytes to use >>>> # set max to at least 4MB, or higher if you use very high BDP paths >>>> net.ipv4.tcp_rmem = 4096 87380 16777216 >>>> net.ipv4.tcp_wmem = 4096 32768 16777216 >>>> >>>> When I have more time, I will vary a number of different Erlang TCP/IP >>>> parameters and get a data set together that gives a broader picture of >>>> the >>>> effect of the parameters. >>>> >>>> Thanks again for taking the time. >>>> >>>> 2008/6/24 David Mercer >: >>>> >>>> I tried some alternative block sizes (using the blksize option). I >>>> found that from 1 to somewhere around??maybe a bit short of??1000 >>>> bytes, the test was able to send about 300,000 blocks in 10 seconds >>>> regardless of size. (That means, 0.03 MB/sec for block size of 1, >>>> 0.3 MB/sec for block size of 10, 3 MB/sec for block size of 100, >>>> etc.) I suspect the system was CPU bound at those levels. >>>> >>>> >>>> Above 1000, the number of blocks sent seemed to decrease, though >>>> this was more than offset by the increased size of the blocks. >>>> Above >>>> about 10,000 byte blocks (may have been less, I didn't check >>>> any value between 4,000 and 10,000), however, performance peaked and >>>> block size no longer mattered: it always sent between 70 and 80 >>>> MB/sec. My machine is clearly slower than Edwin's? >>>> >>>> >>>> DBM >>>> >>>> >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> *From:* erlang-questions-bounces@REDACTED >>>> >>>> [mailto:erlang-questions-bounces@REDACTED >>>> ] *On Behalf Of *Rapsey >>>> *Sent:* Tuesday, June 24, 2008 14:01 >>>> *To:* erlang-questions@REDACTED >>> > >>>> *Subject:* Re: [erlang-questions] why is gen_tcp:send slow? >>>> >>>> >>>> You're using very large packets. I think the results would be much >>>> more telling if the packets would be a few kB at most. That is >>>> closer to most real life situations. >>>> >>>> >>>> Sergej >>>> >>>> On Tue, Jun 24, 2008 at 8:43 PM, Edwin Fine >>>> >>> > wrote: >>>> >>>> I wrote a small benchmark in Erlang to see how fast I could get >>>> socket communications to go. All the benchmark does is pump the same >>>> buffer to a socket for (by default) 10 seconds. It uses {active, >>>> once} each time, just like you do. >>>> >>>> Server TCP options: >>>> {active, once}, >>>> {reuseaddr, true}, >>>> {packet, 0}, >>>> {packet_size, 65536}, >>>> {recbuf, 1000000} >>>> >>>> Client TCP options: >>>> {packet, raw}, >>>> {packet_size, 65536}, >>>> {sndbuf, 1024 * 1024}, >>>> {send_timeout, 3000} >>>> >>>> Here are some results using Erlang R12B-3 (erl +K true in the Linux >>>> version): >>>> >>>> Linux (Ubuntu 8.10 x86_64, Intel Core 2 Q6600, 8 GB): >>>> - Using localhost (127.0.0.1 ): 7474.14 MB in >>>> >>>> 10.01 secs (746.66 MB/sec) >>>> - Using 192.168.x.x IP address: 8064.94 MB in 10.00 secs (806.22 >>>> MB/sec) [Don't ask me why it's faster than using loopback, I >>>> repeated the tests and got the same result] >>>> >>>> Windows XP SP3 (32 bits), Intel Core 2 Duo E6600: >>>> - Using loopback: 2166.97 MB in 10.02 secs (216.35 MB/sec) >>>> - Using 192.168.x.x IP address: 2140.72 MB in 10.02 secs (213.75 >>>> MB/sec) >>>> - On Gigabit Ethernet to the Q6600 Linux box: 1063.61 MB in 10.02 >>>> secs (106.17 MB/sec) using non-jumbo frames. I don't think my router >>>> supports jumbo frames. >>>> >>>> There's undoubtedly a huge discrepancy between the two systems, >>>> whether because of kernel poll in Linux, or that it's 64 bits, or >>>> unoptimized Windows TCP/IP flags, I don't know. I don't believe it's >>>> the number of CPUs (there's only 1 process sending and one >>>> receiving), or the CPU speed (they are both 2.4 GHz Core 2s). >>>> >>>> Maybe some Erlang TCP/IP gurus could comment. >>>> >>>> I've attached the code for interest. It's not supposed to be >>>> production quality, so please don't beat me up :) although I am >>>> always open to suggestions for improvement. If you do improve it, >>>> I'd like to see what you've done. Maybe there is another simple >>>> Erlang tcp benchmark program out there (i.e. not Tsung), but I >>>> couldn't find one in a cursory Google search. >>>> >>>> To run: >>>> >>>> VM1: >>>> >>>> tb_server:start(Port, Opts). >>>> tb_server:stop() to stop. >>>> >>>> Port = integer() >>>> Opts = []|[opt()] >>>> opt() = {atom(), term()} (Accepts inet setopts options, too) >>>> >>>> The server prints out the transfer rate (for simplicity). >>>> >>>> VM2: >>>> tb_client(Host, Port, Opts). >>>> >>>> Host = atom()|string() hostname or IP address >>>> Port, Opts as in tb_server >>>> >>>> Runs for 10 seconds, sending a 64K buffer as fast as possible to >>>> Host/Port. >>>> You can change this to 20 seconds (e.g.) by adding the tupls >>>> {time_limit, 20000} to Opts. >>>> You can change buffer size by adding the tuple {blksize, Bytes} to >>>> Opts. >>>> >>>> 2008/6/20 Rapsey >: >>>> >>>> All data goes through nginx which acts as a proxy. Its CPU >>>> consumption is never over 1%. >>>> >>>> >>>> Sergej >>>> >>>> >>>> On Thu, Jun 19, 2008 at 9:35 PM, Javier Par?s Fern?ndez >>>> > wrote: >>>> >>>> >>>> El 19/06/2008, a las 20:06, Rapsey escribi?: >>>> >>>> >>>> It loops from another module, that way I can update the code at >>>> any time without disrupting anything. >>>> The packets are generally a few hundred bytes big, except >>>> keyframes which tend to be in the kB range. I haven't tried >>>> looking with wireshark. Still it seems a bit odd that a large >>>> CPU consumption would be the symptom. The traffic is strictly >>>> one way. Either someone is sending the stream or receiving it. >>>> The transmit could of course be written with a passive receive, >>>> but the code would be significantly uglier. I'm sure someone >>>> here knows if setting {active, once} every packet is CPU >>>> intensive or not. >>>> It seems the workings of gen_tcp is quite platform dependent. If >>>> I run the code in windows, sending more than 128 bytes per >>>> gen_tcp call significantly decreases network output. >>>> Oh and I forgot to mention I use R12B-3. >>>> >>>> >>>> Hi, >>>> >>>> Without being an expert. >>>> >>>> 200-300 mb/s in small (hundreds of bytes) packets means a *lot* of >>>> system calls if you are doing a gen_tcp:send for each one. If you >>>> buffer 3 packets, you are reducing that by a factor of 3 :). I'd try >>>> to do an small test doing the same thing in C and compare the >>>> results. I think it will also eat a lot of CPU. >>>> >>>> About the proxy CPU... I'm a bit lost about it, but speculating >>>> wildly it is possible that the time spent doing the system calls >>>> that gen_tcp is doing is added to the proxy CPU process. >>>> >>>> Regards. >>>> >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >>> >> > > -- > Johnny Billquist || "I'm on a bus > || on a psychedelic trip > email: bqt@REDACTED || Reading murder books > pdp is alive! || tryin' to stay hip" - B. Idol > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Jun 25 03:33:10 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 25 Jun 2008 13:33:10 +1200 Subject: [erlang-questions] re moving nth element from a list In-Reply-To: <18092530.post@talk.nabble.com> References: <18092530.post@talk.nabble.com> Message-ID: <5FDA9E63-CCAA-4BA3-BC87-576DF2DE324C@cs.otago.ac.nz> On 25 Jun 2008, at 2:40 am, anupamk wrote: > can you please let me know what would be a more efficient approach to > removing nth element from a list. (Have the rest of the program) pretend it is gone. That's actually a serious reply. More generally, why do you want to do this? Removing the nth element from a list is going to be O(n) no matter what you do; perhaps some kind of tree structure would suit your needs better. If you were going for clarity, then remove_nth1(List, N) -> remove_nth0(List, N-1). remove_nth0(List, N) -> {Front, [_|Back]} = lists:split(N, List), Front ++ Back. I think that's what you were trying to do with remove_nth_2/2, but the clarity just wasn't there. For example, "lists:sublist(Split_right, length(Split_right))" is a slow and confusing way of writing "Split_right". If you were really going all out for speed, something like remove_nth1(List, N) when integer(N), N >= 1 -> remove_nth0_loop(N-1, List). remove_nth0(List, N) when integer(N), N >= 0 -> remove_nth0_loop(N, List). remove_nth0_loop(N, [A,B,C,D|Xs]) when N >= 4 -> [A,B,C,D | remove_nth0_loop(N-4, Xs)]; remove_nth0_loop(3, [A,B,C,_|Xs]) -> [A,B,C|Xs]; remove_nth0_loop(2, [A,B,_ |Xs]) -> [A,B |Xs]; remove_nth0_loop(1, [A,_ |Xs]) -> [A |Xs]; remove_nth0_loop(0, [_ |Xs]) -> Xs . This is (currently) body-recursive, not tail-recursive (although there is no reason in principle why it _couldn't_ be tail-recursive; the Prolog equivalent is), but if your lists are short enough that you are willing to put up with the function at all, they are short enough that this should not be a problem. From anupam.kapoor@REDACTED Wed Jun 25 07:46:24 2008 From: anupam.kapoor@REDACTED (Anupam Kapoor) Date: Wed, 25 Jun 2008 11:16:24 +0530 Subject: [erlang-questions] re moving nth element from a list In-Reply-To: <5FDA9E63-CCAA-4BA3-BC87-576DF2DE324C@cs.otago.ac.nz> (Richard A. O'Keefe's message of "Wed\, 25 Jun 2008 13\:33\:10 +1200") References: <18092530.post@talk.nabble.com> <5FDA9E63-CCAA-4BA3-BC87-576DF2DE324C@cs.otago.ac.nz> Message-ID: <87skv283tb.fsf@seldon.divitas.com> Richard A. O'Keefe wrote: On 25 Jun 2008, at 2:40 am, anupamk wrote: ,---- | > That's actually a serious reply. More generally, why do you want to | > do this? Removing the nth element from a list is going to be O(n) | > no matter what you do; perhaps some kind of tree structure would | > suit your needs better. `---- here is the (slightly long, and perhaps boring story): i wanted to get a good taste of erlang core language itself. everyone tries to get into distributed+fault-tolerant nature of erlang. before going that far, i wanted to write a simple application in it. i have written the simplex (linear programming) in couple of languages before, and i just wanted to contrast that experience in erlang. now, for doing this, i can : - either learn erlang's ffi for interfacing with external math libs - or develop my own elementary matrix/vector manipulation routines. for better or for worse, i chose the latter. now, vector-dot-products are easy, however while computing vector-cross-products, i need the 'remove-nth' thingie. sorry for boring you with this. and thank you very much for your insightful replies. kind regards anupam From wenewboy@REDACTED Wed Jun 25 07:01:38 2008 From: wenewboy@REDACTED (wenew zhang) Date: Tue, 24 Jun 2008 22:01:38 -0700 Subject: [erlang-questions] : ODBC library - link check failed on mac leopard Message-ID: <4eaa09eb0806242201o339c17e6y7ea74e50844c7c26@mail.gmail.com> ********************************************************************* ********************** APPLICATIONS DISABLED ********************** ********************************************************************* odbc : ODBC library - link check failed i use macports install the erlang, but can't start odbc:start(). so ,i port install unixodbc and then try config otp_src 12B-3 from source code,i got the warning above, and then,i install iodbc from source code, i still got "ODBC library - link check failed" on FreeBSD, use port install erlang will install iodbc automatic, how can i make odbc works? thanks a lots wenew zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Jun 25 08:41:34 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 25 Jun 2008 18:41:34 +1200 Subject: [erlang-questions] re moving nth element from a list In-Reply-To: <87skv283tb.fsf@seldon.divitas.com> References: <18092530.post@talk.nabble.com> <5FDA9E63-CCAA-4BA3-BC87-576DF2DE324C@cs.otago.ac.nz> <87skv283tb.fsf@seldon.divitas.com> Message-ID: On 25 Jun 2008, at 5:46 pm, Anupam Kapoor wrote: > i have written the simplex (linear programming) in couple of languages > before, and i just wanted to contrast that experience in erlang. now, > for doing this, i can : > > - either learn erlang's ffi for interfacing with external math libs Strictly speaking, there isn't one. Various people have been saying that a decent interface to something like (say) GSL would be nice, and it's hard to disagree (although I might be willing to give it a go (:-)). > > - or develop my own elementary matrix/vector manipulation routines. > > for better or for worse, i chose the latter. now, vector-dot-products > are easy, however while computing vector-cross-products, i need the > 'remove-nth' thingie. Eh? The vector cross product is only defined for 3-dimensional space, [A,B,C] x [D,E,F] = [B*F-C*E, C*D-A*F, A*E-B*D] (Warning: untested code.) If you mean something other than the traditional vector cross product, what might it be? For what it's worth, I wrote a simplex algorithm implementation in Burroughs Algol back in, oh, maybe 1977 or 1978, and I don't recall any cross products turning up in it at all. With hindsight, I think everything I needed was in the level 1 or level 2 BLAS, not that they existed as such back then. I'm not sure that this particular exercise will teach you anything other than "Erlang wasn't designed for this." Implementing the TFTP protocol might be more illuminating, or even the Dining Philosophers. From per@REDACTED Wed Jun 25 08:51:27 2008 From: per@REDACTED (Per Hedeland) Date: Wed, 25 Jun 2008 08:51:27 +0200 (CEST) Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <48618003.3030506@softjar.se> Message-ID: <200806250651.m5P6pRVQ043299@pluto.hedeland.org> Johnny Billquist wrote: > >No. RTT can not be used to calculate anything regarding traffic bandwidth. >You can keep sending packets until the window is exhausted, no matter what the >RTT says. The RTT is only used to calculate when to do retransmissions if you >haven't received an ACK. Well, yes and no - RTT by itself cannot be used to calculate bandwidth, and TCP itself doesn't need to "know" the bandwidth anyway, but the possible throughput is dependant on RTT: Since you can have at most one window size of un-ack'ed data outstanding, and data can't be ack'ed until it's been received:-), the throughput is bounded by the ratio of (max) window size to RTT. With only 16 bits of window size available and an RTT of 300 ms, the theoretical max throughput is 65535/0.3 bytes/s or ~ 1.75 Mbit/s. Of course this problem, a.k.a. "long fat pipe", was solved long ago as far as TCP is concerned - enter window scaling (RFC 1323), which allows for the 16 bit window size to have a unit of anything from 1 to (theoretically) 2^255 bytes. These days it should also actually work most everywhere. Nevertheless, the max window size is under the control of the TCP "user", and if the kernel and/or the application limits the size of the receive buffer to something less than 64kB, window scaling can't help. Whether this is Edwin's problem I don't know - the "fixed packet rate" observation may actually be more or less correct: As you explained, TCP doesn't ack packets, it acks bytes - but the actual *sending* of acks is definitely related to the reception of packets (or "segments" if you prefer), in particular in a one-way data transfer where there are no outgoing data packets that can have acks "piggy-backed". The details may vary, but in general in such a case an ack is sent for every other packet received, or after a ("long" - 200 ms) timeout if no packets are received. --Per Hedeland From vladdu55@REDACTED Wed Jun 25 09:24:44 2008 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 25 Jun 2008 09:24:44 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18529.883.586609.71953@hamberg.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> Message-ID: <95be1d3b0806250024x4003704pa1fbb8cd39d0b61b@mail.gmail.com> Hi! On Tue, Jun 24, 2008 at 4:23 PM, Sven-Olof Nystr|m wrote: > I have written down some thoughts and ideas on the future development > of Erlang. > FWIW, I agree wholeheartedly to the motivation of this work. The language must evolve and we'll have to try to explore all the possible futures. > Main topics: > - some notes on implementation and backward compatibility > - an alternative to Erlang's selective receive > - a simple language mechanism to allow function in-lining across > module boundaries > - a new mechanism for introducing local variables with a more cleanly > defined semantics > - a mini-language to allow the efficent implementation of low-level > algorithms A general comment is that I think it would be useful with more realistic examples. The counter process gives little insight into why channels would be useful, and the examples for the new binding mechanism are only of the form "Var = expr" instead of more general pattern matches. Maybe we should have a place where such suggestions could be presented? Not everything is ready for an EEP. Searching the list archives is slightly tedious and it's difficult to keep the discussion in the right thread. Maybe at trapexit? ---- Regarding channels, there are several other paradigms that might be interesting to explore. One is the "process as stream processor" one, where processes work just like the UNIX toolkit programs by processing input messages and sending them forward. This allows piping of processes to compose functionality. With an appropriate syntax and machinery, the programmer need only specify the relevant functionality desired. This is just a teaser for an improvement to my previous attempt at http://www.erlang.org/pipermail/erlang-questions/2007-May/026467.html :-) I will try to make available a presentation, with more details, examples and implementation proposals. (Since it's vacation time, it may take a few weeks) For example, tk:find(".", "*.erl") => tk:lines() => tk:grep("-author('me').") => tk:print() is similar to find . -name "*.erl" | grep "-author('me')." tk:lines() for example is a process conceptually running fun(OutPid) -> receive stop -> OutPid ! stop; {data, FileName} -> X = file:readfile(FileName), L = lists:split(X, "\n"), [OutPid ! {data, Row} || Row <- L] end end. Or we could even have lists:seq(1,100) => fun(X) -> {X, X*X} end => [] where the last element in the pipe means "turn the stream into a list and return it", as an alternative to lists:map(fun(X) -> {X, X*X} end, lists:seq(1, 100)) The framework would wrap lists:seq(1,100) into a process running fun(OutPid) -> [OutPid ! {data, X} || X <- lists:seq(1, 100)], OutPid ! stop end , chain together all the processes and wait for the end result. Of course this is only a sketch, we'd want flow control and things like that. best regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From lemenkov@REDACTED Wed Jun 25 09:43:31 2008 From: lemenkov@REDACTED (Peter Lemenkov) Date: Wed, 25 Jun 2008 11:43:31 +0400 Subject: [erlang-questions] : ODBC library - link check failed on mac leopard In-Reply-To: <4eaa09eb0806242201o339c17e6y7ea74e50844c7c26@mail.gmail.com> References: <4eaa09eb0806242201o339c17e6y7ea74e50844c7c26@mail.gmail.com> Message-ID: Hello! 2008/6/25 wenew zhang : > ********************************************************************* > ********************** APPLICATIONS DISABLED ********************** > ********************************************************************* > odbc : ODBC library - link check failed > i use macports install the erlang, > but can't start odbc:start(). Very strange, because odbc:start() has nothing to do with actual connection to ODBC. Looks like you've got broken erlang installation. > so ,i port install unixodbc and then try config otp_src 12B-3 from source > code,i got the warning above, > and then,i install iodbc from source code, > i still got "ODBC library - link check failed" > on FreeBSD, use port install erlang will install iodbc automatic, > how can i make odbc works? If you want to test ODBC-connection, then you must try to contact ODBC with isql, firstly. But I suspect that in your case there is a broiken erlang installation since odbc:start() not working. BTW what are actual results wen you're trying odbc:start() ? -- With best regards! From bqt@REDACTED Wed Jun 25 09:44:04 2008 From: bqt@REDACTED (Johnny Billquist) Date: Wed, 25 Jun 2008 09:44:04 +0200 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <6c2563b20806241738o64987690i10c89933c6c9070d@mail.gmail.com> References: <97619b170806190155u7ba34cafred6435841e9601e8@mail.gmail.com> <399E231E-E06E-4FCE-901A-24C6B7B2DFA8@udc.es> <97619b170806192135r5504e3a4s80179885510971d9@mail.gmail.com> <6c2563b20806241143x1fd06acdq34b14b04058428c1@mail.gmail.com> <97619b170806241200t6a1e8c35vd53494ad9f3806ff@mail.gmail.com> <019501c8d635$31455fe0$f21ea8c0@SSI.CORP> <6c2563b20806241416r71fdb225v90930bead9c95d65@mail.gmail.com> <48616E93.2010300@softjar.se> <6c2563b20806241555k269a0848n302fc7335deb0c23@mail.gmail.com> <48618003.3030506@softjar.se> <6c2563b20806241738o64987690i10c89933c6c9070d@mail.gmail.com> Message-ID: <4861F744.8080401@softjar.se> Hi, no, unfortunately I don't have an answer for their observed performance. Although I haven't really looked at it either. Time, you know... :-) But keep testing and try to figure it out. You always learn something, and hopefully you'll find the answer as well. Johnny Edwin Fine skrev: > Ok Johnny, I think I get it now. Thanks for the detailed explanation. I > wonder why the original poster (Sergej/Rapsey) is seeing such poor TCP/IP > performance? In any case, I am still going to do some more benchmarks to see > if I can understand how the different components of TCP/IP communication in > Erlang (inet:setopts() and gen_tcp) affect performance, CPU overhead and so > on. > > The reason I got into all this is because I was seeing very good performance > between two systems on a LAN, and terrible performance over a non-local > overseas link that had an RTT of about 290ms. Through various measurements > and Wireshark usage I found the link was carrying only 3.4 packets per > second, with only about 56 data bytes in each packet. When I investigated > further, I found that a function I thought was running asynchronously was > actually running synchronously inside a gen_server:call(). When I spawned > the function, I still only saw 3.4 packets per second (using Wireshark > timestamps) but each packet was now full of multiple blocks of data, not > just 56 bytes, so the actual throughput went up hugely. Nothing else > changed. When I tried to find out where the 3.4 was coming from, I > calculated 1/3.4 = 0.294ms which was (coincidentally?) the exact RTT. That's > why I thought there was a relationship between RTT and the number of > packets/second a link could carry. > > Now I have to go back and try to figure it all all over again :( unless you > can explain it to me (he said hopefully). > > Thanks > Ed > > > On Tue, Jun 24, 2008 at 7:15 PM, Johnny Billquist wrote: > >> Edwin, always happy to help out... >> >> Edwin Fine skrev: >> >>> Johnny, >>> >>> Thanks for the lesson! I am always happy to learn. Like I said, I am not >>> an >>> expert in TCP/IP. >>> >>> What I was writing about when I said that packets are acknowledged is what >>> I >>> saw in Wireshark while trying to understand performance issues. I perhaps >>> should have said "TCP/IP" instead of just "TCP". There were definitely >>> acknowledgements, but I guess they were at the IP level. >>> >> No. IP don't have any acknowledgements. IP (as well as UDP) is basically >> just sending packets without any guarantee that they will ever reach the >> other end. >> What you saw was TCP acknowledgements, but you misunderstood how they work. >> >> Think of a TCP connection as an eternal length stream of bytes. Each byte >> in this stream have a sequence number. TCP sends bytes in this stream, >> packed into IP packets. Each IP packet will have one or several bytes from >> that stream. >> TCP at the other end will acknowledge the highest ordered byte that is has >> received. How many packets it took to get to that byte is irrelevant, as is >> any retransmissions, and so on... The window size tells how many additional >> bytes from this stream can be sent, which is further on based in the point >> which the acknowledgement points at. >> >> (In reality, the sequence numbers are not infitite, but are actually a >> 32-bit number, which wraps. But since window sizes normally fits in a 16-bit >> quantity, there is no chance of ever getting back to the same sequence >> number again before it has long been passed by the time before, so no risk >> of confusion or errors there.) >> >> I wonder what the MSS is for loopback? I think it's about 1536 on my eth0 >>> interface, but not sure. >>> >> Smart implementations use the MTU - 40 of the interface as the MSS for a >> loopback connection. Otherwise the thumb of rule is that if it's on the same >> network, MSS is usually set to 1460 and 536 for destinations on other >> networks. >> This comes from the fact that the local network (usually ethernet) have an >> MTU of 1500, and the IP header is normally 20 bytes, and so is the standard >> TCP header, leaving 1460 bytes of data in an ethernet frame. >> For non-local destinations, IP requires that atleast 576 byte packets can >> go through unfragmented. The rest follows. :-) >> >> As for RTT, I sent data over a link that had a very long (290ms) RTT, and >>> that definitely limited the rate at which packets could be sent. Can RTT >>> be >>> used to calculate the theoretical maximum traffic that a link can carry? >>> For example, a satellite link with a 400ms RTT but 2 Mbps bandwidth? >>> >> No. RTT can not be used to calculate anything regarding traffic bandwidth. >> You can keep sending packets until the window is exhausted, no matter what >> the RTT says. The RTT is only used to calculate when to do retransmissions >> if you haven't received an ACK. >> The only other thing that affects packet rates are the slow start >> algorithm. That will be affected by the round trip delays, since it adds a >> throttling effect on the window, in addition to what the received says. The >> reason for it being affected by the rount trip delay is that the slow start >> window size is only increased when you get ACK packets back. >> But, assuming the link can take the load, and you don't loose a lot of >> packets, the slow start algorithm will pretty quickly stop being a factor. >> >> Johnny >> >> >> >>> Ed >>> >>> On Tue, Jun 24, 2008 at 6:00 PM, Johnny Billquist wrote: >>> >>> No. TCP don't acknowledge every packet. In fact, TCP don't acknowledge >>>> packets as such at all. TCP is not packet based. It's just that if you >>>> use >>>> IP as the carrier, IP itself it packet based. >>>> TCP can in theory generate any number of packets per second. However, the >>>> amount of unacknowledged data that can be outstanding at any time is >>>> limited >>>> by the transmit window. Each packet carries a window size, which is how >>>> much >>>> more data that can be accepted by the reciever. TCP can (is allowed to) >>>> send >>>> that much data and no more. >>>> >>>> The RTT calculations are used for figuring out how long to wait before >>>> doing retransmissions. You also normally have a slow start transmission >>>> algorithm which prevents the sender from even using the full window size >>>> from the start, as a way of avoiding congestions. That is used in >>>> combination with a backoff algorithm when retransmissions are needed to >>>> further decrease congestions, but all of this only really comes into >>>> effect >>>> if you start loosing data, and TCP actually needs to do retransmissions. >>>> >>>> Another thing you have is an algorithm called Nagle, which tries to >>>> collect >>>> small amount of data sent into larger packets before sending it, so that >>>> you >>>> don't flood the net with silly small packets. >>>> >>>> One addisional detail is that receivers normally, when the receive >>>> buffers >>>> becomes full, don't announce newly freed space immediately, since that is >>>> normally rather small amounts, but instead wait a while, until a larger >>>> part >>>> of the receive buffer is free, so that the sender actually can send some >>>> full sized packets once it starts sending again. >>>> >>>> In addition to all this, you also have a max segment size which is >>>> negotiated between the TCP ends, which limit the size of a single IP >>>> packet >>>> sent by the TCP protocol. This is done in order to try to avoid packet >>>> fragmentation. >>>> >>>> So the window size is actually a flow control mechanism, and is in >>>> reality >>>> limiting the amount of data that can be sent. And it varies all the time. >>>> And the number of packets that will be used for sending that much data is >>>> determined by the MSS (Max Segment Size). >>>> >>>> Sorry for the long text on how TCP works. :-) >>>> >>>> Johnny >>>> >>>> Edwin Fine wrote: >>>> >>>> David, >>>>> Thanks for trying out the benchmark. >>>>> >>>>> With my limited knowledge of TCP/IP, I believe you are seeing the >>>>> 300,000 >>>>> limit because TCP/IP requires acknowledgements to each packet, and >>>>> although >>>>> it can batch up multiple acknowledgements in one packet, there is a >>>>> theoretical limit of packets per seconds beyond which it cannot go due >>>>> to >>>>> the laws of physics. I understand that limit is determined by the >>>>> Round-Trip >>>>> Time (RTT), which can be shown by ping. On my system, pinging 127.0.0.1< >>>>> http://127.0.0.1> gives a minimum RTT of 0.018 ms (out of 16 pings). >>>>> That >>>>> means that the maximum number of packets that can make it to and dest >>>>> and >>>>> back per second is 1/0.000018 seconds, or 55555 packets per second. The >>>>> TCP/IP stack is evidently packing 5 or 6 blocks into each packet to get >>>>> the >>>>> 300K blocks/sec you are seeing. Using Wireshark or Ethereal would >>>>> confirm >>>>> this. I am guessing that this means that the TCP window is about 6 * >>>>> 1000 >>>>> bytes or 6KB. >>>>> >>>>> What I neglected to tell this group is that I have modified the Linux >>>>> sysctl.conf as follows, which might have had an effect (like I said, I >>>>> am >>>>> not an expert): >>>>> >>>>> # increase Linux autotuning TCP buffer limits >>>>> # min, default, and max number of bytes to use >>>>> # set max to at least 4MB, or higher if you use very high BDP paths >>>>> net.ipv4.tcp_rmem = 4096 87380 16777216 >>>>> net.ipv4.tcp_wmem = 4096 32768 16777216 >>>>> >>>>> When I have more time, I will vary a number of different Erlang TCP/IP >>>>> parameters and get a data set together that gives a broader picture of >>>>> the >>>>> effect of the parameters. >>>>> >>>>> Thanks again for taking the time. >>>>> >>>>> 2008/6/24 David Mercer >: >>>>> >>>>> I tried some alternative block sizes (using the blksize option). I >>>>> found that from 1 to somewhere around??maybe a bit short of??1000 >>>>> bytes, the test was able to send about 300,000 blocks in 10 seconds >>>>> regardless of size. (That means, 0.03 MB/sec for block size of 1, >>>>> 0.3 MB/sec for block size of 10, 3 MB/sec for block size of 100, >>>>> etc.) I suspect the system was CPU bound at those levels. >>>>> >>>>> >>>>> Above 1000, the number of blocks sent seemed to decrease, though >>>>> this was more than offset by the increased size of the blocks. >>>>> Above >>>>> about 10,000 byte blocks (may have been less, I didn't check >>>>> any value between 4,000 and 10,000), however, performance peaked and >>>>> block size no longer mattered: it always sent between 70 and 80 >>>>> MB/sec. My machine is clearly slower than Edwin's? >>>>> >>>>> >>>>> DBM >>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------------ >>>>> >>>>> *From:* erlang-questions-bounces@REDACTED >>>>> >>>>> [mailto:erlang-questions-bounces@REDACTED >>>>> ] *On Behalf Of *Rapsey >>>>> *Sent:* Tuesday, June 24, 2008 14:01 >>>>> *To:* erlang-questions@REDACTED >>>> *Subject:* Re: [erlang-questions] why is gen_tcp:send slow? >>>>> >>>>> >>>>> You're using very large packets. I think the results would be much >>>>> more telling if the packets would be a few kB at most. That is >>>>> closer to most real life situations. >>>>> >>>>> >>>>> Sergej >>>>> >>>>> On Tue, Jun 24, 2008 at 8:43 PM, Edwin Fine >>>>> >>>> > wrote: >>>>> >>>>> I wrote a small benchmark in Erlang to see how fast I could get >>>>> socket communications to go. All the benchmark does is pump the same >>>>> buffer to a socket for (by default) 10 seconds. It uses {active, >>>>> once} each time, just like you do. >>>>> >>>>> Server TCP options: >>>>> {active, once}, >>>>> {reuseaddr, true}, >>>>> {packet, 0}, >>>>> {packet_size, 65536}, >>>>> {recbuf, 1000000} >>>>> >>>>> Client TCP options: >>>>> {packet, raw}, >>>>> {packet_size, 65536}, >>>>> {sndbuf, 1024 * 1024}, >>>>> {send_timeout, 3000} >>>>> >>>>> Here are some results using Erlang R12B-3 (erl +K true in the Linux >>>>> version): >>>>> >>>>> Linux (Ubuntu 8.10 x86_64, Intel Core 2 Q6600, 8 GB): >>>>> - Using localhost (127.0.0.1 ): 7474.14 MB in >>>>> >>>>> 10.01 secs (746.66 MB/sec) >>>>> - Using 192.168.x.x IP address: 8064.94 MB in 10.00 secs (806.22 >>>>> MB/sec) [Don't ask me why it's faster than using loopback, I >>>>> repeated the tests and got the same result] >>>>> >>>>> Windows XP SP3 (32 bits), Intel Core 2 Duo E6600: >>>>> - Using loopback: 2166.97 MB in 10.02 secs (216.35 MB/sec) >>>>> - Using 192.168.x.x IP address: 2140.72 MB in 10.02 secs (213.75 >>>>> MB/sec) >>>>> - On Gigabit Ethernet to the Q6600 Linux box: 1063.61 MB in 10.02 >>>>> secs (106.17 MB/sec) using non-jumbo frames. I don't think my router >>>>> supports jumbo frames. >>>>> >>>>> There's undoubtedly a huge discrepancy between the two systems, >>>>> whether because of kernel poll in Linux, or that it's 64 bits, or >>>>> unoptimized Windows TCP/IP flags, I don't know. I don't believe it's >>>>> the number of CPUs (there's only 1 process sending and one >>>>> receiving), or the CPU speed (they are both 2.4 GHz Core 2s). >>>>> >>>>> Maybe some Erlang TCP/IP gurus could comment. >>>>> >>>>> I've attached the code for interest. It's not supposed to be >>>>> production quality, so please don't beat me up :) although I am >>>>> always open to suggestions for improvement. If you do improve it, >>>>> I'd like to see what you've done. Maybe there is another simple >>>>> Erlang tcp benchmark program out there (i.e. not Tsung), but I >>>>> couldn't find one in a cursory Google search. >>>>> >>>>> To run: >>>>> >>>>> VM1: >>>>> >>>>> tb_server:start(Port, Opts). >>>>> tb_server:stop() to stop. >>>>> >>>>> Port = integer() >>>>> Opts = []|[opt()] >>>>> opt() = {atom(), term()} (Accepts inet setopts options, too) >>>>> >>>>> The server prints out the transfer rate (for simplicity). >>>>> >>>>> VM2: >>>>> tb_client(Host, Port, Opts). >>>>> >>>>> Host = atom()|string() hostname or IP address >>>>> Port, Opts as in tb_server >>>>> >>>>> Runs for 10 seconds, sending a 64K buffer as fast as possible to >>>>> Host/Port. >>>>> You can change this to 20 seconds (e.g.) by adding the tupls >>>>> {time_limit, 20000} to Opts. >>>>> You can change buffer size by adding the tuple {blksize, Bytes} to >>>>> Opts. >>>>> >>>>> 2008/6/20 Rapsey >: >>>>> >>>>> All data goes through nginx which acts as a proxy. Its CPU >>>>> consumption is never over 1%. >>>>> >>>>> >>>>> Sergej >>>>> >>>>> >>>>> On Thu, Jun 19, 2008 at 9:35 PM, Javier Par?s Fern?ndez >>>>> > wrote: >>>>> >>>>> >>>>> El 19/06/2008, a las 20:06, Rapsey escribi?: >>>>> >>>>> >>>>> It loops from another module, that way I can update the code at >>>>> any time without disrupting anything. >>>>> The packets are generally a few hundred bytes big, except >>>>> keyframes which tend to be in the kB range. I haven't tried >>>>> looking with wireshark. Still it seems a bit odd that a large >>>>> CPU consumption would be the symptom. The traffic is strictly >>>>> one way. Either someone is sending the stream or receiving it. >>>>> The transmit could of course be written with a passive receive, >>>>> but the code would be significantly uglier. I'm sure someone >>>>> here knows if setting {active, once} every packet is CPU >>>>> intensive or not. >>>>> It seems the workings of gen_tcp is quite platform dependent. If >>>>> I run the code in windows, sending more than 128 bytes per >>>>> gen_tcp call significantly decreases network output. >>>>> Oh and I forgot to mention I use R12B-3. >>>>> >>>>> >>>>> Hi, >>>>> >>>>> Without being an expert. >>>>> >>>>> 200-300 mb/s in small (hundreds of bytes) packets means a *lot* of >>>>> system calls if you are doing a gen_tcp:send for each one. If you >>>>> buffer 3 packets, you are reducing that by a factor of 3 :). I'd try >>>>> to do an small test doing the same thing in C and compare the >>>>> results. I think it will also eat a lot of CPU. >>>>> >>>>> About the proxy CPU... I'm a bit lost about it, but speculating >>>>> wildly it is possible that the time spent doing the system calls >>>>> that gen_tcp is doing is added to the proxy CPU process. >>>>> >>>>> Regards. >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------------ >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> >>>> >> -- >> Johnny Billquist || "I'm on a bus >> || on a psychedelic trip >> email: bqt@REDACTED || Reading murder books >> pdp is alive! || tryin' to stay hip" - B. Idol >> >> > -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: bqt@REDACTED || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol From erlang@REDACTED Wed Jun 25 09:59:22 2008 From: erlang@REDACTED (Peter Lund) Date: Wed, 25 Jun 2008 09:59:22 +0200 Subject: [erlang-questions] re moving nth element from a list In-Reply-To: <18092530.post@talk.nabble.com> References: <18092530.post@talk.nabble.com> Message-ID: <4861FADA.5090000@lundata.se> Use OTP stdlib! Remove = R = fun(L,N) -> string:substr(L,1,N-1)++string:substr(L,N+1) end. 1> L=lists:seq(1,10). [1,2,3,4,5,6,7,8,9,10] 2> R(L,2). [1,3,4,5,6,7,8,9,10] 3> R(L,10). [1,2,3,4,5,6,7,8,9] 4> R(L,1). [2,3,4,5,6,7,8,9,10] (if performance is an issue, measure different approaches) /Peter anupamk skrev: > hi all, > > can you please let me know what would be a more efficient approach to > removing nth element from a list. > > here are 2 versions that i have come up with: > > ,---- > | %% remove nth element from a list > | remove_nth(L, N) -> > | do_remove_nth(L, N, 1, []). > | > | do_remove_nth([_ | Rest], N, Start, Result) when N =:= Start -> > | do_remove_nth(Rest, N, Start+1, Result); > | > | do_remove_nth([First | Rest], N, Start, Result) -> > | do_remove_nth(Rest, N, Start+1, [First | Result]); > | > | do_remove_nth([], _, _, Result) -> > | lists:reverse(Result). > | > | remove_nth_2(L, N) -> > | {Split_left, Split_right} = lists:split(N, L), > | > | lists:append(lists:sublist(Split_left, length(Split_left)-1), > | lists:sublist(Split_right, length(Split_right))). > `---- > > i will be more than happy to know of other approaches to do the same > too. > > thanks > kind regards > anupam > > ps: i posted this message earlier, but i didn't see it appear on the list. > apologies if there are multiple copies of it. > From bqt@REDACTED Wed Jun 25 10:01:33 2008 From: bqt@REDACTED (Johnny Billquist) Date: Wed, 25 Jun 2008 10:01:33 +0200 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <200806250651.m5P6pRVQ043299@pluto.hedeland.org> References: <200806250651.m5P6pRVQ043299@pluto.hedeland.org> Message-ID: <4861FB5D.8010208@softjar.se> Per Hedeland skrev: > Johnny Billquist wrote: >> No. RTT can not be used to calculate anything regarding traffic bandwidth. >> You can keep sending packets until the window is exhausted, no matter what the >> RTT says. The RTT is only used to calculate when to do retransmissions if you >> haven't received an ACK. > > Well, yes and no - RTT by itself cannot be used to calculate bandwidth, > and TCP itself doesn't need to "know" the bandwidth anyway, but the > possible throughput is dependant on RTT: Since you can have at most one > window size of un-ack'ed data outstanding, and data can't be ack'ed > until it's been received:-), the throughput is bounded by the ratio of > (max) window size to RTT. With only 16 bits of window size available and > an RTT of 300 ms, the theoretical max throughput is 65535/0.3 bytes/s or > ~ 1.75 Mbit/s. Yes. If you manage to fill the whole window, then the bandwidth starts to be reduced from the theoretical max of the media, and then RTT is related to how much your BW is reduced. And of course, your "ability" to fill the window is related to RTT as well. > Of course this problem, a.k.a. "long fat pipe", was solved long ago as > far as TCP is concerned - enter window scaling (RFC 1323), which allows > for the 16 bit window size to have a unit of anything from 1 to > (theoretically) 2^255 bytes. These days it should also actually work > most everywhere. Nevertheless, the max window size is under the control > of the TCP "user", and if the kernel and/or the application limits the > size of the receive buffer to something less than 64kB, window scaling > can't help. > > Whether this is Edwin's problem I don't know - the "fixed packet rate" > observation may actually be more or less correct: As you explained, TCP > doesn't ack packets, it acks bytes - but the actual *sending* of acks is > definitely related to the reception of packets (or "segments" if you > prefer), in particular in a one-way data transfer where there are no > outgoing data packets that can have acks "piggy-backed". The details may > vary, but in general in such a case an ack is sent for every other > packet received, or after a ("long" - 200 ms) timeout if no packets are > received. Hmm. Well, a tcpdump will quickly tell if the window size is zero, and he's hitting the RTT-limiting factor. But unless the receiver is acting funny, it should only announce a new window size when there is substantial space available in the receive buffer, at which time the sender should send large packets, and not lots of small ones. Oh well, someone needs to look into this a bit more obviously. I haven't even properly looked at the problem description. I just thought I'd point out some wrong assumptions on how TCP works. :-) Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: bqt@REDACTED || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol From chsu79@REDACTED Wed Jun 25 10:31:01 2008 From: chsu79@REDACTED (Christian S) Date: Wed, 25 Jun 2008 10:31:01 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18529.883.586609.71953@hamberg.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> Message-ID: On Tue, Jun 24, 2008 at 4:23 PM, Sven-Olof Nystr|m wrote: > I have written down some thoughts and ideas on the future development > of Erlang. This is fresh air compared to the constant complaints that only contain description of what people do not want (without the suggested replacements). > - an alternative to Erlang's selective receive I have not been following the discussions around problems related to very long message queues, why people get into the situation and such. But to me the problem of slow selective receive on large mailboxes suggests not to get into that situation to begin with, not to optimize so it doesn't hurt as much. :) A single process can only process the mailbox sequentially. Ideally, if you got a bottleneck there, throw more processes at it (i.e. cpu cores), make it scale! Another reflection is what the implementation of gen_server:call using channels would look like. The current one-time-monitor used now looks very similar to a one-time channel that is used for the synchronous reply. Almost as if one would want a monitor that is "messageable/receiveable". > - a new mechanism for introducing local variables with a more cleanly > defined semantics I like your thinking here. Very good examples showing how let-forms make code less succinct. When it comes to making changes to the Erlang syntax, I think those creating automatic refactoring tools should review it and steer in a way that make their job easier, allowing us all to get better automatic refactoring tools. The worst(?) that can happen is that we get a lisp syntax. :) > - a mini-language to allow the efficent implementation of low-level > algorithms This is very interesting. I do not understand it fully. But it is very interesting. You had me at compiling down to the Cell processor SPEs. With a language like this, then common-lisp like macros are even more interesting. My experience with performance at SIMD parallelism is that only 20% is being able to perform the SIMD operations, and the remaining 80% work is in managing memory accesses so data already in cache lines are efficiently used while it is still there. Applying approaches such as http://en.wikipedia.org/wiki/Loop_nest_optimization aid here, and then it is nice to parameterize various block-sizes for the specific model and generation of the cpu used. When sqeezing mips out of a cpu everything is game. C++ programmers typically use their templates to parameterize this. I can imagine code wanting to generate and compile these small number-crunchers at runtime for the conditions of the specific task at hand. From fuad@REDACTED Wed Jun 25 11:22:46 2008 From: fuad@REDACTED (Fuad Tabba) Date: Wed, 25 Jun 2008 21:22:46 +1200 Subject: [erlang-questions] Writing a Concurrent Binary Search Tree Data Structure in Erlang Message-ID: <909c265f0806250222q76e6651emd164ea221f15202f@mail.gmail.com> Hi, I'm trying to write a concurrent Binary Search Tree (BST) in Erlang with the operations insert/delete/contains. My goal is for multiple processes to be able to concurrently modify this structure and query it correctly. My problem is, I can't think of a way to take advantage of the inherent parallelism in the BST structure. BSTs should be easy to parallelize, since different inserts/deletes of nodes from the tree could be in completely different branches. In C for example, it's not hard to write a nonblocking implementation (lock-free) where adding nodes to the tree can happen concurrently (unless there's a conflict) using Compare and Exchange (CAS). Adding concurrency with deleting nodes is a bit more challenging but should still be doable. That said, I've only managed to think of an Erlang implementation that is capable of running contains() in parallel with an insert or delete. However, I cannot think of a way where I could have the server thread somehow distribute its updates if they are to nonconflicting branches. I feel like the solution should be a simple, recursive one, and that the answer is on the tip of my brain so to speak :) But I'm still baffled. My question is this; I can reason about shared memory objects and how concurrent accesses should be, either using locks or for the simpler cases using atomic instructions such as CAS. However, I'm finding it difficult to do that with Erlang/functional languages in general. I did a google search, couldn't really find any examples of what I'm looking for. Any tips or pointers? Here's what my code looks like (the whole thing is attached to this email). Apologies in advance for the quality of my code, still learning :) The basic idea, to be able to have contains() run in parallel with insertions and deletions, is that whenever the server gets a request to add something or remove something from the tree, its continue applying contains() to the old version until it receives the modifications back. The code for the tree itself is pretty much the same as that in Concurrent Programming in Erlang. % No changes pending for the tree parallelServer(Tree, stable, _) -> receive {Requester, insert, Key} -> Self = self(), spawn(fun() -> insertProcessor(Self, Key, Tree) end), parallelServer(Tree, changing, Requester); {Requester, delete, Key} -> Self = self(), spawn(fun() -> deleteProcessor(Self, Key, Tree) end), parallelServer(Tree, changing, Requester); {Requester, contains, Key} -> spawn(fun() -> containsProcessor(Requester, Key, Tree) end), parallelServer(Tree, stable, void); {Requester, dump} -> Requester ! Tree, parallelServer(Tree, stable, void) end; % Tree is being modified parallelServer(Tree, changing, Requester) -> receive {Requester, contains, Key} -> spawn(fun() -> containsProcessor(Requester, Key, Tree) end), parallelServer(Tree, changing, Requester); {Requester, dump} -> Requester ! Tree, parallelServer(Tree, changing, Requester); {update, NewTree} -> Requester ! done, parallelServer(NewTree, stable, void) end. % insertProcessor and deleteProcessor cannot notify the requester directly to % ensure that parallelServer knows what the new tree is like first for % consistency. insertProcessor(Server, Key, Tree) -> Server ! {update, insert(Key, Tree)}. deleteProcessor(Server, Key, Tree) -> Server ! {update, delete(Key, Tree)}. containsProcessor(Requester, Key, Tree) -> Requester ! contains(Key, Tree). % Can report it directly Thanks ! Cheers, /Fuad -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bst.erl Type: application/octet-stream Size: 5237 bytes Desc: not available URL: From vychodil.hynek@REDACTED Wed Jun 25 11:40:48 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Wed, 25 Jun 2008 11:40:48 +0200 Subject: [erlang-questions] Writing a Concurrent Binary Search Tree Data Structure in Erlang In-Reply-To: <909c265f0806250222q76e6651emd164ea221f15202f@mail.gmail.com> References: <909c265f0806250222q76e6651emd164ea221f15202f@mail.gmail.com> Message-ID: <4d08db370806250240gbf0e527n798a296292124ef8@mail.gmail.com> Because there is no shared state between processes in Erlang, I would start think about process per tree node, not process per action as usual in classic shared memory (for example C) implementations. I know, it sounds weird, but may be help you :-) 2008/6/25 Fuad Tabba : > Hi, > > I'm trying to write a concurrent Binary Search Tree (BST) in Erlang with > the operations insert/delete/contains. My goal is for multiple processes to > be able to concurrently modify this structure and query it correctly. > > My problem is, I can't think of a way to take advantage of the inherent > parallelism in the BST structure. BSTs should be easy to parallelize, since > different inserts/deletes of nodes from the tree could be in completely > different branches. In C for example, it's not hard to write a nonblocking > implementation (lock-free) where adding nodes to the tree can happen > concurrently (unless there's a conflict) using Compare and Exchange (CAS). > Adding concurrency with deleting nodes is a bit more challenging but should > still be doable. > > That said, I've only managed to think of an Erlang implementation that is > capable of running contains() in parallel with an insert or delete. However, > I cannot think of a way where I could have the server thread somehow > distribute its updates if they are to nonconflicting branches. I feel like > the solution should be a simple, recursive one, and that the answer is on > the tip of my brain so to speak :) But I'm still baffled. > > My question is this; I can reason about shared memory objects and how > concurrent accesses should be, either using locks or for the simpler cases > using atomic instructions such as CAS. However, I'm finding it difficult to > do that with Erlang/functional languages in general. I did a google search, > couldn't really find any examples of what I'm looking for. Any tips or > pointers? > > Here's what my code looks like (the whole thing is attached to this email). > Apologies in advance for the quality of my code, still learning :) > > The basic idea, to be able to have contains() run in parallel with > insertions and deletions, is that whenever the server gets a request to add > something or remove something from the tree, its continue applying > contains() to the old version until it receives the modifications back. > > The code for the tree itself is pretty much the same as that in Concurrent > Programming in Erlang. > > % No changes pending for the tree > parallelServer(Tree, stable, _) -> > receive > {Requester, insert, Key} -> > Self = self(), > spawn(fun() -> insertProcessor(Self, Key, Tree) end), > parallelServer(Tree, changing, Requester); > > {Requester, delete, Key} -> > Self = self(), > spawn(fun() -> deleteProcessor(Self, Key, Tree) end), > parallelServer(Tree, changing, Requester); > > {Requester, contains, Key} -> > spawn(fun() -> containsProcessor(Requester, Key, Tree) end), > parallelServer(Tree, stable, void); > > {Requester, dump} -> > Requester ! Tree, > parallelServer(Tree, stable, void) > end; > > % Tree is being modified > parallelServer(Tree, changing, Requester) -> > receive > {Requester, contains, Key} -> > spawn(fun() -> containsProcessor(Requester, Key, Tree) end), > parallelServer(Tree, changing, Requester); > > {Requester, dump} -> > Requester ! Tree, > parallelServer(Tree, changing, Requester); > > {update, NewTree} -> > Requester ! done, > parallelServer(NewTree, stable, void) > end. > > > % insertProcessor and deleteProcessor cannot notify the requester directly > to > % ensure that parallelServer knows what the new tree is like first for > % consistency. > > insertProcessor(Server, Key, Tree) -> > Server ! {update, insert(Key, Tree)}. > > > deleteProcessor(Server, Key, Tree) -> > Server ! {update, delete(Key, Tree)}. > > > containsProcessor(Requester, Key, Tree) -> > Requester ! contains(Key, Tree). % Can report it directly > > > > Thanks ! > > Cheers, > /Fuad > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuad@REDACTED Wed Jun 25 11:46:05 2008 From: fuad@REDACTED (Fuad Tabba) Date: Wed, 25 Jun 2008 21:46:05 +1200 Subject: [erlang-questions] Writing a Concurrent Binary Search Tree Data Structure in Erlang In-Reply-To: <4d08db370806250240gbf0e527n798a296292124ef8@mail.gmail.com> References: <909c265f0806250222q76e6651emd164ea221f15202f@mail.gmail.com> <4d08db370806250240gbf0e527n798a296292124ef8@mail.gmail.com> Message-ID: <909c265f0806250246s7abbcb2at6884f50e31c5d81c@mail.gmail.com> I understand that, but what if that structure (binary tree) is needed to store some sort of persistent state for all processes to see - reservations table for example or something like that. I know there's ETS, but what I'm trying to do is understand the basics... Thanks Hynek, /Fuad On Wed, Jun 25, 2008 at 9:40 PM, Hynek Vychodil wrote: > Because there is no shared state between processes in Erlang, I would start > think about process per tree node, not process per action as usual in > classic shared memory (for example C) implementations. > I know, it sounds weird, but may be help you :-) > > 2008/6/25 Fuad Tabba : > >> Hi, >> >> I'm trying to write a concurrent Binary Search Tree (BST) in Erlang with >> the operations insert/delete/contains. My goal is for multiple processes to >> be able to concurrently modify this structure and query it correctly. >> >> My problem is, I can't think of a way to take advantage of the inherent >> parallelism in the BST structure. BSTs should be easy to parallelize, since >> different inserts/deletes of nodes from the tree could be in completely >> different branches. In C for example, it's not hard to write a nonblocking >> implementation (lock-free) where adding nodes to the tree can happen >> concurrently (unless there's a conflict) using Compare and Exchange (CAS). >> Adding concurrency with deleting nodes is a bit more challenging but should >> still be doable. >> >> That said, I've only managed to think of an Erlang implementation that is >> capable of running contains() in parallel with an insert or delete. However, >> I cannot think of a way where I could have the server thread somehow >> distribute its updates if they are to nonconflicting branches. I feel like >> the solution should be a simple, recursive one, and that the answer is on >> the tip of my brain so to speak :) But I'm still baffled. >> >> My question is this; I can reason about shared memory objects and how >> concurrent accesses should be, either using locks or for the simpler cases >> using atomic instructions such as CAS. However, I'm finding it difficult to >> do that with Erlang/functional languages in general. I did a google search, >> couldn't really find any examples of what I'm looking for. Any tips or >> pointers? >> >> Here's what my code looks like (the whole thing is attached to this >> email). Apologies in advance for the quality of my code, still learning :) >> >> The basic idea, to be able to have contains() run in parallel with >> insertions and deletions, is that whenever the server gets a request to add >> something or remove something from the tree, its continue applying >> contains() to the old version until it receives the modifications back. >> >> The code for the tree itself is pretty much the same as that in Concurrent >> Programming in Erlang. >> >> % No changes pending for the tree >> parallelServer(Tree, stable, _) -> >> receive >> {Requester, insert, Key} -> >> Self = self(), >> spawn(fun() -> insertProcessor(Self, Key, Tree) end), >> parallelServer(Tree, changing, Requester); >> >> {Requester, delete, Key} -> >> Self = self(), >> spawn(fun() -> deleteProcessor(Self, Key, Tree) end), >> parallelServer(Tree, changing, Requester); >> >> {Requester, contains, Key} -> >> spawn(fun() -> containsProcessor(Requester, Key, Tree) end), >> parallelServer(Tree, stable, void); >> >> {Requester, dump} -> >> Requester ! Tree, >> parallelServer(Tree, stable, void) >> end; >> >> % Tree is being modified >> parallelServer(Tree, changing, Requester) -> >> receive >> {Requester, contains, Key} -> >> spawn(fun() -> containsProcessor(Requester, Key, Tree) end), >> parallelServer(Tree, changing, Requester); >> >> {Requester, dump} -> >> Requester ! Tree, >> parallelServer(Tree, changing, Requester); >> >> {update, NewTree} -> >> Requester ! done, >> parallelServer(NewTree, stable, void) >> end. >> >> >> % insertProcessor and deleteProcessor cannot notify the requester directly >> to >> % ensure that parallelServer knows what the new tree is like first for >> % consistency. >> >> insertProcessor(Server, Key, Tree) -> >> Server ! {update, insert(Key, Tree)}. >> >> >> deleteProcessor(Server, Key, Tree) -> >> Server ! {update, delete(Key, Tree)}. >> >> >> containsProcessor(Requester, Key, Tree) -> >> Requester ! contains(Key, Tree). % Can report it directly >> >> >> >> Thanks ! >> >> Cheers, >> /Fuad >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Wed Jun 25 11:57:43 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Wed, 25 Jun 2008 11:57:43 +0200 Subject: [erlang-questions] Writing a Concurrent Binary Search Tree Data Structure in Erlang In-Reply-To: <909c265f0806250246s7abbcb2at6884f50e31c5d81c@mail.gmail.com> References: <909c265f0806250222q76e6651emd164ea221f15202f@mail.gmail.com> <4d08db370806250240gbf0e527n798a296292124ef8@mail.gmail.com> <909c265f0806250246s7abbcb2at6884f50e31c5d81c@mail.gmail.com> Message-ID: <4d08db370806250257h63fa7cb7m2dbf1a8b682768e2@mail.gmail.com> You should store "persistent" state in server process. When you fall in performance problem you can partition this state in sub processes and server starts work as dispatcher. If dispatching is algorithmic you can spawn many dispatchers of course, but it usually performs better. You can repeat this fragmentation again and again on sub processes and you end up with process per node. For performance reasons there will be best solution some sort of hybrid structure. IMHO ets not performs fully scalable because I think there is only one "process" which can every time update it. On Wed, Jun 25, 2008 at 11:46 AM, Fuad Tabba wrote: > I understand that, but what if that structure (binary tree) is needed to > store some sort of persistent state for all processes to see - reservations > table for example or something like that. I know there's ETS, but what I'm > trying to do is understand the basics... > > Thanks Hynek, > /Fuad > > > On Wed, Jun 25, 2008 at 9:40 PM, Hynek Vychodil > wrote: > >> Because there is no shared state between processes in Erlang, I would >> start think about process per tree node, not process per action as usual in >> classic shared memory (for example C) implementations. >> I know, it sounds weird, but may be help you :-) >> >> 2008/6/25 Fuad Tabba : >> >>> Hi, >>> >>> I'm trying to write a concurrent Binary Search Tree (BST) in Erlang with >>> the operations insert/delete/contains. My goal is for multiple processes to >>> be able to concurrently modify this structure and query it correctly. >>> >>> My problem is, I can't think of a way to take advantage of the inherent >>> parallelism in the BST structure. BSTs should be easy to parallelize, since >>> different inserts/deletes of nodes from the tree could be in completely >>> different branches. In C for example, it's not hard to write a nonblocking >>> implementation (lock-free) where adding nodes to the tree can happen >>> concurrently (unless there's a conflict) using Compare and Exchange (CAS). >>> Adding concurrency with deleting nodes is a bit more challenging but should >>> still be doable. >>> >>> That said, I've only managed to think of an Erlang implementation that is >>> capable of running contains() in parallel with an insert or delete. However, >>> I cannot think of a way where I could have the server thread somehow >>> distribute its updates if they are to nonconflicting branches. I feel like >>> the solution should be a simple, recursive one, and that the answer is on >>> the tip of my brain so to speak :) But I'm still baffled. >>> >>> My question is this; I can reason about shared memory objects and how >>> concurrent accesses should be, either using locks or for the simpler cases >>> using atomic instructions such as CAS. However, I'm finding it difficult to >>> do that with Erlang/functional languages in general. I did a google search, >>> couldn't really find any examples of what I'm looking for. Any tips or >>> pointers? >>> >>> Here's what my code looks like (the whole thing is attached to this >>> email). Apologies in advance for the quality of my code, still learning :) >>> >>> The basic idea, to be able to have contains() run in parallel with >>> insertions and deletions, is that whenever the server gets a request to add >>> something or remove something from the tree, its continue applying >>> contains() to the old version until it receives the modifications back. >>> >>> The code for the tree itself is pretty much the same as that in >>> Concurrent Programming in Erlang. >>> >>> % No changes pending for the tree >>> parallelServer(Tree, stable, _) -> >>> receive >>> {Requester, insert, Key} -> >>> Self = self(), >>> spawn(fun() -> insertProcessor(Self, Key, Tree) end), >>> parallelServer(Tree, changing, Requester); >>> >>> {Requester, delete, Key} -> >>> Self = self(), >>> spawn(fun() -> deleteProcessor(Self, Key, Tree) end), >>> parallelServer(Tree, changing, Requester); >>> >>> {Requester, contains, Key} -> >>> spawn(fun() -> containsProcessor(Requester, Key, Tree) end), >>> parallelServer(Tree, stable, void); >>> >>> {Requester, dump} -> >>> Requester ! Tree, >>> parallelServer(Tree, stable, void) >>> end; >>> >>> % Tree is being modified >>> parallelServer(Tree, changing, Requester) -> >>> receive >>> {Requester, contains, Key} -> >>> spawn(fun() -> containsProcessor(Requester, Key, Tree) end), >>> parallelServer(Tree, changing, Requester); >>> >>> {Requester, dump} -> >>> Requester ! Tree, >>> parallelServer(Tree, changing, Requester); >>> >>> {update, NewTree} -> >>> Requester ! done, >>> parallelServer(NewTree, stable, void) >>> end. >>> >>> >>> % insertProcessor and deleteProcessor cannot notify the requester >>> directly to >>> % ensure that parallelServer knows what the new tree is like first for >>> % consistency. >>> >>> insertProcessor(Server, Key, Tree) -> >>> Server ! {update, insert(Key, Tree)}. >>> >>> >>> deleteProcessor(Server, Key, Tree) -> >>> Server ! {update, delete(Key, Tree)}. >>> >>> >>> containsProcessor(Requester, Key, Tree) -> >>> Requester ! contains(Key, Tree). % Can report it directly >>> >>> >>> >>> Thanks ! >>> >>> Cheers, >>> /Fuad >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://www.erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> >> -- >> --Hynek (Pichi) Vychodil > > > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From anupam.kapoor@REDACTED Wed Jun 25 13:08:09 2008 From: anupam.kapoor@REDACTED (Anupam Kapoor) Date: Wed, 25 Jun 2008 16:38:09 +0530 Subject: [erlang-questions] re moving nth element from a list In-Reply-To: (Richard A. O'Keefe's message of "Wed\, 25 Jun 2008 18\:41\:34 +1200") References: <18092530.post@talk.nabble.com> <5FDA9E63-CCAA-4BA3-BC87-576DF2DE324C@cs.otago.ac.nz> <87skv283tb.fsf@seldon.divitas.com> Message-ID: <87y74t93hi.fsf@seldon.divitas.com> Richard A. O'Keefe wrote: ,---- | Strictly speaking, there isn't one. Various people have been saying | that a decent interface to something like (say) GSL would be nice, and | it's hard to disagree (although I might be willing to give it a go | (:-)). `---- cool. ,---- | Eh? The vector cross product is only defined for 3-dimensional space, `---- it is defined for n-dimensional vectors too. ,---- | For what it's worth, I wrote a simplex algorithm implementation in | Burroughs Algol back in, oh, maybe 1977 or 1978, and I don't recall | any cross products turning up in it at all. With hindsight, I think | everything I needed was in the level 1 or level 2 BLAS, not that they | existed as such back then. `---- but you are ofcourse right about it not appearing in simplex at all. but once you have a dot-product, you owe it to yourself to have a cross-product too :) ,---- | I'm not sure that this particular exercise will teach you anything | other than "Erlang wasn't designed for this." Implementing the TFTP | protocol might be more illuminating, or even the Dining Philosophers. `---- that is very true indeed. but i just wanted to see how hard it was to do something it is not designed for...more fundamentally, i guess, i wanted to 'see' how i can define layers of primitive operations over simple representations, combine them etc. etc. once the whole thing is there, it would be fun to see where the bottlenecks are, then optimize those portions locally (hopefully !) lather-rinse-repeat. kind regards anupam ps: thanks for indulging me. From anupam.kapoor@REDACTED Wed Jun 25 13:12:53 2008 From: anupam.kapoor@REDACTED (Anupam Kapoor) Date: Wed, 25 Jun 2008 16:42:53 +0530 Subject: [erlang-questions] re moving nth element from a list In-Reply-To: <4861FADA.5090000@lundata.se> (Peter Lund's message of "Wed\, 25 Jun 2008 09\:59\:22 +0200") References: <18092530.post@talk.nabble.com> <4861FADA.5090000@lundata.se> Message-ID: <87tzfh939m.fsf@seldon.divitas.com> Peter Lund wrote: ,---- | > Remove = R = | > fun(L,N) -> | > string:substr(L,1,N-1)++string:substr(L,N+1) | > end. `---- cool ! thanks. i haven't looked at the 'string' module. yet. kind regards anupam From anupam.kapoor@REDACTED Wed Jun 25 13:20:00 2008 From: anupam.kapoor@REDACTED (Anupam Kapoor) Date: Wed, 25 Jun 2008 16:50:00 +0530 Subject: [erlang-questions] re moving nth element from a list In-Reply-To: <48614ED4.1080909@di.uminho.pt> ("Paulo =?iso-8859-1?Q?S=E9rg?= =?iso-8859-1?Q?io?= Almeida"'s message of "Tue\, 24 Jun 2008 20\:45\:24 +0100") References: <18092530.post@talk.nabble.com> <56a0a2840806240751h4336cd26ga09f d9e267d04aba@mail.gmail.com> <48614ED4.1080909@di.uminho.pt> Message-ID: <87prq592xr.fsf@seldon.divitas.com> Paulo S?rgio Almeida wrote: > This is one of those cases where it is not necessarily better to use > tail calls; one must measure what happens. Here are two versions: > ,---- | > Body recursive: | > | > remove(_, []) -> []; | > remove(1, [_|T]) -> T; | > remove(N, [H|T]) -> [H | remove(N-1, T)]. `---- this looks pretty nice ! thanks. ,---- | > Tail recursive: | > | > remove2(N, L) -> remove2(N, L, []). | > remove2(_, [], Acc) -> lists:reverse(Acc); | > remove2(1, [_|T], Acc) -> lists:reverse(Acc, T); | > remove2(N, [H|T], Acc) -> remove2(N-1, T, [H|Acc]). | > | > The second one seems to be better for removing near the end of a large | > list, due to lists:reverse being builtin. `---- true. kind regards anupam From camuig@REDACTED Wed Jun 25 12:47:08 2008 From: camuig@REDACTED (Andrey Shnayder) Date: Wed, 25 Jun 2008 14:47:08 +0400 Subject: [erlang-questions] adding slashes for special chars Message-ID: <3df44f150806250347g69f8bd2doccf39a2e98f291b7@mail.gmail.com> Hi, Erlang team. I want to add slashes before some characters in a string - single quote, double quote, backslash. I have not found any function that implement this, therefore I use regexp:gsub function, but I have a trouble with backslashes: *69> regexp:gsub("qwert\\asdf", "\\", "\\\\"). {error,{unterminated,"\\"}}* May you help me, please? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jozsef.berces@REDACTED Wed Jun 25 12:55:03 2008 From: jozsef.berces@REDACTED (=?iso-8859-1?Q?J=F3zsef_B=E9rces?=) Date: Wed, 25 Jun 2008 12:55:03 +0200 Subject: [erlang-questions] Dual core Message-ID: Hi, I made a few tests for the utilization of dual core CPU under Windows 2000 with Erlang/OTP R11B5. It seems that: - if I start 2 CPU consuming erlang processes within the same Erlang node they use one core only - if I start those 2 processes in different Erlang nodes then both cores are used reducing the execution time to about 60% My question is if Vista or OTP R12 brings improvement so that Erlang processes in the same Erlang node would utilize both cores? Thanks, Jozsef -------------- next part -------------- An HTML attachment was scrubbed... URL: From chsu79@REDACTED Wed Jun 25 13:36:00 2008 From: chsu79@REDACTED (Christian S) Date: Wed, 25 Jun 2008 13:36:00 +0200 Subject: [erlang-questions] Dual core In-Reply-To: References: Message-ID: http://www.erlang.org/doc/highlights.html "This document lists some highlights of Erlang 5.6/OTP R12B (unpatched), compared to the previous version Erlang 5.5/OTP R11B, with focus on things not already released as R11B patches." "The Windows version of Erlang now has SMP support. On all platforms, the SMP emulator is run by default on machines which has more than one virtual or physical processor." From vychodil.hynek@REDACTED Wed Jun 25 13:36:55 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Wed, 25 Jun 2008 13:36:55 +0200 Subject: [erlang-questions] adding slashes for special chars In-Reply-To: <3df44f150806250347g69f8bd2doccf39a2e98f291b7@mail.gmail.com> References: <3df44f150806250347g69f8bd2doccf39a2e98f291b7@mail.gmail.com> Message-ID: <4d08db370806250436w121dbdd2ubab109d36c49b7df@mail.gmail.com> It is little bit difficult. In regexp you have to escape backslash and secondary must escape backslash for erlang parser but for substituting string you have to only escape for erlang parser. Weird, so ... 5> {ok, T, _} = regexp:gsub("qwert\\asdf", "\\\\", "\\\\"). {ok,"qwert\\\\asdf",1} 6> io:format("~s~n", [T]). qwert\\asdf ok 2008/6/25 Andrey Shnayder : > Hi, Erlang team. > I want to add slashes before some characters in a string - single quote, > double quote, backslash. > I have not found any function that implement this, therefore I use > regexp:gsub function, but I have a trouble with backslashes: > > *69> regexp:gsub("qwert\\asdf", "\\", "\\\\"). > {error,{unterminated,"\\"}}* > > May you help me, please? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From camuig@REDACTED Wed Jun 25 13:52:01 2008 From: camuig@REDACTED (Andrey Shnayder) Date: Wed, 25 Jun 2008 15:52:01 +0400 Subject: [erlang-questions] adding slashes for special chars In-Reply-To: <4d08db370806250436w121dbdd2ubab109d36c49b7df@mail.gmail.com> References: <3df44f150806250347g69f8bd2doccf39a2e98f291b7@mail.gmail.com> <4d08db370806250436w121dbdd2ubab109d36c49b7df@mail.gmail.com> Message-ID: <3df44f150806250452p1e179248k653ec43ff2b1c4e2@mail.gmail.com> Thank you, Hyrek! I understand it now! 2008/6/25, Hynek Vychodil : > > It is little bit difficult. In regexp you have to escape backslash and > secondary must escape backslash for erlang parser but for substituting > string you have to only escape for erlang parser. Weird, so ... > > 5> {ok, T, _} = regexp:gsub("qwert\\asdf", "\\\\", "\\\\"). > {ok,"qwert\\\\asdf",1} > 6> io:format("~s~n", [T]). > qwert\\asdf > ok > > -- > --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From ahmed.nawras@REDACTED Wed Jun 25 14:00:19 2008 From: ahmed.nawras@REDACTED (Ahmed Ali) Date: Wed, 25 Jun 2008 16:00:19 +0400 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: References: Message-ID: Hi, You can find this logging module in: http://code.google.com/p/log4erl/. Please let me know if it didn't work. You can find instructions for using it in "README.txt" in the downloaded file. Best regards, Ahmed Al-Issaei On Sat, Jun 21, 2008 at 5:00 PM, Ahmed Ali wrote: > Hi, > > Sure. Let me clean up my code and then post it somewhere in the > internet. I'll do that within the coming week. > > Best regards, > > Ahmed Al-Issaei > > On Tue, Jun 17, 2008 at 5:49 PM, devdoer bird wrote: >> yes, that's the one I need. >> >> 2008/6/14, Ahmed Ali : >>> >>> Hi, >>> >>> I understand what you want. Having used log4j myself, I've tried to >>> find something similar for my own. Eventually, I had to write it >>> myself. This module only supports file logs but it also have support >>> for multiple log levels, including critical (fatal) level, different >>> loggers for different modules, size-based rotation...etc. >>> >>> I haven't shared the project yet, as it has some shortcomings I need >>> to address before releasing it, however, I'll announce it soon. I hope >>> this is what you're look for. I'll keep you updated. >>> >>> Best regards, >>> >>> Ahmed Al-Issaei >>> >>> 2008/6/12 devdoer bird : >>> > Hi: >>> > >>> > I wonder whether any logging module for erlang exists,like python's >>> > logging >>> > module which support :critical level setting ,filtering ,diffrent >>> > handlers >>> > (stream handler,socket handler,ram handler,file handler...). >>> > >>> > Thanks a lot! >>> > _______________________________________________ >>> > erlang-questions mailing list >>> > erlang-questions@REDACTED >>> > http://www.erlang.org/mailman/listinfo/erlang-questions >>> > >> >> > From wenewboy@REDACTED Wed Jun 25 14:06:05 2008 From: wenewboy@REDACTED (wenew zhang) Date: Wed, 25 Jun 2008 05:06:05 -0700 Subject: [erlang-questions] : ODBC library - link check failed on mac leopard In-Reply-To: References: <4eaa09eb0806242201o339c17e6y7ea74e50844c7c26@mail.gmail.com> Message-ID: <4eaa09eb0806250506t57131625yf0f24df4877900bf@mail.gmail.com> error below:do you user mac now? so i think macport doesn't add the odbc suports in erlang, enew-zhangs-macintosh:bin wenewzhang$ ./erl Erlang (BEAM) emulator version 5.6.2 [source] [async-threads:0] [kernel-poll:false] Eshell V5.6.2 (abort with ^G) 1> odbc:start(). ** exception error: undefined function odbc:start/0 2> -------------- next part -------------- An HTML attachment was scrubbed... URL: From lemenkov@REDACTED Wed Jun 25 14:17:54 2008 From: lemenkov@REDACTED (Peter Lemenkov) Date: Wed, 25 Jun 2008 16:17:54 +0400 Subject: [erlang-questions] : ODBC library - link check failed on mac leopard In-Reply-To: <4eaa09eb0806250506t57131625yf0f24df4877900bf@mail.gmail.com> References: <4eaa09eb0806242201o339c17e6y7ea74e50844c7c26@mail.gmail.com> <4eaa09eb0806250506t57131625yf0f24df4877900bf@mail.gmail.com> Message-ID: 2008/6/25 wenew zhang : > error below:do you user mac now? Yes, I'm a mac user. > so i think macport doesn't add the odbc suports in erlang, Exactly. auriga:~ petro$ erl Erlang (BEAM) emulator version 5.6.2 [source] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.6.2 (abort with ^G) 1> odbc:start(). ** exception error: undefined function odbc:start/0 2> -- With best regards! From scott.gregory@REDACTED Wed Jun 25 14:26:15 2008 From: scott.gregory@REDACTED (Scott Gregory) Date: Wed, 25 Jun 2008 08:26:15 -0400 Subject: [erlang-questions] Beginner: Windows UDP multicast receive In-Reply-To: <48614203.8080009@rldn.net> References: <000c01c8d61d$01922610$04b67230$@gregory@infiniumgroup.com> <48614203.8080009@rldn.net> Message-ID: <006301c8d6be$a9d6b510$fd841f30$@gregory@infiniumgroup.com> Thanks for the reply, but no real change - is anybody using UDP multicast receive successfully on Windows ? I presume that below the "Addr" is the Multicast-address ? And LAddr is either my actual IP or ... ? Do I not need the {multicast_loop,true} ? Thanks! Scott -----Original Message----- From: mog [mailto:mog-lists@REDACTED] Sent: June 24, 2008 2:51 PM To: Scott Gregory Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] Beginner: Windows UDP multicast receive -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I believe you are missing {active, false}. On my gnu/linux box the following works ~ {ok, Socket} = gen_udp:open(Port, [binary, {active, false}, {reuseaddr, true}, {ip, Addr}, {add_membership, {Addr, LAddr}}]), hope this helps mog -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIYUICeq+tARrxhnsRAgkUAKCd0xQtNPk8Qix/U4ExFKo59VwizwCeLRIw aBEvP9W8M8cpJNjmI8gG2gE= =t8VN -----END PGP SIGNATURE----- From svenolof@REDACTED Wed Jun 25 14:36:13 2008 From: svenolof@REDACTED (Sven-Olof Nystr|m) Date: Wed, 25 Jun 2008 14:36:13 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <95be1d3b0806250024x4003704pa1fbb8cd39d0b61b@mail.gmail.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <95be1d3b0806250024x4003704pa1fbb8cd39d0b61b@mail.gmail.com> Message-ID: <18530.15293.857971.238387@hamberg.it.uu.se> Vlad Dumitrescu writes: > A general comment is that I think it would be useful with more realistic > examples. The counter process gives little insight into why channels would > be useful, and the examples for the new binding mechanism are only of the > form "Var = expr" instead of more general pattern matches. True. OTOH, more substantial examples take time to write and might sometimes obscure the ideas. In my case, it was a choice of using brief examples or not presenting these ideas at all. I hope the examples were sufficient to explain the ideas. > Maybe we should have a place where such suggestions could be presented? Not > everything is ready for an EEP. Searching the list archives is slightly > tedious and it's difficult to keep the discussion in the right thread. Maybe > at trapexit? > > ---- > Regarding channels, there are several other paradigms that might be > interesting to explore. One is the "process as stream processor" one, where > processes work just like the UNIX toolkit programs by processing input > messages and sending them forward. This allows piping of processes to > compose functionality. With an appropriate syntax and machinery, the > programmer need only specify the relevant functionality desired. I was going to explain how this could be easily implemented in Erlang but now I see that you have already done that :-) These are of course ancient ideas; the data flow languages are very similar. You might find the paper by Jack Dennis (referred to in my paper) worth reading. > This is just a teaser for an improvement to my previous attempt at > http://www.erlang.org/pipermail/erlang-questions/2007-May/026467.html :-) > I will try to make available a presentation, with more details, examples and > implementation proposals. (Since it's vacation time, it may take a few > weeks) > Nice example snipped. Sven-Olof From icfp.publicity@REDACTED Wed Jun 25 16:01:09 2008 From: icfp.publicity@REDACTED (Matthew Fluet (ICFP Publicity Chair)) Date: Wed, 25 Jun 2008 09:01:09 -0500 Subject: [erlang-questions] ICFP08 Final Call for posters Message-ID: <53ff55480806250701v718cc0fex1cc04f417f7cbdb6@mail.gmail.com> ICFP 2008 poster session September 21, 2008 Call for presentation proposals ICFP 2008 will feature a poster session for researchers and practitioners, including students. The session will provide friendly feedback for work that is in gestation or ongoing, as well as opportunities to meet each other and exchange ideas. We welcome poster submissions on all ICFP topics, especially presentations of - applications of and to functional programming; - recent work presented at more distant venues; and - ongoing work, whether or not submitted to ICFP. There will be no formal proceedings, but presenters will be invited to submit working notes, demo code, and other materials to supplement their abstract and poster. These materials will be released informally on a Web page dedicated to the poster session. An accepted submission is not intended to replace conference or journal publication. Persons interested in presenting a poster are invited to submit a one-page abstract in SIGPLAN conference style http://www.acm.org/sigs/sigplan/authorInformation.htm to the Web site https://www.softconf.com/s08/icfp08-posters/submit.html by June 30, 2008. The program committee will review the submissions for relevance and interest, and notify the authors by July 14, 2008. Accepted posters must be presented by the authors in person on Sunday, September 21, 2008. Important dates: Submission: Monday, June 30, 2008 Notification: Monday, July 14, 2008 Presentation: Sunday, September 21, 2008 Program committee: Benjamin Pierce (University of Pennsylvania) Colin Runciman (University of York) Chung-chieh Shan (Rutgers University) From bjorn@REDACTED Wed Jun 25 16:13:19 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 25 Jun 2008 16:13:19 +0200 Subject: [erlang-questions] My favorite syntax error: semicolon before end In-Reply-To: References: Message-ID: "Doug Edmunds" writes: > In the docs: > at /doc/design_principles/fsm.html > AKA OTP Design Principles 3.2 > Version 5.6.3 > > Example: Section 3.2 > > > 1> c(code_lock). > ./code_lock.erl:26: syntax error before: 'end' Thanks! Corrected (in the development branch for R12B-4). /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From svenolof@REDACTED Wed Jun 25 17:27:23 2008 From: svenolof@REDACTED (Sven-Olof Nystr|m) Date: Wed, 25 Jun 2008 17:27:23 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: References: <18529.883.586609.71953@hamberg.it.uu.se> Message-ID: <18530.25563.606442.33206@hamberg.it.uu.se> > From: "Christian S" > > I have written down some thoughts and ideas on the future development > > of Erlang. > > This is fresh air compared to the constant complaints that only > contain description of what people do not want (without the > suggested replacements). Thanks. > > > - an alternative to Erlang's selective receive > > I have not been following the discussions around problems related > to very long message queues, why people get into the situation and > such. But to me the problem of slow selective receive on large > mailboxes suggests not to get into that situation to begin with, > not to optimize so it doesn't hurt as much. :) True. But selective receive seems almost to be designed to create that problem. > A single process can only process the mailbox > sequentially. Ideally, if you got a bottleneck there, throw more > processes at it (i.e. cpu cores), make it scale! :-) ? Searching the mailbox in parallel? Note that we still need to determine which message is the first in the mailbox that matches. Also, if we can do without selective receive, this is a problem that does not need to be solved. > Another reflection is what the implementation of gen_server:call > using channels would look like. The current one-time-monitor used > now looks very similar to a one-time channel that is used for the > synchronous reply. Almost as if one would want a monitor that is > "messageable/receiveable". As far as I can see, there would not be any problems implementing gen_server using channels. Of course, this is an example that is worth some attention. > When it comes to making changes to the Erlang syntax, I think those creating > automatic refactoring tools should review it and steer in a way that make > their job easier, allowing us all to get better automatic refactoring tools. Yes, I forgot to mention that. Tools that work with Erlang source certainly become harder to write when the language is complex. > The worst(?) that can happen is that we get a lisp syntax. :) Coming up with a syntax that is even more confusing to newbies than the current one? :-) > > > - a mini-language to allow the efficent implementation of > > > low-level algorithms > > This is very interesting. I do not understand it fully. Perhaps because it is not fully defined :-) > But it is very interesting. You had me at compiling down to the > Cell processor SPEs. See the white paper for more details. > With a language like this, then common-lisp like macros are even more > interesting. My experience with performance at SIMD parallelism is that > only 20% is being able to perform the SIMD operations, and the remaining > 80% work is in managing memory accesses so data already in cache lines > are efficiently used while it is still there. > > Applying approaches such as > http://en.wikipedia.org/wiki/Loop_nest_optimization aid here, and > then it is nice to parameterize various block-sizes for the > specific model and generation of the cpu used. When sqeezing mips > out of a cpu everything is game. C++ programmers typically use > their templates to parameterize this. Precisely the sort of thing I had in mind. > I can imagine code wanting to generate and compile these small > number-crunchers at runtime for the conditions of the specific task > at hand. Certainly a workable approach. Of course, a compiler could produce different versions of a routine depending on the size of input and other conditions, so this does not need to be visible to the programmer. OTOH, doing the same trade-offs in an application (and recompile compute-intensive code) could also work. Sven-Olof From sebastian@REDACTED Wed Jun 25 17:41:08 2008 From: sebastian@REDACTED (Sebastian Dehne) Date: Wed, 25 Jun 2008 17:41:08 +0200 Subject: [erlang-questions] Parsing binaries performance Message-ID: <48626714.6030708@dehnes.com> Hello list, I'm trying to write a parser in Erlang for a byte-stream (which I receive from the TCP socket), but I realise that my code is slow compared to the java version which I have. I've attached both version. The java prints version: Done: 93 The erlang version prints: $ erl Erlang (BEAM) emulator version 5.5.5 [source] [async-threads:0] [kernel-poll:false] Eshell V5.5.5 (abort with ^G) 1> c(test). {ok,test} 2> Data1 = <<"Somestringcontainingcrlf\r\nandsomemoretextbehind\r\n">>. <<"Somestringcontainingcrlf\r\nandsomemoretextbehind\r\n">> 3> 3> test:testwrapper(Data1, 500000). Erlang Elapsed 1570 (runtime) 1586 (wall clock) milliseconds ok Both tests are run on the same machine. I also run the test several times. Quite a difference between 93ms and 1586ms. What am I doing wrong? How should I write the code in Erlang such that it matches the performance of Java? Thanks Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: Test.java Type: text/x-java Size: 1095 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.erl Type: text/x-erlang Size: 636 bytes Desc: not available URL: From chsu79@REDACTED Wed Jun 25 17:49:08 2008 From: chsu79@REDACTED (Christian S) Date: Wed, 25 Jun 2008 17:49:08 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18530.17062.650561.451935@hamberg.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> Message-ID: > > A single process can only process the mailbox > > sequentially. Ideally, if you got a bottleneck there, throw more > > processes at it (i.e. cpu cores), make it scale! > > Searching the mailbox in parallel? Note that we still need to > determine which message is the first in the mailbox that > matches. Also, if we can do without selective receive, this is a > problem that does not need to be solved. No, having more consumers each scanning their own mailboxes. Distributing the load over multiple consumers. More consumer processes in itself is a good thing, since one can make use of concurrency to a higher degree. My own thoughts about first class mailboxes have been around the lines of: First, introducing erlang:self/1 so that Pid = self(worker) can be handed out. Second, follow the grammar of exception types, but applied to receive: loop(Status) -> receive worker: {available, Worker} -> addWorker(Status, Worker); true: {job, Job} -> dispatch(Status, Job) end. I'm pretending that the above implements a load balancer where workers register that they're available to consume a job and communicate this on a Pid made with "self(worker)". Pids from self(true) would be the same as self(). The receive clause allows scanning of the hopefully shorter worker mailbox first, not having to wade through jobs to be dispatched until messages from workers have been processed. I wonder what can of worms the above would open. From vlm@REDACTED Wed Jun 25 18:16:01 2008 From: vlm@REDACTED (Lev Walkin) Date: Wed, 25 Jun 2008 09:16:01 -0700 Subject: [erlang-questions] Parsing binaries performance In-Reply-To: <48626714.6030708@dehnes.com> References: <48626714.6030708@dehnes.com> Message-ID: <48626F41.50104@lionet.info> Sebastian Dehne wrote: > Hello list, > > I'm trying to write a parser in Erlang for a byte-stream (which I > receive from the TCP socket), but I realise that my code is slow > compared to the java version which I have. I've attached both version. Yes, Erlang is slow doing that. Try running 30000 processes in parallel in Java and measure the difference that way. You'll find yourself writing into some Java mailing list asking for a way to make Java code match Erlang performance. Meanwhile, try using +native mode if you have HiPE enabled on your platform. This might help a bit. > The java prints version: > Done: 93 > > The erlang version prints: > $ erl > Erlang (BEAM) emulator version 5.5.5 [source] [async-threads:0] > [kernel-poll:false] > > Eshell V5.5.5 (abort with ^G) > 1> c(test). > {ok,test} > 2> Data1 = <<"Somestringcontainingcrlf\r\nandsomemoretextbehind\r\n">>. > <<"Somestringcontainingcrlf\r\nandsomemoretextbehind\r\n">> > 3> > 3> test:testwrapper(Data1, 500000). > Erlang Elapsed 1570 (runtime) 1586 (wall clock) milliseconds > ok > > Both tests are run on the same machine. I also run the test several > times. Quite a difference between 93ms and 1586ms. > > What am I doing wrong? How should I write the code in Erlang such that > it matches the performance of Java? > > > Thanks > > Sebastian > > > ------------------------------------------------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From sebastian@REDACTED Wed Jun 25 18:27:34 2008 From: sebastian@REDACTED (Sebastian Dehne) Date: Wed, 25 Jun 2008 18:27:34 +0200 Subject: [erlang-questions] Parsing binaries performance In-Reply-To: <48626F41.50104@lionet.info> References: <48626714.6030708@dehnes.com> <48626F41.50104@lionet.info> Message-ID: <486271F6.1010904@dehnes.com> > Yes, Erlang is slow doing that. Try running 30000 processes in parallel > in Java and measure the difference that way. You'll find yourself > writing into some Java mailing list asking for a way to make Java > code match Erlang performance. > > Meanwhile, try using +native mode if you have HiPE enabled > on your platform. This might help a bit. Java is not made for running 30000 threads, so that comparison makes no sense, does it. Since I'm new to Erlang, I was just assuming that my erlang code was wrong and there would be a more efficient way of finding a '\r\n' in a stream. If there isn't, then that's fine for me; but I just wanted to make sure, so I asked. I'll try the +native now. From wenewboy@REDACTED Wed Jun 25 18:40:54 2008 From: wenewboy@REDACTED (wenew zhang) Date: Thu, 26 Jun 2008 00:40:54 +0800 Subject: [erlang-questions] erlang odbc always driver me crazy "freebsd 7" Message-ID: <4eaa09eb0806250940h6b9744a8r1b3b1e02f0aefdd2@mail.gmail.com> i always use odbc connect to mysql, today i update otp-R12B-2 to otp-R12B-3,my odbc configure fine, i use iodbctest to select data from mysql successful, when i start erl shell, run odbc:start(),it's works better, and then i run odbc:connect() get message: Odbc connection failed in authserver init i 100% confirm the password,user name are correct,because it's the same as my ubuntu laptop, i got this error message on ubuntu ever, when i delete the line "127.0.0.1localhost" on /etc/hosts file (that mean ping localhost fail) cause odbc connect failed, but on freebsd, ping localhost *normal* , i don't know how to fix this problem,or how can i debug the odbc and got more message? tks a lot wenew zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From wenewboy@REDACTED Wed Jun 25 18:42:52 2008 From: wenewboy@REDACTED (wenew zhang) Date: Thu, 26 Jun 2008 00:42:52 +0800 Subject: [erlang-questions] erlang odbc always driver me crazy "freebsd 7" Message-ID: <4eaa09eb0806250942l45c01ec4q5e991a78b8bbf33@mail.gmail.com> i always use odbc connect to mysql, today i update otp-R12B-2 to otp-R12B-3,my odbc configure fine, i use iodbctest to select data from mysql successful, when i start erl shell, run odbc:start(),it's works better, and then i run odbc:connect() get message: Odbc connection failed in authserver init i 100% confirm the password,user name are correct,because it's the same as my ubuntu laptop, i got this error message on ubuntu ever, when i delete the line "127.0.0.1localhost" on /etc/hosts file (that mean ping localhost fail) cause odbc connect failed, but on freebsd, ping localhost *normal* , i don't know how to fix this problem,or how can i debug the odbc and got more message? tks a lot wenew zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From lemenkov@REDACTED Wed Jun 25 18:49:14 2008 From: lemenkov@REDACTED (Peter Lemenkov) Date: Wed, 25 Jun 2008 20:49:14 +0400 Subject: [erlang-questions] erlang odbc always driver me crazy "freebsd 7" In-Reply-To: <4eaa09eb0806250940h6b9744a8r1b3b1e02f0aefdd2@mail.gmail.com> References: <4eaa09eb0806250940h6b9744a8r1b3b1e02f0aefdd2@mail.gmail.com> Message-ID: 2008/6/25 wenew zhang : > i always use odbc connect to mysql, Just FYI - there is a better way to connect ro MySQL than using ODBC - native erlang-mysql-driver: http://code.google.com/p/erlang-mysql-driver/ You'd better to use it instead of odbc. -- With best regards! From sebastian@REDACTED Wed Jun 25 18:49:47 2008 From: sebastian@REDACTED (Sebastian Dehne) Date: Wed, 25 Jun 2008 18:49:47 +0200 Subject: [erlang-questions] Parsing binaries performance In-Reply-To: <486271F6.1010904@dehnes.com> References: <48626714.6030708@dehnes.com> <48626F41.50104@lionet.info> <486271F6.1010904@dehnes.com> Message-ID: <4862772B.4030500@dehnes.com> > I'll try the +native now. Just for the record, using the latest version of Erlang and HiPE enabled resulted in: 390ms instead of the 1500ms found earlier. From francesco@REDACTED Wed Jun 25 18:58:23 2008 From: francesco@REDACTED (Francesco Cesarini) Date: Wed, 25 Jun 2008 17:58:23 +0100 Subject: [erlang-questions] Streaming the Erlang eXchange Message-ID: <4862792F.50805@erlang-consulting.com> Hi All, with less than 24 hours to go, speakers and guests are finding their way to London for the Erlang eXchange. Nice to see so many friendly faces pop by the office. For those who will not make it, we will be streaming the event through the Erlang channel on yaika.com You can find the stream on www.erlang-exchange.com starting at 8.45 UK time. The Keynotes will start at 9.15 on Thursday and Friday. If you register on Yaika.com, you will be able to ask questions remotely via IM (And if you have a webcam, we might be able to project your face on the screen). We kept this announcement to the very last minute, as the service is in beta mode we are unsure of what loads the servers can handle (We expect 500+ viewers across all the channels, but it has not been tested yet). If all goes well, expect it to be a regular occurrence of all our user group meetings and conferences. We are also taping the events and will post the talks on the eXchange site. Regards, Francesco -- http://www.erlang-consulting.com From kevin@REDACTED Wed Jun 25 18:59:39 2008 From: kevin@REDACTED (Kevin Scaldeferri) Date: Wed, 25 Jun 2008 09:59:39 -0700 Subject: [erlang-questions] Parsing binaries performance In-Reply-To: <486271F6.1010904@dehnes.com> References: <48626714.6030708@dehnes.com> <48626F41.50104@lionet.info> <486271F6.1010904@dehnes.com> Message-ID: <0A0E1F97-4D9A-47E1-9BF4-243522167F7D@scaldeferri.com> On Jun 25, 2008, at 9:27 AM, Sebastian Dehne wrote: >> Yes, Erlang is slow doing that. Try running 30000 processes in >> parallel >> in Java and measure the difference that way. You'll find yourself >> writing into some Java mailing list asking for a way to make Java >> code match Erlang performance. >> >> Meanwhile, try using +native mode if you have HiPE enabled >> on your platform. This might help a bit. > Java is not made for running 30000 threads, so that comparison makes > no > sense, does it. It makes perfect sense. Different languages are designed and optimized for different goals. Erlang is designed to scale to massive concurrency, not for performance of a single serial computation. (And, despite that, it's performance in that arena is actually generally quite good for a dynamic language.) -kevin From dnew@REDACTED Wed Jun 25 19:09:03 2008 From: dnew@REDACTED (Darren New) Date: Wed, 25 Jun 2008 10:09:03 -0700 Subject: [erlang-questions] Parsing binaries performance In-Reply-To: <48626714.6030708@dehnes.com> References: <48626714.6030708@dehnes.com> Message-ID: <48627BAF.5030007@san.rr.com> Sebastian Dehne wrote: > I'm trying to write a parser in Erlang for a byte-stream (which I > receive from the TCP socket), but I realise that my code is slow > compared to the java version which I have. I've attached both version. You might want to look at the re module, the regexp module, and/or the string module, any of which might be faster. Test, of course. Another possibility would be to try code that simply indexes into the binary instead of breaking it apart into a new binary, as that might not need to copy things around as much. Something like check_pos(Bin, Inx, Chr) -> <<_:Inx/binary, MaybeCh:integer, _/binary>>, Chr == MaybeCh. find_cr(Bin) -> find_cr(Bin, 0). find_cr(Bin, Inx) when Inx >= size(Bin) -> false; find_cr(Bin, Inx) -> check_pos(Bin, Inx, $\r) andalso check_pos(Bin, Inx+1, $\n). Completely untested, not even compiled, but that's what I'd try next. I don't know if the compiler will wind up copying any parts of the binary in check_pos or not. -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From wenewboy@REDACTED Wed Jun 25 19:30:46 2008 From: wenewboy@REDACTED (wenew zhang) Date: Thu, 26 Jun 2008 01:30:46 +0800 Subject: [erlang-questions] erlang odbc always driver me crazy "freebsd 7" In-Reply-To: References: <4eaa09eb0806250940h6b9744a8r1b3b1e02f0aefdd2@mail.gmail.com> Message-ID: <4eaa09eb0806251030r28e76f8s3acf91b11acc9d43@mail.gmail.com> tks,my friends, i find R12B-3 have a new mysql-1.0 application,and it's can connect to mysql, before i use odbc,i 'm confuse with erlang-mysql, this time,i will try mysql native driver in my project, and the ejabberd wenew zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From nem@REDACTED Wed Jun 25 20:13:35 2008 From: nem@REDACTED (Geoff Cant) Date: Wed, 25 Jun 2008 20:13:35 +0200 Subject: [erlang-questions] : ODBC library - link check failed on mac leopard In-Reply-To: (Peter Lemenkov's message of "Wed, 25 Jun 2008 16:17:54 +0400") References: <4eaa09eb0806242201o339c17e6y7ea74e50844c7c26@mail.gmail.com> <4eaa09eb0806250506t57131625yf0f24df4877900bf@mail.gmail.com> Message-ID: "Peter Lemenkov" writes: > 2008/6/25 wenew zhang : >> error below:do you user mac now? > > Yes, I'm a mac user. > >> so i think macport doesn't add the odbc suports in erlang, > > Exactly. > > auriga:~ petro$ erl > Erlang (BEAM) emulator version 5.6.2 [source] [async-threads:0] [hipe] > [kernel-poll:false] > > Eshell V5.6.2 (abort with ^G) > 1> odbc:start(). > ** exception error: undefined function odbc:start/0 > 2> This is a problem I tend to fixup in macports - the odbc application needs to link (and link check) against libiodbc not libodbc. Libiodbc is shipped as part of Mac OS X (Tiger and Leopard I'm sure of) and provides the exact same API as libodbc. It'd be nice if there was a special case for darwin/macos in the configure script that would linkcheck against iodbc. Something like: diff --git a/lib/odbc/configure.in b/lib/odbc/configure.in index 114f86a..d1df895 100644 --- a/lib/odbc/configure.in +++ lib/odbc/configure.in @@ -136,6 +136,16 @@ have_odbc_lib=no AC_SUBST(TARGET_FLAGS) case $host_os in + darwin*) + TARGET_FLAGS="-DUNIX" + AC_CHECK_LIB(pthread, pthread_create, + [AC_DEFINE(HAVE_LIBPTHREAD, 1, [Define if you have the pthread library (-lpthread).]) + LIBS="$LIBS -lpthread" + have_pthread_lib=yes]) + ODBC_LIB=-L"/usr/lib" + ODBC_INCLUDE="-I/usr/lib/include" + AC_CHECK_LIB(iodbc, SQLAllocHandle,[ODBC_LIB="$ODBC_LIB -liodbc" odbc_lib_link_sucess=yes]) + ;; win32|cygwin) TARGET_FLAGS="-DWIN32" AC_CHECK_LIB(ws2_32, main) This was prepared for R12B-2 and works : 1> application:start(odbc). ok Available as a git patch at (http://git.erlang.geek.nz/?p=erlang-otp.git;a=commit;h=5d71d8f6c8a5743422c9d9104574d7fa4945a912). Maybe something like this could be rolled into R12B-4 ? Cheers, -- Geoff Cant From paul-trapexit@REDACTED Wed Jun 25 21:53:22 2008 From: paul-trapexit@REDACTED (Paul Mineiro) Date: Wed, 25 Jun 2008 12:53:22 -0700 (PDT) Subject: [erlang-questions] mnesia slow startup Message-ID: hello. we've been scaling our cluster and we're currently at 24 nodes. lately we've noticed that when a node crashes (by running out of memory, oops, software defect), that restarting mnesia on that node takes a while. by "restarting that node", btw, i mean getting through things like mnesia_controller:try_merge_schema/1 . when we at say 4 nodes we never really noticed this. today we literally went to lunch after a node failed and wasn't (finishing) starting up, figuring a full stomach would lubricate cognition. we came back and it had resolved itself. hurray for laziness! however there are still some questions: * i'm having a hard time getting some visibility into what is causing a particular schema transaction(s) to be blocked. any tricks here? * is it expected that certain transactional protocols scale badly with the number of nodes? any feedback would be appreciated. thanks! -- p In an artificial world, only extremists live naturally. -- Paul Graham From per@REDACTED Wed Jun 25 22:24:48 2008 From: per@REDACTED (Per Hedeland) Date: Wed, 25 Jun 2008 22:24:48 +0200 (CEST) Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <4861FB5D.8010208@softjar.se> Message-ID: <200806252024.m5PKOmW2059592@pluto.hedeland.org> Johnny Billquist wrote: > >Per Hedeland skrev: >> >> Whether this is Edwin's problem I don't know - the "fixed packet rate" >> observation may actually be more or less correct: As you explained, TCP >> doesn't ack packets, it acks bytes - but the actual *sending* of acks is >> definitely related to the reception of packets (or "segments" if you >> prefer), in particular in a one-way data transfer where there are no >> outgoing data packets that can have acks "piggy-backed". The details may >> vary, but in general in such a case an ack is sent for every other >> packet received, or after a ("long" - 200 ms) timeout if no packets are >> received. > >Hmm. Well, a tcpdump will quickly tell if the window size is zero, and he's >hitting the RTT-limiting factor. Hmm hmm, I think I wasn't thinking straight when I wrote the above - I was speculating that this could be something other than the RTT-limiting factor (otherwise small vs big packets shouldn't make a difference in throughput), along the lines of "the receiver has data to ack but is delaying the sending of the ack until he recieves another segment". But this can't be the explanation I believe - if you're sending smaller packets, you can send that many more into a given window before you need to wait for an ack, so there are more "opprtunities" for the reciever to send a delayed ack. >Oh well, someone needs to look into this a bit more obviously. I haven't even >properly looked at the problem description. I just thought I'd point out some >wrong assumptions on how TCP works. :-) Agreed! --Per From vladdu55@REDACTED Wed Jun 25 23:17:32 2008 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 25 Jun 2008 23:17:32 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> Message-ID: <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> Hi, On Wed, Jun 25, 2008 at 5:49 PM, Christian S wrote: > loop(Status) -> > receive > worker: {available, Worker} -> > addWorker(Status, Worker); > true: {job, Job} -> > dispatch(Status, Job) > end. > I'm not sure I get the reason why channels would be useful. I see two possibilities: a) for performance, when each channel has a different mailbox b) to simplify the code/architecture Regarding (a), this can be achieved without any syntax change (but not without a price) and with a convention that is already in use. A kind of "poor man's channels". Channels are identified by an Erlang term (a constant) and can be created dynamically. The default one has an empty id. A channel can be a separate mailbox or a data structure superimposed the mailbox to allow traversing only selected messages. Messages are tuples where the first element is the channel id. Any message not matching that is in the default channel. The price is that receiving messages is somewhat slower, since they must be sorted into the right channel. If no channels are defined, this price is minimal and everything works as usual. Also, the receive statements' patterns must explicitly contain the channel tag as a constant or a bound variable. The above may also be utterly flawed, but it doesn't feel so right now, to me :-) Regarding (b), I can't see how Erlang code can be made simpler. IMHO, in other languages/environments, the use of channels is a way to structure the code around the fact that there aren't any processes. The channels can be asynchronous and Erlang-like coding can be used. But we already have processes, and I fail to see how a channel would be distinguished from a process like the ones we have now. Am I missing anything? best regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Wed Jun 25 23:42:24 2008 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 25 Jun 2008 23:42:24 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18530.15293.857971.238387@hamberg.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> <95be1d3b0806250024x4003704pa1fbb8cd39d0b61b@mail.gmail.com> <18530.15293.857971.238387@hamberg.it.uu.se> Message-ID: <95be1d3b0806251442s421d0aaaqac33b14080b1256a@mail.gmail.com> hi, On Wed, Jun 25, 2008 at 2:36 PM, Sven-Olof Nystr|m wrote: > Vlad Dumitrescu writes: > > Regarding channels, there are several other paradigms that might be > > interesting to explore. One is the "process as stream processor" one, > where > > processes work just like the UNIX toolkit programs by processing input > > messages and sending them forward. This allows piping of processes to > > compose functionality. With an appropriate syntax and machinery, the > > programmer need only specify the relevant functionality desired. > > I was going to explain how this could be easily implemented in Erlang > but now I see that you have already done that :-) These are of course > ancient ideas; the data flow languages are very similar. You might > find the paper by Jack Dennis (referred to in my paper) worth reading. > Of course it's not a new idea. It's just that I didn't see before that the data flow paradigm is so easily mappable onto Erlang. I think that a simple framework like I described before is relatively easy to implement. Probably the most difficult part would be the to decide a good syntax :-) The really useful stuff is more complicated: flow control, fault tolerance. Also, I suppose linear pipes are nice enough, but Hartmann pipelines (http://en.wikipedia.org/wiki/Hartmann_pipeline) are even cooler! So many ideas, so little time... :-) best regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Wed Jun 25 23:57:37 2008 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 25 Jun 2008 23:57:37 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> Message-ID: <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> 2008/6/25 Vlad Dumitrescu : > > I'm not sure I get the reason why channels would be useful. I see two > possibilities: > a) for performance, when each channel has a different mailbox > b) to simplify the code/architecture I don't think channels would simplify the code, but they might make it easier to reason about it. The thing that selective receive brings to the table is scoped message reception. The Erlang approach of pattern-matching on a single message queue is extremely flexible, but it's difficult to "prove" the code correct. For example, it is quite possible for a piece of code to consume someone else's 'DOWN' message, by ignoring the Ref part. If channels followed the same scoping rules as variables (and channels aren't just syntactic sugar on top of a single message queue), a function could not steal someone else's messages. Not that these kinds of bugs are commonplace in Erlang. It's quite easy to avoid them using simple conventions. But there is, after all, a difference between "easy to avoid" and "impossible", which can sometimes be significant. Haskell has channels, and so does .NET (mailbox objects). There is therefore an opportunity to compare programs and try to determine whether programming with channels makes for more or less readable code than erlang's selective receive. BR, Ulf W From fuad@REDACTED Thu Jun 26 00:10:03 2008 From: fuad@REDACTED (Fuad Tabba) Date: Thu, 26 Jun 2008 10:10:03 +1200 Subject: [erlang-questions] Writing a Concurrent Binary Search Tree Data Structure in Erlang In-Reply-To: <4d08db370806250257h63fa7cb7m2dbf1a8b682768e2@mail.gmail.com> References: <909c265f0806250222q76e6651emd164ea221f15202f@mail.gmail.com> <4d08db370806250240gbf0e527n798a296292124ef8@mail.gmail.com> <909c265f0806250246s7abbcb2at6884f50e31c5d81c@mail.gmail.com> <4d08db370806250257h63fa7cb7m2dbf1a8b682768e2@mail.gmail.com> Message-ID: <909c265f0806251510x60ab819aqfb074b5558d7fc46@mail.gmail.com> I think I understand what you're saying here; but I still can't wrap my head around putting that into code. Is there similar code that you know of available online? The structure doesn't have to be a BST, just anything that demonstrates the idea of having a structure that stores lots persistent data to be viewed/modified by many processes and is scalable... Thanks, /Fuad On Wed, Jun 25, 2008 at 9:57 PM, Hynek Vychodil wrote: > You should store "persistent" state in server process. When you fall in > performance problem you can partition this state in sub processes and server > starts work as dispatcher. If dispatching is algorithmic you can spawn many > dispatchers of course, but it usually performs better. You can repeat this > fragmentation again and again on sub processes and you end up with process > per node. For performance reasons there will be best solution some sort of > hybrid structure. IMHO ets not performs fully scalable because I think there > is only one "process" which can every time update it. > > > On Wed, Jun 25, 2008 at 11:46 AM, Fuad Tabba > wrote: > >> I understand that, but what if that structure (binary tree) is needed to >> store some sort of persistent state for all processes to see - reservations >> table for example or something like that. I know there's ETS, but what I'm >> trying to do is understand the basics... >> >> Thanks Hynek, >> /Fuad >> >> >> On Wed, Jun 25, 2008 at 9:40 PM, Hynek Vychodil >> wrote: >> >>> Because there is no shared state between processes in Erlang, I would >>> start think about process per tree node, not process per action as usual in >>> classic shared memory (for example C) implementations. >>> I know, it sounds weird, but may be help you :-) >>> >>> 2008/6/25 Fuad Tabba : >>> >>>> Hi, >>>> >>>> I'm trying to write a concurrent Binary Search Tree (BST) in Erlang with >>>> the operations insert/delete/contains. My goal is for multiple processes to >>>> be able to concurrently modify this structure and query it correctly. >>>> >>>> My problem is, I can't think of a way to take advantage of the inherent >>>> parallelism in the BST structure. BSTs should be easy to parallelize, since >>>> different inserts/deletes of nodes from the tree could be in completely >>>> different branches. In C for example, it's not hard to write a nonblocking >>>> implementation (lock-free) where adding nodes to the tree can happen >>>> concurrently (unless there's a conflict) using Compare and Exchange (CAS). >>>> Adding concurrency with deleting nodes is a bit more challenging but should >>>> still be doable. >>>> >>>> That said, I've only managed to think of an Erlang implementation that >>>> is capable of running contains() in parallel with an insert or delete. >>>> However, I cannot think of a way where I could have the server thread >>>> somehow distribute its updates if they are to nonconflicting branches. I >>>> feel like the solution should be a simple, recursive one, and that the >>>> answer is on the tip of my brain so to speak :) But I'm still baffled. >>>> >>>> My question is this; I can reason about shared memory objects and how >>>> concurrent accesses should be, either using locks or for the simpler cases >>>> using atomic instructions such as CAS. However, I'm finding it difficult to >>>> do that with Erlang/functional languages in general. I did a google search, >>>> couldn't really find any examples of what I'm looking for. Any tips or >>>> pointers? >>>> >>>> Here's what my code looks like (the whole thing is attached to this >>>> email). Apologies in advance for the quality of my code, still learning :) >>>> >>>> The basic idea, to be able to have contains() run in parallel with >>>> insertions and deletions, is that whenever the server gets a request to add >>>> something or remove something from the tree, its continue applying >>>> contains() to the old version until it receives the modifications back. >>>> >>>> The code for the tree itself is pretty much the same as that in >>>> Concurrent Programming in Erlang. >>>> >>>> % No changes pending for the tree >>>> parallelServer(Tree, stable, _) -> >>>> receive >>>> {Requester, insert, Key} -> >>>> Self = self(), >>>> spawn(fun() -> insertProcessor(Self, Key, Tree) end), >>>> parallelServer(Tree, changing, Requester); >>>> >>>> {Requester, delete, Key} -> >>>> Self = self(), >>>> spawn(fun() -> deleteProcessor(Self, Key, Tree) end), >>>> parallelServer(Tree, changing, Requester); >>>> >>>> {Requester, contains, Key} -> >>>> spawn(fun() -> containsProcessor(Requester, Key, Tree) end), >>>> parallelServer(Tree, stable, void); >>>> >>>> {Requester, dump} -> >>>> Requester ! Tree, >>>> parallelServer(Tree, stable, void) >>>> end; >>>> >>>> % Tree is being modified >>>> parallelServer(Tree, changing, Requester) -> >>>> receive >>>> {Requester, contains, Key} -> >>>> spawn(fun() -> containsProcessor(Requester, Key, Tree) end), >>>> parallelServer(Tree, changing, Requester); >>>> >>>> {Requester, dump} -> >>>> Requester ! Tree, >>>> parallelServer(Tree, changing, Requester); >>>> >>>> {update, NewTree} -> >>>> Requester ! done, >>>> parallelServer(NewTree, stable, void) >>>> end. >>>> >>>> >>>> % insertProcessor and deleteProcessor cannot notify the requester >>>> directly to >>>> % ensure that parallelServer knows what the new tree is like first for >>>> % consistency. >>>> >>>> insertProcessor(Server, Key, Tree) -> >>>> Server ! {update, insert(Key, Tree)}. >>>> >>>> >>>> deleteProcessor(Server, Key, Tree) -> >>>> Server ! {update, delete(Key, Tree)}. >>>> >>>> >>>> containsProcessor(Requester, Key, Tree) -> >>>> Requester ! contains(Key, Tree). % Can report it directly >>>> >>>> >>>> >>>> Thanks ! >>>> >>>> Cheers, >>>> /Fuad >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >>> >>> -- >>> --Hynek (Pichi) Vychodil >> >> >> > > > -- > --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From pfisher@REDACTED Thu Jun 26 00:17:32 2008 From: pfisher@REDACTED (Paul Fisher) Date: Wed, 25 Jun 2008 17:17:32 -0500 Subject: [erlang-questions] Writing a Concurrent Binary Search Tree Data Structure in Erlang In-Reply-To: <909c265f0806251510x60ab819aqfb074b5558d7fc46@mail.gmail.com> References: <909c265f0806250222q76e6651emd164ea221f15202f@mail.gmail.com> <4d08db370806250240gbf0e527n798a296292124ef8@mail.gmail.com> <909c265f0806250246s7abbcb2at6884f50e31c5d81c@mail.gmail.com> <4d08db370806250257h63fa7cb7m2dbf1a8b682768e2@mail.gmail.com> <909c265f0806251510x60ab819aqfb074b5558d7fc46@mail.gmail.com> Message-ID: <1214432252.6289.1.camel@localhost> On Thu, 2008-06-26 at 10:10 +1200, Fuad Tabba wrote: > I think I understand what you're saying here; but I still can't wrap > my head around putting that into code. Is there similar code that you > know of available online? The structure doesn't have to be a BST, just > anything that demonstrates the idea of having a structure that stores > lots persistent data to be viewed/modified by many processes and is > scalable... This link was posted a few days ago, but if you read "erlang process" when it talks about a separate processing node, it should give you an idea as to a workable structure. http://www.hpl.hp.com/techreports/2007/HPL-2007-193.pdf -- paul From alexey@REDACTED Thu Jun 26 03:00:08 2008 From: alexey@REDACTED (Alexey Shchepin) Date: Thu, 26 Jun 2008 04:00:08 +0300 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18530.25563.606442.33206@hamberg.it.uu.se> (Sven-Olof Nystr|m's message of "Wed, 25 Jun 2008 17:27:23 +0200") References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.25563.606442.33206@hamberg.it.uu.se> Message-ID: <87ej6lhuxz.fsf@alex64.sevcom.net> Hello, Sven-Olof! On Wed, 25 Jun 2008 17:27:23 +0200, you said: SN> True. But selective receive seems almost to be designed to create that SN> problem. I think that problem can be solved for most practical cases without drastic changes to the language. I usually see that problem when one gen_server process processing quite a big message flow calls gen_server:call to another process (or gen_tcp:send, file:write, etc). Those calls do selective receive, which can cause slippery slope effect if e.g. the flow has temporary increased, or gen_tcp:send have been blocked for some time. That's because selective receive execution time in those cases depends lineary on message queue size. Now let's see what those calls do. gen_server:call does something like: Ref = make_ref(), %% (A) Pid ! {request, Ref, ...}, receive {response, Ref, ...} -> ... end. And gen_tcp:send or file:write: %% (A) port_command(Socket, Data), receive {inet_reply, Socket, ...} -> ... end. At points (A) we can be sure, that the current message queue doesn't have messages we will match in the next receive. So if there would be a function which shifts current message pointer to the latest message, we could do Ref = make_ref(), erlang:skip_queue(), Pid ! {request, Ref, ...}, receive {response, Ref, ...} -> ... end. thus practically avoiding the dependency on a mesage queue length. I think its implementation in the beam VM can be quite simple: p->msg.save = p->msg.last; That approach has some disadvantages: you need to be accurate enough to avoid exceptions between that call and receive which can spoil another receive, and gen:do_call is actually more complicated and would need more changes. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2061 bytes Desc: not available URL: From erlang-questions_efine@REDACTED Thu Jun 26 04:50:36 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Wed, 25 Jun 2008 22:50:36 -0400 Subject: [erlang-questions] Parsing binaries performance In-Reply-To: <4862772B.4030500@dehnes.com> References: <48626714.6030708@dehnes.com> <48626F41.50104@lionet.info> <486271F6.1010904@dehnes.com> <4862772B.4030500@dehnes.com> Message-ID: <6c2563b20806251950n2a86c480o5d60a2cd059878d0@mail.gmail.com> Have you read the Efficiency Guide? It has a lot of info about binaries. Also, from the Efficiency Guide: Use the bin_opt_info option to have the compiler print a lot of information about binary optimizations. It can be given either to the compiler or erlc erlc +bin_opt_info Mod.erl or passed via an environment variable export ERL_COMPILER_OPTIONS=bin_opt_info Hope this helps. On Wed, Jun 25, 2008 at 12:49 PM, Sebastian Dehne wrote: > > I'll try the +native now. > > Just for the record, using the latest version of Erlang and HiPE enabled > resulted in: > 390ms instead of the 1500ms found earlier. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From circularfunc@REDACTED Thu Jun 26 04:33:47 2008 From: circularfunc@REDACTED (Circular Function) Date: Thu, 26 Jun 2008 02:33:47 +0000 (GMT) Subject: [erlang-questions] erlang emacsmode problem Message-ID: <286156.61083.qm@web28303.mail.ukl.yahoo.com> i have in .emacs: (setq load-path (cons? "C:/Progam Files/erl5.6.2/lib/tools-2.6.1/emacs" ????? load-path)) ????? (setq erlang-root-dir "C:/Progam Files/erl5.6.2") ????? (setq exec-path (cons "C:/Progam Files/erl5.6.2/bin" exec-path)) ????? (require 'erlang-start) ("C:\\Emacs\\emacs-22.2\\bin\\emacs.exe") Loading encoded-kb...done An error has occurred while loading `c:/Users/saftarn/AppData/Roaming/.emacs': File error: Cannot open load file, erlang-start To ensure normal operation, you should investigate and remove the cause of the error in your initialization file.? Start Emacs with the `--debug-init' option to view a complete error backtrace. For information about GNU Emacs and the GNU system, type C-h C-a. ___________________________________________________ S?k efter k?rleken! Hitta din tvillingsj?l p? Yahoo! Dejting: http://ad.doubleclick.net/clk;185753627;24584539;x?http://se.meetic.yahoo.net/index.php?mtcmk=148783 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Thu Jun 26 05:57:10 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 26 Jun 2008 15:57:10 +1200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18529.883.586609.71953@hamberg.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> Message-ID: On 25 Jun 2008, at 2:23 am, Sven-Olof Nystr|m wrote: > > Hello everyone, > > I have written down some thoughts and ideas on the future development > of Erlang. I spent last night reading "Ideas for a new Erlang". It's not clear whether it would be better to write all the responses in one message or to split them into several messages, each with its own Subject:. Let's keep it as one to start with. SELECTIVE RECEIVE VS MAILBOX OBJECTS ==================================== p1. "Consider, for example, selective receive (no other programming language requires anything similar)," Untrue. Ada and Occam should not just spring to mind but jump up and down screaming loudly "pick me, pick ME!" 9.7 Select Statements [There are four forms of the select_statement. One form provides a selective wait for one or more select_alternatives. Two provide timed and conditional entry calls. The fourth provides asynchronous transfer of control.] 9.7.1 Selective Accept [This form of the select_statement allows a combination of waiting for, and selecting from, one or more alternatives. The selection may depend on conditions associated with each alternative of the selective_accept. {time-out: See selective_accept} ] Syntax selective_accept ::= select [guard] select_alternative { or [guard] select_alternative } [ else sequence_of_statements ] end select; guard ::= when condition => select_alternative ::= accept_alternative | delay_alternative | terminate_alternative accept_alternative ::= accept_statement [sequence_of_statements] delay_alternative ::= delay_statement [sequence_of_statements] terminate_alternative ::= terminate; A selective_accept shall contain at least one accept_alternative. In addition, it can contain: ? a terminate_alternative (only one); or ? one or more delay_alternatives; or ? {else part (of a selective_accept)} an else part (the reserved word else followed by a sequence_of_statements). These three possibilities are mutually exclusive. The quoted material is from the Ada'05 (actually ISO something:2007) reference manual, however the construct was already present in Ada 81. The placement of the guard *before* the accept tells you something relevant. In Ada (and Occam), the guards serve only to selectively enable or disable which messages will be looked for *before* they are considered for accepting, not to select or postpone messages based on their content after they arrive as in Erlang. However, nothing in the Erlang language requires the guards in a 'receive' to depend on the message contents, so the Erlang selective receive can be used to do the Ada (and Occam) thing. The classic example is a bounded buffer. buffer(Status, Contents) -> receive {put,Msg} when Status /= full -> {Status1, Contents1} = add(Contents, Msg) ; {get,Who} when Status /= empty -> {Status1, Contents1, Msg} = pop(Contents), Who ! Msg end, buffer(Status1, Contents1). Without selective receive (or in Ada, selective accept), this would require three receives: one for the full case, one for the empty case, and one for the in-between case. In general, this lets us replace up to 2**n-1 receives with a single receive having n guarded alternatives. p3. "It is strange that Erlang has a feature that is on one hand so general and powerful, but yet programmers are not supposed to take advantage of it!" It would be strange, IF it were true. But it isn't. Erlang programmers ARE supposed to take advantage of selective receive. Just as selective accept is indispensable in Ada and Occam, it is indispensable in Erlang. There is no other way, in any of these three languages, to wait until one of N things happens. Note that "simplifying" Erlang receives would do nothing whatever to address the problem of mailbox overflow. Supposing Ch to be a channel, L = lists:seq(1,1000), lists:foreach(fun (X) -> lists:foreach(fun (Y) -> lists:foreach(fun (Z) -> Ch ! ok end, L) end, L) end, L) is going to be trouble even if there is only unconditional-receive-from- channel and no selective receive at all. It would be absurd to cripple the language in order to avoid a problem and then find the problem snickering on your shoulder as lively and as malevolent as ever. p3. "To the best of my knowledge, selective receive has only two uses:" The bounded buffer example above shows that this is not so. p4. Channels as first class objects. But they already are. There are three proposed functions and an operator. We'll represent a channel as a process. The only even slightly tricky thing is that we need to distinguish between other messages sent to the current process and replies sent from a channel. We'll use the rather subtle fact that a reply can only arrive when we've asked for one, and that when we ask for one, we'll wait until we get it, so there never will be one in the mailbox at other times. current() -> self(). new_channel() -> Original_Self = self(), spawn_link(fun () -> new_channel_loop(Original_Self) end). new_channel_loop(Original_Self) -> receive Message -> receive Original_Self -> Original_Self ! {self(),Message} end, end, new_channel_loop(Original_Self). ne_receive(Pid) -> Pid ! Self, receive {Pid,Message} -> Message end. ne_receive() -> receive Message -> Message end. So anyone who wants to program in this style already may. In fact it is useful to distinguish between a single-shot channel and a multi-shot channel. I hope that we can rely on Erlang garbage collection to collect any process suspended in a receive construct and not registered and not referenced by any other process. If we can, the distinction is only useful to people and not really needed by the computer. new_reply() -> Original_self = self(), spawn_link(fun() -> receive Message -> receive Original_Self -> Original_Self ! {self(),Message} end, end, end). The example f(Counter) -> Counter ! inc, Counter ! inc, Q = new_channel(), // create a new channel... counter ! {value, Q}, Value = ne_receive(Q), % since when has Erlang had BCPL-style // comments? should really be f(Counter) -> Counter ! inc, Counter ! inc, R = new_reply(), % a new single-shot channel counter ! {value, R}, Value = ne_receive(R), The big question, as always, is "what does SIMPLE mean?" and the answer is seldom simple. Nystrom claims that replacing Erlang's one-mailbox-per-process-with-selective-receive approach with processes-plus-channels-with-unconditional-receive provides "a much simpler mechanism". The *mechanism* may be simpler, though given the support for pattern matching that has to be there anyway, it is not clear to me that it really *would* be simpler. Whether it is a simpler system to *use* is another matter entirely. The thing that strikes me forcibly is that I have seen all this before. John Reppy's Concurrent ML. It's supported in SML/NJ and Mlton and perhaps other ML implementations. What Nystrom calls a "channel" is what CML calls a "mailbox". (What CML calls a "channel" is what Occam calls a "channel".) And comparing the CML code I've seen to Erlang makes Erlang look stunningly simple *in use*. GUARDS ====== p6. "The complex selective receive primitive of old Erlang has forced the Erlang designers to introduce a more restricted type of expressions called _guards_." This is fiction, and it's fiction with a major error in it. Guards are *NOT* a restricted type of expression. This was much clearer in old Erlang. Sadly, it has been obscured by the disastrous design bungle of giving guard predicates aliases that can be used in expressions and by allowing 'or' and 'and' -- which should be avoided AT ALL TIMES AND IN ALL PLACES -- in both contexts. It would be really really useful if there were a compiler option to complain loudly whenever a guard predicate's alias is used in an expression. The *syntax* of guards in Erlang is similar to expressions, but the *semantics* is very different. Consider: Guard Expression succeeds returns 'true' fails returns 'false' returns something else raises an exception Some forms that would raise an exception as an expression fail as guards. Some forms that would raise an exception as an expression succeed as guards. The order of evaluation of expressions is defined. The order of evaluation of guards is not defined. If you take a guard and treat it as an expression, expecting it to always answer 'true' or 'false', you are in for a nasty surprise: some guards that fail may raise an exception some guards that succeed may raise an exception some guards that succeed may return a value other than 'true'. The whole point of section 3.4 is "SINCE guards are restricted expressions, they ought to be unrestricted expressions" but since the assumption is flat-out false, the conclusion is unwarranted. The limitations of guards have NO intrinsic connection with receive, although they turn out to be very pleasant for receive. Erlang guards have their roots in concurrent logic programming languages such as Strand88, GHC, FCP, and Parlog. Whenever I see people demanding that guards should be complexified, I find myself wondering "do I know something about readability and maintainability that they don't, or is it the other way around?" The way I see it, if your selection process is complex enough that you *want* to write complex guards with user defined functions, then what you *need* is either - abstract patterns, or - a rewrite. Let me offer an example I saw today, in http://www.infoq.com/articles/erlang-dsl It doesn't involve a user defined guard, but it does show what happens when you have a "keep-guards-simple" mindset. Here's the original code. Its purpose is to return a function that sends a message if something has a value within a limit. to_function(Pid, Action, Quantity, Ticker, Operator, Limit) -> fun (Ticker_, Price) -> if Ticker =:= Ticker_ andalso ( ( Price < Limit andalso Operator =:= less ) orelse ( Price > Limit andalso Operator =:= greater ) ) -> Pid ! {Action, Quantity, Ticker} % place an order ; true -> erlang:display("no rule applied") end end. I looked at that and, well, it didn't make me smile. That complex guard *had* to go. A few seconds later: to_function(Pid, Action, Quantity, Ticker, less, Limit) -> fun (T, Price) when T =:= Ticker, Price < Limit -> Pid ! {Action, Quantity, Ticker} % place an order ; (_, _) -> erlang:display("no rule applied") end; to_function(Pid, Action, Quantity, Ticker, greater, Limit) -> fun (T, Price) when T =:= Ticker, Price > Limit -> Pid ! {Action, Quantity, Ticker} % place an order ; (_, _) -> erlang:display("no rule applied") end. In the original context, this now gives us the benefit of a 'compile time' failure if Operator is not 'less' or 'greater', rather than a 'run time' failure. But it gets better. This is about buying or selling stocks, and it so happens that we always want to buy when the price is LOW, and sell when the price is HIGH. So now we can write to_function(Pid, buy, Quantity, Ticker, less, Limit) -> fun (T, Price) when T =:= Ticker, Price < Limit -> Pid ! {buy, Quantity, Ticker} ; (_, _) -> erlang:display("buy rule did not apply") end; to_function(Pid, sell, Quantity, Ticker, greater, Limit) -> fun (T, Price) when T =:= Ticker, Price > Limit -> Pid ! {sell, Quantity, Ticker} ; (_, _) -> erlang:display("sell rule did not apply") end. Now we can *see* that the returned function will buy if the price is low or sell when the price is high, but not vice versa. This was not possible with the original version. What I've done here is to trade size against readability. The next step would be to replace Pid by Broker, but that's a different readability issue. LINKED MODULES ============== p6. "Erlang's record facility is rather unusual (at least in the world of functional programming languages)." If you are willing to count the Lisp family as functional, then this is historically quite untrue. I've lost count of the number of times I've written Scheme code like (define (make-node Op Left Right) (vector 'node Op Left Right)) (define (node? Node) (and (vector? Node) (= (vector-length Node) 4) (eq? (vector-ref Node 0) 'node))) (define-macro (node-op Node) `(vector-ref ,Node 1)) (define-macro (node-left Node)) `(vector-ref ,Node 2)) (define-macro (set-node-left! Node New-Left) `(vector-set! ,Node 2 ,New-Left)) ... The problems of -include have been known for a long time. My "Delenda est preprocessor" paper was written some time in July 1998 (at least, that's the date on the only copy I can find). p7. "the obvious alternative would be to expand a record definition into a set of function definitions". Well, no, because you can't use function calls as patterns. That's why abstract patterns are needed. The work I did for Fergus O'Brien back in 1998 -- I miss him, and I especially miss getting paid for thinking about Erlang! -- included a proposal that I'll now reconstruct for you, with two small changes. Two new directives replace -import(Mod, Funs). -import(Module, When, Funs). -use(Module, When, Funs). Module : an atom When : 'early' or 'late' Funs : a list of Name/Arity (or #Name/Arity) pairs. A -use directive creates a dependency (recorded in the .beam file) between this module and Module. It also notifies the compiler that this module may call any of the functions (and/or abstract patterns) listed in Funs, but if so, a module prefix is required. If When is 'late', the dependency is a load-time one: before EXECUTING this module, Module should be LOADED. Cyclic late dependencies are perfectly OK; all the members of a cycle should be loaded before running anything from any of them. This dependency exists even when Funs is empty; you might be using some kind of dynamic call to call functions from Module. The only thing that the compiler may assume about Module is that loading will have ensured that the named functions all exist, even if only as loader-generated stubs that raise an undefined function exception. If When is 'early', the dependency is a compile-time one: before COMPILING this module, Module should be COMPILED, and so if Module is changed, this module also needs recompiling. Cyclic early dependencies are not allowed. The compiler may assume about Module anything whatever that it discovers in the .beam file. (Arguably interface information should be in a separate file, not unlike the .hi files used by GHC. I propose ".ern" (for .ERlang iNfo), in honour of the late Ern Malley.) The difference between -import and -use is that -import allows (but does not require) the listed functions to be called without a module prefix, provided there is no clash with any other imported function or any function defined in this module. There may be more than one -use or -import for a Module, and there may be both -import and -use directives for the same Module. There may be both early and late dependencies on a module, but if so, 'early' wins. early/late is split out as a separate argument so that it can be determined by a macro. BINDINGS VIA A TYPE SYSTEM ========================== p8. "If we compare with the equivalent code in ... SML" But why choose SML as the comparison? Haskell's y*y where x = 42, y = x+x requires only one keyword ('where') instead of SML's four. I'm not arguing for adding 'where' to Erlang, just pointing out that comparisons against a limited choice of comparands can be misleading. As a matter of fact, I rather like this proposal, and think it might be a nice style check to have. Although it _is_ a bit dodgy that X = 1 exports X X = 1, ok does NOT export X and it's a bit of a worry that X = Y = f() is a type error in the proposed scheme. I found another problem last night, but it was 1am and I can't read what I wrote. I think this proposal should be finished and written up separately and EEPed as a style check. A MINI LANGUAGE =============== The reference to the CELL is a little inaccurate, not that it matters. (The heart of a CELL is a two-way CMT version of the PowerPC, with AltiVec.) This is an interesting idea, but there isn't really a proposal here yet. It will be very interesting indeed to discover what Apple's OpenCL looks like; in the mean time, there's CUDA. The trick with something like this is that there really isn't a whole lot of point in it unless you go native. OpenCL, of course, relies on LLVM. I suspect that - some sort of opaque numeric array values (not unlike APL) - calling out to high performance libraries like ATLAS, the CBLAS, GSL, perflib, &c might be a more effective way to take advantage of the vector processing facilities in modern machines. Whoo, wouldn't that be fun! A hybrid of Erlang and APL2! MACROS ====== p14 "I don't know of any language that combines Lisp-style macros with a syntax that isn't Lisp-like." `C (Tick-C), and arguably, Template Haskell. 'forms' in WonderPop (the Pop-2 implementation for TOPS-10 that Robert Rae worked on) are closer to Scheme's macros. From my experience with Lisp, there are macros that could have been written using Scheme syntax forms, and there are macros that make essential use of assignment. Using Nystrom's own example, you really _can't_ translate a "def-nt" form into (part of) a LALR parser in isolation; the information has to be set aside until you have seen all of the rules for that grammar. That setting aside involves some sort of data base or other mutable variable. I note that the closures proposal for Java includes bending Java syntax so that things like with_lock (Some_Lock) { statements... } which are traditionally handled in Lisp using macros are in fact procedure calls, so that inlining does practically everything you want. Here's the trick] [for] Primary (FormalParameters : [Expressions]) Statement ==> Primary(Expressions, {FormalParameters => Statement}); [for] Primary ([Expressions]) Statement ==> Primary(Expressons, {=> Statement}); where {formals => statements [expression]} will be Java's syntax for a lambda-expression. An Erlang version might look like 'do' [module:]function([patterns||][expressions]) '->' expressions 'end' => [module:]function('fun'([patterns]) '->' expressions 'end'[, expressions]) e.g. do lists:foreach(Toy || Sack) -> play_with(Toy), break(Toy), cry() end For Erlang, no big deal. For Java, a giant step. See http://www.javac.info/closures-v05.html The big deal about Template Haskell, of course, is both fitting in with *and* taking advantage of the type system, something which is not an issue in Erlang. From martin.gignac@REDACTED Thu Jun 26 06:06:07 2008 From: martin.gignac@REDACTED (Martin Gignac) Date: Thu, 26 Jun 2008 00:06:07 -0400 Subject: [erlang-questions] erlang emacsmode problem In-Reply-To: <286156.61083.qm@web28303.mail.ukl.yahoo.com> References: <286156.61083.qm@web28303.mail.ukl.yahoo.com> Message-ID: 2008/6/25 Circular Function : > i have in .emacs: > (setq load-path (cons "C:/Progam Files/erl5.6.2/lib/tools-2.6.1/emacs" > load-path)) > (setq erlang-root-dir "C:/Progam Files/erl5.6.2") > (setq exec-path (cons "C:/Progam Files/erl5.6.2/bin" exec-path)) > (require 'erlang-start) > File error: Cannot open load file, erlang-start Double check your paths. Make sure they actually exist as written. If you've been getting this error since the upgrade to R12B-3 the cause of the error should be obvious. -Martin From ok@REDACTED Thu Jun 26 06:06:03 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 26 Jun 2008 16:06:03 +1200 Subject: [erlang-questions] re moving nth element from a list In-Reply-To: <87y74t93hi.fsf@seldon.divitas.com> References: <18092530.post@talk.nabble.com> <5FDA9E63-CCAA-4BA3-BC87-576DF2DE324C@cs.otago.ac.nz> <87skv283tb.fsf@seldon.divitas.com> <87y74t93hi.fsf@seldon.divitas.com> Message-ID: <6B7EE60C-C5F2-49AE-BDE1-EB88EA5FC0EB@cs.otago.ac.nz> On 25 Jun 2008, at 11:08 pm, Anupam Kapoor wrote: > ,---- > | Eh? The vector cross product is only defined for 3-dimensional > space, > `---- > it is defined for n-dimensional vectors too. WEDGE product is defined for n-dimensional vectors (basically the antisymmetric part of the tensor product), but CROSS product is not. Since this is the operation you wanted 'remove the kth element from a list' for, what exactly is it that you are calling a cross product? ^^^^^^^ For real fun, write an Erlang function that takes a numeric vector, such as [1,2,0,3,0,4], generates a special purpose dot-function fun ([X,Y,_,Z,_,W]) -> X+2*Y+3*Z+4*W end and compiles it. From matthew.reilly@REDACTED Thu Jun 26 06:25:51 2008 From: matthew.reilly@REDACTED (Matthew Reilly) Date: Wed, 25 Jun 2008 21:25:51 -0700 Subject: [erlang-questions] Ideas for a new Erlang Message-ID: <48631A4F.6050701@sipphone.com> On Wed, 2008-06-25 at 16:33 -0700, Matthew Reilly wrote: > > On Tue, 2008-06-24 at 16:23 +0200, Sven-Olof Nystr|m wrote: > >> > > Hello everyone, >> > > >> > > I have written down some thoughts and ideas on the future development >> > > of Erlang. >> > > >> > > Main topics: >> > > >> > > - some notes on implementation and backward compatibility >> > > >> > > - an alternative to Erlang's selective receive >> > > > > In terms of Erlang's selective receive, it's possible to get around the > > O(N) look up issue w/o a language change (albeit with a slight change to > > the VM) if you restrict which patterns you use. > > > > (Review of the selective receive problem) > > The problem is that given code like: > > receive > > some_term -> some_value > > end. > > each time this receive is hit, the whole message queue is scanned. If > > there are N unmatched messages pending, then this take O(N) each time > > this bit of code is hit. > > > > However, if the receive pattern is static (i.e. it does not include an > > already bound variable), then these duplicate scans would be redundant > > -- we already know that if a message is not matched once, it won't match > > the same pattern in the future. The next time we hit the same receive, > > we'd only need to scan messages that have been added since we last did > > the same selective receive. (See below about dealing with non-static > > patterns). > > > > In this way, each message would be patterned matched at most once per > > selective receive, and thus, regardless of the size of the message > > queue, selective receives would become an amoritized O(1) task. > > > > The current message queue is implemented in the VM by a set of 3 > > pointers: "first", "save", "last". "first" points to the first item in > > the queue, "last" points to the place to add the next item, and "save" > > is a temp pointer used to traverse the message queue. > > > > A way to implement O(1) selective receives would be to add an > > (initially empty) list of saved {id,pointer} pairs to each message > > queue. When a selective receive finishes scans through the message > > queue, it stores a pointer to the next message would have processed > > (keyed to something that uniquely identifies this receive). When the > > same selective receives occurs again, it just needs to start at that > > saved location (instead of at the "first" message in the list). As an > > optimization, once the message queue becomes empty, this list can be > > wiped out. > > > > This adds one pointer's worth of space to each process. Very little > > overhead is added for the normal case where all receives have a default > > clause. > > > > This is a sketch of how the VM would need to be modified to have > > amortized O(1) selective receives. > > > > Currently a selective receive such as > > func() -> > > receive > > {high_priority, Message} -> Message > > after 0 -> no_message > > end. > > is implemented in bytecode such as: > > {label,14}. > > {loop_rec,{f,16},{x,0}}. > > {test,is_tuple,{f,15},[{x,0}]}. > > {test,test_arity,{f,15},[{x,0},2]}. > > {get_tuple_element,{x,0},0,{x,1}}. > > {get_tuple_element,{x,0},1,{x,2}}. > > {test,is_eq_exact,{f,15},[{x,1},{atom,high_priority}]}. > > remove_message. > > {move,{x,2},{x,0}}. > > return. > > {label,15}. > > {loop_rec_end,{f,14}}. > > {label,16}. > > timeout. > > {move,{atom,no_message},{x,0}}. > > > > Here the "loop_rec" opcode returns the message pointed to by the "save" > > pointer (which is normally the same as "first"). > > If the message queue is empty, it jumps to the "timeout" opcode which > > resets the "save" pointer to "first". > > If the message queue is not empty, the message is pattern matched > > against {high_priority,_}. > > If it doesn't match, it calls the opcode "loop_rec_end" when sets > > "save" to "save->next", and jumps back to the "loop_rec" opcode. > > If it does match, it returns the second item of the tuple, and calls > > the "remove_message" opcode which removes the message from the queue and > > also sets "save" to "first". > > > > Suppose we introduce two new opcodes: > > restore_queue_pos > > save_queue_pos > > > > save_queue_pos stores the current value of "save" indexed against some > > unique key. > > restore_queue_pos returns either the value saved by save_queue_pos > > (possibly modified by remove_message) or "first" if there was no saved > > value. > > The remove_message opcode needs be modified to update any saved queues > > positions (in case they point to the message that is being deleted) or > > remove them all if the queue becomes empty. > > > > The VM code for the same selected receive using these new opcodes: > > > > {restore_queue_pos,XXX}. > > {label,14}. > > {loop_rec,{f,16},{x,0}}. > > {test,is_tuple,{f,15},[{x,0}]}. > > {test,test_arity,{f,15},[{x,0},2]}. > > {get_tuple_element,{x,0},0,{x,1}}. > > {get_tuple_element,{x,0},1,{x,2}}. > > {test,is_eq_exact,{f,15},[{x,1},{atom,high_priority}]}. > > {save_queue_pos,XXX}. > > remove_message. > > {move,{x,2},{x,0}}. > > return. > > {label,15}. > > {loop_rec_end,{f,14}}. > > {label,16}. > > timeout. > > {move,{atom,ok},{x,0}}. > > In the above, XXX represents a value unique to the selective receive and > > is generated an unspecified way. > > > > Care would need to be taken for dynamic code reloading. E.g. when a > > module is reloaded, the VM might want to reset all the values previously > > stored by save_queue_pos. > > > > Static vs. Non-Static selective receives > > > This technique will fail for receives that contain bound variables. e.g. > > Ref = make_ref(), > > Pid ! {command, Ref}, > > receive > > {some_rather_unique_response_atom, Ref} -> ... > > after Timeout -> ... > > end > > This can not use the above technique since each time this runs, the > > receive is matching against a different reference. > > > > It's possible to re-write the code to: > > func1(Pid) -> > > Ref = make_ref(), > > Pid ! {command,Ref}, > > func2(Ref,Pid). > > > > func2(Ref1,Pid) -> > > % This is now a static pattern since Ref2 is unbound > > receive > > {some_rather_unique_response_atom, Ref2} -> > > if > > Ref1 == Ref2 -> > > ...; > > true -> > > % We drop messages the we wouldn't have before > > func2(Ref1,Pid) > > end > > after Timeout -> ... > > end. > > > Actually, there's a better way of handling that particular case. One of the most common selective receive pattern is one that has a bound variable containing a freshly generated reference. Since, semantically, references are supposed to be unique, when doing a the receive, we can ignore all the messages that existed in our queue before the reference was created (there is no proper way that any of those messages could contain the reference). Note: it's techincally possible for an existing message to contain a reference that has not yet been created (e.g. if the reference was stored in mnesia and a node restarted), but we wouldn't want to match those anyway. This could be implemented with another new opcode "last_queue_pos" which sets "save" to "last". For example, this code: func(Pid) -> MonRef = erlang:monitor(process,Pid), Ref = make_ref(), Pid ! {request, Ref}, receive {reply, Ref} -> got_response; {'DOWN',MonRef,_,_,_} -> pid_is_gone after 1000 -> timeout end. could be implemented as this bytecode: {function, func, 1, 18}. {label,17}. {func_info,{atom,small},{atom,func},1}. {label,18}. {allocate_zero,2,1}. {move,{x,0},{x,1}}. {move,{atom,process},{x,0}}. {move,{x,1},{y,1}}. {'%live',2}. last_queue_pos. {call_ext,2,{extfunc,erlang,monitor,2}}. {move,{x,0},{y,0}}. {'%live',0}. {call_ext,0,{extfunc,erlang,make_ref,0}}. {test_heap,3,1}. {put_tuple,2,{x,1}}. {put,{atom,request}}. {put,{x,0}}. {move,{x,0},{x,2}}. {move,{y,1},{x,0}}. {move,{x,2},{y,1}}. {'%live',2}. send. {label,19}. {loop_rec,{f,23},{x,0}}. {test,is_tuple,{f,22},[{x,0}]}. {select_tuple_arity,{x,0},{f,22},{list,[2,{f,20},5,{f,21}]}}. {label,20}. {get_tuple_element,{x,0},0,{x,1}}. {get_tuple_element,{x,0},1,{x,2}}. {test,is_eq_exact,{f,22},[{x,1},{atom,reply}]}. {test,is_eq_exact,{f,22},[{x,2},{y,1}]}. remove_message. {move,{atom,got_response},{x,0}}. {deallocate,2}. return. {label,21}. {get_tuple_element,{x,0},0,{x,1}}. {get_tuple_element,{x,0},1,{x,2}}. {test,is_eq_exact,{f,22},[{x,1},{atom,'DOWN'}]}. {test,is_eq_exact,{f,22},[{x,2},{y,0}]}. remove_message. {move,{atom,pid_is_gone},{x,0}}. {deallocate,2}. return. {label,22}. {loop_rec_end,{f,19}}. {label,23}. {wait_timeout,{f,19},{integer,1000}}. timeout. {move,{atom,timeout},{x,0}}. {deallocate,2}. return. The compiler just needs to know which BIFs return new references (e.g. erlang:monitor/2 and make_ref/0), and detect if the results of these functions are used in *each* branch of a receive. If so, it could do a last_queue_pos before the BIF that returns the reference, so that the receive would start with the first message that arrived after the last_queue_pos was called. > > Because we now drop certain messages, this is not exactly semantically > > the same as the first version. However, in general, if a message does > > match this pattern, it's most likely a response to a previous call on > > which we timed out. The approriate action in that case would be to drop > > the message. > > > > > > > >> > > >> > > - a simple language mechanism to allow function in-lining across >> > > module boundaries >> > > >> > > - a new mechanism for introducing local variables with a more cleanly >> > > defined semantics >> > > >> > > - a mini-language to allow the efficent implementation of low-level >> > > algorithms >> > > >> > > The paper can be downloaded here: >> > > >> > > >> > > >> > > >> > > Sven-Olof Nystr?m >> > > >> > > _______________________________________________ >> > > erlang-questions mailing list >> > > erlang-questions@REDACTED >> > > http://www.erlang.org/mailman/listinfo/erlang-questions >> > > > From matthew.reilly@REDACTED Thu Jun 26 06:35:51 2008 From: matthew.reilly@REDACTED (Matthew Reilly) Date: Wed, 25 Jun 2008 21:35:51 -0700 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: References: <18529.883.586609.71953@hamberg.it.uu.se> Message-ID: <48631CA7.5090309@sipphone.com> Richard A. O'Keefe wrote: > > > I hope that we can rely on Erlang garbage collection to > collect any process suspended in a receive construct and not registered > and not referenced by any other process. If we can, the distinction is > only useful to people and not really needed by the computer. > This will doubtfully ever happen. Since erlang is distributed, a pid may be referenced via a process on a completely different node, so any garbage collection of pids would therefore need to be distributed. Adding on to this term_to_binary, mnesia, ... pids could be stored in different formats or even on disk. You also may wish to have pids that are unreferenced by any other, but still do important work e.g. loop() -> after 1000 -> % Touch the hardware watchdog to prevent the system from reboooting file:read_file("/dev/watchdog"), ?MODULE:loop() end From ok@REDACTED Thu Jun 26 07:14:22 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 26 Jun 2008 17:14:22 +1200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> Message-ID: <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> On 26 Jun 2008, at 9:57 am, Ulf Wiger wrote: > Haskell has channels, and so does .NET (mailbox objects). And Concurrent ML, amongst others, had them even earlier. > There is > therefore an opportunity to compare programs and try to determine > whether programming with channels makes for more or less > readable code than erlang's selective receive. It seems to me that programming with Erlang's receive is far simpler to do, to read, and to reason about than "channels" (strictly speaking, mailboxes). The reason is not far to seek: if channels are "objects", they can go ANYWHERE. There's another one. Although Nystrom presented two receive forms, the text made it clear that two more were required. But he had forgotten the need for something like BSD select()/System V poll()/Ada selective accept/ other languages' multiwaits. That means another two forms. nystrom_receive() % default channel nystrom_receive(Timeout) nystrom_receive_from(Channel) nystrom_receive_from(Channel, Timeout) nystrom_receive_from_any(Channel_Set) nystrom_receive_from_any(Channel_Set, Timeout) But it gets worse. if channels are "objects", they can go anywhere, BUT THEY DON'T GET THERE WITHOUT BEING CARRIED! You have to explictly pass them around, especially loops. So to handle the simple bounded buffer you would find yourself writing stuff like this: buffer(Status, Contents, GetChan, PutChan) -> Channels = case Status of full -> [GetChan] ; empty -> [PutChan] ; _ -> [GetChan,PutChan] end, case nystrom_receive_from_any(Channels) of {GetChan,Who} -> {Status1,Contents1,Msg} = pop(Contents), Who ! Msg ; {PutChan,Msg} -> {Status1,Contents1} = add(Contents, Msg) end, buffer(Status1, Contents1, GetChan, PutChan). That's assuming that multireceive returns a {Channel,Message} pair. Another approach would be to pass a list of {Channel, Handler} pairs, when the code would look like buffer(Status, Contents, GetChan, PutChan) -> GetHandler = fun (Who) -> {Status1,Contents1,Msg} = pop(Contents), Who ! Msg, buffer(Status1, Contents1, GetChan, PutChan) end, PutHandler = fun (Msg) -> {Status1,Contents1} = add(Contents, Msg), buffer(Status1, Contents1, GetChan, PutChan) end, nystrom_receive_from_any( case Status of full -> [{GetChan,GetHandler}] ; empty -> [{PutChan,PutHandler}] ; _ -> [{GetChan,GetHandler}, {PutChan,PutHandler}] end). How this is in any way simpler than a selective receive entirely escapes me. There's another problem with this. I can easily fire up a bounded buffer process in Erlang: Buffer = spawn (fun () -> buffer(empty, empty_buffer_contents()) end), because spawn/1 delivers the Pid *outside* the process. But new_channel/0 delivers its result *inside* the process. This isn't so in Concurrent ML, where mailboxes will talk to anybody. In CML, the parent process would create the mailboxes GetChan and PutChan: GetChan = Mailbox.mailbox(), PutChan = Mailbox.mailbox(), Buffer = spawn (fun () -> buffer(empty, empty_buffer_contents(), GetChan, PutChan) end), But in Nystrom's proposal, "only the creator of a channel may receive messages from it" (p 5). So the channels have to be created *inside* the new buffer process. How then do other processes get their "hands" on them? Ahah! It's all so EASY without selective receive! GetChanChan = new_channel(), PutChanChan = new_channel(), Buffer = spawn (fun () -> GetChan = new_channel(), PutChan = new_channel(), GetChanChan ! GetChan, PutChanChan ! PutChan, buffer(empty, empty_buffer_contents(), GetChan, PutChan) end), GetChan = ne_receive(GetChanChan), PutChan = ne_receive(PutChanChan), ... By the way, recall that the problem that this "simplification" is supposed to solve is this: "it is considered bad style to leave too many messages in the mailbox". Let me quote from the Concurrent ML documentation for the Mailbox structure: Note that mailbox buffers are unbounded, which means that there is no flow control to prevent a producer from greatly outstripping a consumer, and thus exhausting memory. If your mailbox is getting full, it's not another questionably simpler language construct you need. It's FLOW CONTROL, which is a higher level protocol issue. One thing I particularly like about 'receive' in Erlang is the fact that it is visually hard to miss. Things that look like function calls are much easier to lose sight of in the thick undergrowth of things that look like function calls. In fact it won't even do to just look for 'ne_receive' (or 'nystrom_receive*'). (It _is_ possible to hide 'receive', but it's a whole lot harder.) From dgud@REDACTED Thu Jun 26 07:23:00 2008 From: dgud@REDACTED (Dan Gudmundsson) Date: Thu, 26 Jun 2008 07:23:00 +0200 Subject: [erlang-questions] : ODBC library - link check failed on mac leopard In-Reply-To: References: <4eaa09eb0806242201o339c17e6y7ea74e50844c7c26@mail.gmail.com> <4eaa09eb0806250506t57131625yf0f24df4877900bf@mail.gmail.com> Message-ID: <486327B4.4010601@erix.ericsson.se> I have filed an issue on it, will see if it makes to R12B-4 Ingela is on vacation but I know she have other stuff on her agenda when she comes back :-) /Dan Geoff Cant wrote: > "Peter Lemenkov" writes: > >> 2008/6/25 wenew zhang : >>> error below:do you user mac now? >> Yes, I'm a mac user. >> >>> so i think macport doesn't add the odbc suports in erlang, >> Exactly. >> >> auriga:~ petro$ erl >> Erlang (BEAM) emulator version 5.6.2 [source] [async-threads:0] [hipe] >> [kernel-poll:false] >> >> Eshell V5.6.2 (abort with ^G) >> 1> odbc:start(). >> ** exception error: undefined function odbc:start/0 >> 2> > > This is a problem I tend to fixup in macports - the odbc application > needs to link (and link check) against libiodbc not libodbc. Libiodbc is > shipped as part of Mac OS X (Tiger and Leopard I'm sure of) and provides > the exact same API as libodbc. > > It'd be nice if there was a special case for darwin/macos in the > configure script that would linkcheck against iodbc. Something like: > > diff --git a/lib/odbc/configure.in b/lib/odbc/configure.in > index 114f86a..d1df895 100644 > --- a/lib/odbc/configure.in > +++ lib/odbc/configure.in > @@ -136,6 +136,16 @@ have_odbc_lib=no > > AC_SUBST(TARGET_FLAGS) > case $host_os in > + darwin*) > + TARGET_FLAGS="-DUNIX" > + AC_CHECK_LIB(pthread, pthread_create, > + [AC_DEFINE(HAVE_LIBPTHREAD, 1, [Define if you have the pthread library (-lpthread).]) > + LIBS="$LIBS -lpthread" > + have_pthread_lib=yes]) > + ODBC_LIB=-L"/usr/lib" > + ODBC_INCLUDE="-I/usr/lib/include" > + AC_CHECK_LIB(iodbc, SQLAllocHandle,[ODBC_LIB="$ODBC_LIB -liodbc" odbc_lib_link_sucess=yes]) > + ;; > win32|cygwin) > TARGET_FLAGS="-DWIN32" > AC_CHECK_LIB(ws2_32, main) > > This was prepared for R12B-2 and works : > > 1> application:start(odbc). > ok > > Available as a git patch at (http://git.erlang.geek.nz/?p=erlang-otp.git;a=commit;h=5d71d8f6c8a5743422c9d9104574d7fa4945a912). > > Maybe something like this could be rolled into R12B-4 ? > > Cheers, From ok@REDACTED Thu Jun 26 07:32:38 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 26 Jun 2008 17:32:38 +1200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <48631CA7.5090309@sipphone.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <48631CA7.5090309@sipphone.com> Message-ID: <615AF6CA-D6EB-45D4-8EB5-53C30356D4AE@cs.otago.ac.nz> > I wrote: > I hope that we can rely on Erlang garbage collection to >> >> collect any process suspended in a receive construct and not >> registered >> and not referenced by any other process. If we can, the >> distinction is >> only useful to people and not really needed by the computer. > Perhaps I should make it clear that I was not entirely serious in that hope. It *would* be nice, but it was really a stick to beat the 'channels' proposal with. Since channels can be used in the way that processes are now, and since a reference to a Nystrom channel amounts to a reference to its process, channels have much the same consequences for/problems with distributed GC as processes. On 26 Jun 2008, at 4:35 pm, Matthew Reilly wrote: > This will doubtfully ever happen. Since erlang is distributed, a pid > may be referenced via a process on a completely different node, so > any garbage collection of pids would therefore need to be distributed. Distributed GC is no longer rocket wizardry. (Well, it _is_, but the wizards have published.) > Adding on to this term_to_binary, mnesia, ... pids could be stored > in different formats or even on disk. The same, of course, applies to channels. So when DO channels go away? > > > You also may wish to have pids that are unreferenced by any other, > but still do important work e.g. > > loop() -> > after 1000 -> > % Touch the hardware watchdog to prevent the system from > reboooting > file:read_file("/dev/watchdog"), > ?MODULE:loop() > end Imprecise wording on my part; I didn't intend waiting for a timeout to count, only waiting for a message. Interestingly, if you have channels, and a process is waiting for a message on a channel, and no other process has a reference to that channel, the process is effectively dead, no matter how many other channels it has or how many other processes have references to them. From svenolof@REDACTED Thu Jun 26 09:32:53 2008 From: svenolof@REDACTED (Sven-Olof Nystr|m) Date: Thu, 26 Jun 2008 09:32:53 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> Message-ID: <18531.17957.436474.580910@hamberg.it.uu.se> Ulf Wiger writes: > 2008/6/25 Vlad Dumitrescu : > > > > I'm not sure I get the reason why channels would be useful. I see two > > possibilities: > > a) for performance, when each channel has a different mailbox > > b) to simplify the code/architecture > > I don't think channels would simplify the code, but they might make > it easier to reason about it. The thing that selective receive brings to > the table is scoped message reception. The Erlang approach of > pattern-matching on a single message queue is extremely flexible, > but it's difficult to "prove" the code correct. For example, it is quite > possible for a piece of code to consume someone else's 'DOWN' > message, by ignoring the Ref part. If channels followed the same > scoping rules as variables (and channels aren't just syntactic sugar > on top of a single message queue), a function could not steal someone > else's messages. Yes. This is actually one of the reasons I am interested in alternatives to selective receive. Any kind of formal reasoning about process communication would benefit from the channel concept. I didn't mention it in the paper since I was concerned that a discussion would be too complicated. Also, I thought that the reasons I gave were sufficient. As you say, the use of selective receive makes it harder to determine the set of messages that should (and shouldn't) arrive in the mailbox of a particular process. > Not that these kinds of bugs are commonplace in Erlang. It's quite > easy to avoid them using simple conventions. But there is, after all, > a difference between "easy to avoid" and "impossible", which can > sometimes be significant. No, it seems that the Erlang developers have learned how to live with selective receive. > Haskell has channels, and so does .NET (mailbox objects). There is > therefore an opportunity to compare programs and try to determine > whether programming with channels makes for more or less > readable code than erlang's selective receive. OK. Sven-Olof From mats.cronqvist@REDACTED Thu Jun 26 09:40:50 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Thu, 26 Jun 2008 09:40:50 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> Message-ID: <48634802.3050807@gmail.com> Richard A. O'Keefe wrote: > If your mailbox is getting full, it's not another questionably > simpler language construct you need. It's FLOW CONTROL, which > is a higher level protocol issue. > +1 From erlang@REDACTED Thu Jun 26 09:54:41 2008 From: erlang@REDACTED (Dominic Williams) Date: Thu, 26 Jun 2008 03:54:41 -0400 (EDT) Subject: [erlang-questions] erlang emacsmode problem In-Reply-To: References: <286156.61083.qm@web28303.mail.ukl.yahoo.com> Message-ID: Hi, > Double check your paths. Make sure they actually exist as written. If > you've been getting this error since the upgrade to R12B-3 the cause > of the error should be obvious. And if that was your problem, you could use some more flexible emacs-lisp such as this (no hard-coded tools-x.y.z version): (if (not (boundp 'erlang-root-dir)) (message "Skipping erlang-mode: erlang-root-dir not defined") (progn (set 'erlang-bin (concat erlang-root-dir "/bin/")) (set 'erlang-lib (concat erlang-root-dir "/lib/")) (if (not (boundp 'erlang-mode-path)) (set 'erlang-mode-path (concat erlang-lib (file-name-completion "tools-" erlang-lib) "emacs/erlang.el"))) (if (and (file-readable-p erlang-mode-path) (file-readable-p erlang-bin)) (progn (message "Setting up erlang-mode") (set 'exec-path (cons erlang-bin exec-path)) (set 'load-path (cons (concat erlang-lib (file-name-completion "tools-" erlang-lib) "emacs") load-path)) (set 'load-path (cons (file-name-directory erlang-mode-path) load-path)) (require 'erlang-start)) (message "Skipping erlang-mode: %s and/or %s not readable" erlang-bin erlang-mode-path)))) Regards, Dominic Williams http://dominicwilliams.net From tuncer.ayaz@REDACTED Thu Jun 26 10:16:34 2008 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Thu, 26 Jun 2008 10:16:34 +0200 Subject: [erlang-questions] erlang emacsmode problem In-Reply-To: References: <286156.61083.qm@web28303.mail.ukl.yahoo.com> Message-ID: <4ac8254d0806260116y48803017hf1cecfa320530fcd@mail.gmail.com> On Thu, Jun 26, 2008 at 9:54 AM, Dominic Williams wrote: > Hi, > >> Double check your paths. Make sure they actually exist as written. If >> you've been getting this error since the upgrade to R12B-3 the cause >> of the error should be obvious. > > And if that was your problem, you could use some more flexible emacs-lisp > such as this (no hard-coded tools-x.y.z version): I think it's obvious but in case someone is possibly missing this little bit of info the following or something more Emacs-idiomatic should be done before that: (if (or (eq system-type 'windows-nt) (eq system-type 'ms-dos)) (setq erlang-root-dir "C:/erlang") (setq erlang-root-dir "/usr/local/otp") ) > (if > (not (boundp 'erlang-root-dir)) > (message "Skipping erlang-mode: erlang-root-dir not defined") > (progn > (set 'erlang-bin (concat erlang-root-dir "/bin/")) > (set 'erlang-lib (concat erlang-root-dir "/lib/")) > (if > (not (boundp 'erlang-mode-path)) > (set 'erlang-mode-path > (concat > erlang-lib > (file-name-completion "tools-" erlang-lib) > "emacs/erlang.el"))) > (if > (and > (file-readable-p erlang-mode-path) > (file-readable-p erlang-bin)) > (progn > (message "Setting up erlang-mode") > (set 'exec-path (cons erlang-bin exec-path)) > (set 'load-path (cons > (concat > erlang-lib > (file-name-completion "tools-" erlang-lib) > "emacs") > load-path)) > (set 'load-path (cons (file-name-directory erlang-mode-path) load-path)) > (require 'erlang-start)) > (message "Skipping erlang-mode: %s and/or %s not readable" > erlang-bin erlang-mode-path)))) > > Regards, > > Dominic Williams > http://dominicwilliams.net > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From bjorn@REDACTED Thu Jun 26 10:21:52 2008 From: bjorn@REDACTED (Bjorn Gustavsson) Date: 26 Jun 2008 10:21:52 +0200 Subject: [erlang-questions] Parsing binaries performance In-Reply-To: <48627BAF.5030007@san.rr.com> References: <48626714.6030708@dehnes.com> <48627BAF.5030007@san.rr.com> Message-ID: Darren New writes: > Another possibility would be to try code that simply indexes into the > binary instead of breaking it apart into a new binary, as that might not > need to copy things around as much. Something like > > check_pos(Bin, Inx, Chr) -> > <<_:Inx/binary, MaybeCh:integer, _/binary>>, > Chr == MaybeCh. > > find_cr(Bin) -> find_cr(Bin, 0). > > find_cr(Bin, Inx) when Inx >= size(Bin) -> false; > find_cr(Bin, Inx) -> check_pos(Bin, Inx, $\r) > andalso check_pos(Bin, Inx+1, $\n). That kind of optimization was good before R12B, but in R12B it will generally be slower. See the Efficiency Guide: http://erlang.org/doc/efficiency_guide/part_frame.html Also, binary matching in R12B never copies any binaries. Before R12B, unaligned binaries would be copied, but never binaries starting on a byte boundary. /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From per.gustafsson@REDACTED Thu Jun 26 10:48:18 2008 From: per.gustafsson@REDACTED (Per Gustafsson) Date: Thu, 26 Jun 2008 10:48:18 +0200 Subject: [erlang-questions] Parsing binaries performance In-Reply-To: <48626714.6030708@dehnes.com> References: <48626714.6030708@dehnes.com> Message-ID: <486357D2.50802@it.uu.se> I would suggest that you update your system to R12B-3. On my computer your program is roughly 6 times faster on R12B-3 than on R11B-5 this is due to a specific optimization for this kind of code. It will still be slower than the Java code, but if you use native compilation it should approach the same speed. Per Sebastian Dehne wrote: > Hello list, > > I'm trying to write a parser in Erlang for a byte-stream (which I > receive from the TCP socket), but I realise that my code is slow > compared to the java version which I have. I've attached both version. > > The java prints version: > Done: 93 > > The erlang version prints: > $ erl > Erlang (BEAM) emulator version 5.5.5 [source] [async-threads:0] > [kernel-poll:false] > > Eshell V5.5.5 (abort with ^G) > 1> c(test). > {ok,test} > 2> Data1 = <<"Somestringcontainingcrlf\r\nandsomemoretextbehind\r\n">>. > <<"Somestringcontainingcrlf\r\nandsomemoretextbehind\r\n">> > 3> > 3> test:testwrapper(Data1, 500000). > Erlang Elapsed 1570 (runtime) 1586 (wall clock) milliseconds > ok > > Both tests are run on the same machine. I also run the test several > times. Quite a difference between 93ms and 1586ms. > > What am I doing wrong? How should I write the code in Erlang such that > it matches the performance of Java? > > > Thanks > > Sebastian > ------------------------------------------------------------------------ > > import java.io.File; > import java.io.FileInputStream; > import java.io.IOException; > import java.util.ArrayList; > import java.util.LinkedList; > import java.util.List; > > public class Test { > public static void main(String[] args) throws Exception { > test2(); > } > > public static void test2() throws Exception { > final byte[] data = "Somestringcontainingcrlf\r\nandsomemoretextbehind\r\n".getBytes(); > > long start = System.currentTimeMillis(); > for (int i = 0; i < 500000; i++) { > if (findPos(data) == -1) { > throw new Exception("Error"); > } > } > System.out.println("Done: " + (System.currentTimeMillis() - start)); > } > > public static int findPos(byte[] data) { > boolean cr_found = false; > int pos = 0; > for (byte b : data) { > ++pos; > if (b == '\r') { > cr_found = true; > } else if (cr_found && b == '\n') { > return pos; > } else { > cr_found = false; > } > } > return -1; > } > > } > > > ------------------------------------------------------------------------ > > -module(test). > -compile(export_all). > > testwrapper(Data, I) -> > statistics(runtime), > statistics(wall_clock), > test(Data, I), > {_, Time1} = statistics(runtime), > {_, Time2} = statistics(wall_clock), > io:format("Erlang Elapsed ~p (runtime) ~p (wall clock) milliseconds~n", [Time1, Time2]). > > test(_Data, 0) -> > done; > test(Data, I) -> > {true, _Pos} = find_crlf_or_end(Data, 0), > test(Data, I - 1). > > % Find the position of the first '\r\n' > find_crlf_or_end(<<$\r, $\n, _/binary>>, Pos) -> > {true, Pos}; > find_crlf_or_end(<<_, Rest/binary>>, Pos) -> > find_crlf_or_end(Rest, Pos + 1); > find_crlf_or_end(<<>>, Pos) -> > {true, Pos}. > > > > ------------------------------------------------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From jongretar@REDACTED Thu Jun 26 12:05:46 2008 From: jongretar@REDACTED (Jon Gretar Borgthorsson) Date: Thu, 26 Jun 2008 10:05:46 +0000 Subject: [erlang-questions] Asterisk manager parser question Message-ID: <4ecde87b0806260305t7786b523xae50ec396aa8b58c@mail.gmail.com> Hi.. I've been playing around with erlang for a month and playing around with gen_tcp and more. Now I've decided to redo a script that I have that parses events from the asterisk management api. The old script tended to be a bit unstable and I think that Erlang would be perfect for the job. However I'm having a little problem solving how to read the information in an elegant way. Basically for those who don't know the Asterisk manager API then it's basically a tcp port you connect to and read events as they come. Each event is formatted like this: Event: Agentlogoff Agent: Logintime: Uniqueid: And events are seperated by 2 returns. What I was thinking of doing is something that listens to the port and sends events clause in whole to another process that analyzes it. What I'm having problems with is just the first part. Recognizing the event clouses in some elegant way and sending them on to the parsing function. I have to try to save the message up in a buffer and when it senses 2 returns then creates a process to parse it and clears the buffer to read the next event. I'm sure that there is some elegant way to do this but since this is my first real erlang project then I'm a bit lost on how the best way is. Almost everything else I have figured out and it's just this last problem that needs solving. Anyone have a few free minutes to assist me on this? Regards - Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: From bqt@REDACTED Thu Jun 26 13:38:48 2008 From: bqt@REDACTED (Johnny Billquist) Date: Thu, 26 Jun 2008 13:38:48 +0200 Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <200806252024.m5PKOmW2059592@pluto.hedeland.org> References: <200806252024.m5PKOmW2059592@pluto.hedeland.org> Message-ID: <48637FC8.1070503@softjar.se> Per Hedeland wrote: > Johnny Billquist wrote: >> Per Hedeland skrev: >>> Whether this is Edwin's problem I don't know - the "fixed packet rate" >>> observation may actually be more or less correct: As you explained, TCP >>> doesn't ack packets, it acks bytes - but the actual *sending* of acks is >>> definitely related to the reception of packets (or "segments" if you >>> prefer), in particular in a one-way data transfer where there are no >>> outgoing data packets that can have acks "piggy-backed". The details may >>> vary, but in general in such a case an ack is sent for every other >>> packet received, or after a ("long" - 200 ms) timeout if no packets are >>> received. >> Hmm. Well, a tcpdump will quickly tell if the window size is zero, and he's >> hitting the RTT-limiting factor. > > Hmm hmm, I think I wasn't thinking straight when I wrote the above - I > was speculating that this could be something other than the RTT-limiting > factor (otherwise small vs big packets shouldn't make a difference in > throughput), along the lines of "the receiver has data to ack but is > delaying the sending of the ack until he recieves another segment". But > this can't be the explanation I believe - if you're sending smaller > packets, you can send that many more into a given window before you need > to wait for an ack, so there are more "opprtunities" for the reciever to > send a delayed ack. Yes. However, one thing to also keep an eye out for is if a lot of small packets are generated, and the received can't accept packets at that pace. Then you'll experience lost packets, followed by retransmits. And this is definitely related to the RTT. This is especially a problem if you have a transmitter sitting with an ethernet interface running at a higher speed than the receiver ethernet. Not totally unheard of if you have a switch in between... But this situation will be very visible in the tcp counters on the system. Do the erlang system turn off the nagle algorithm, or force TCP to push packets? Otherwise the underlying tcp layer should not be sending small packets at a high pace. Johnny From per@REDACTED Thu Jun 26 14:42:09 2008 From: per@REDACTED (Per Hedeland) Date: Thu, 26 Jun 2008 14:42:09 +0200 (CEST) Subject: [erlang-questions] why is gen_tcp:send slow? In-Reply-To: <48637FC8.1070503@softjar.se> Message-ID: <200806261242.m5QCg92U079781@pluto.hedeland.org> Johnny Billquist wrote: > >Do the erlang system turn off the nagle algorithm, or force TCP to push >packets? No - i.e. gen_tcp/inet_drv doesn't (you can turn off Nagle with the 'nodelay' option). > Otherwise the underlying tcp layer should not be sending small >packets at a high pace. True. This wasn't a "high pace" though (3-4 packets/s), but of course one has to wonder why this was happening in the first place - possibly the "messages" to be sent were just "naturally" spaced far enough apart to defeat Nagle. --Per From sebastian@REDACTED Thu Jun 26 15:07:19 2008 From: sebastian@REDACTED (Sebastian Dehne) Date: Thu, 26 Jun 2008 15:07:19 +0200 Subject: [erlang-questions] Parsing binaries performance In-Reply-To: <486357D2.50802@it.uu.se> References: <48626714.6030708@dehnes.com> <486357D2.50802@it.uu.se> Message-ID: <48639487.60802@dehnes.com> Thanks for all the comments and explanations. Results: without changing my suggested code (posted in the first message in the thread), I upgraded to the latest version which did the test within 250ms. After enabling +native, I got even 90ms. This doesn't match Java's 21ms, but that's fine with me. (I like Erlang for all the other goodies!) 90ms is much batter than 1500ms which I got first. Thanks! Sebastian On 06/26/2008 10:48 AM, Per Gustafsson wrote: > I would suggest that you update your system to R12B-3. On my computer > your program is roughly 6 times faster on R12B-3 than on R11B-5 this is > due to a specific optimization for this kind of code. It will still be > slower than the Java code, but if you use native compilation it should > approach the same speed. > > Per > > > Sebastian Dehne wrote: >> Hello list, >> >> I'm trying to write a parser in Erlang for a byte-stream (which I >> receive from the TCP socket), but I realise that my code is slow >> compared to the java version which I have. I've attached both version. >> >> The java prints version: >> Done: 93 >> >> The erlang version prints: >> $ erl >> Erlang (BEAM) emulator version 5.5.5 [source] [async-threads:0] >> [kernel-poll:false] >> >> Eshell V5.5.5 (abort with ^G) >> 1> c(test). >> {ok,test} >> 2> Data1 = <<"Somestringcontainingcrlf\r\nandsomemoretextbehind\r\n">>. >> <<"Somestringcontainingcrlf\r\nandsomemoretextbehind\r\n">> >> 3> >> 3> test:testwrapper(Data1, 500000). >> Erlang Elapsed 1570 (runtime) 1586 (wall clock) milliseconds >> ok >> >> Both tests are run on the same machine. I also run the test several >> times. Quite a difference between 93ms and 1586ms. >> >> What am I doing wrong? How should I write the code in Erlang such that >> it matches the performance of Java? >> >> >> Thanks >> >> Sebastian >> ------------------------------------------------------------------------ >> >> import java.io.File; >> import java.io.FileInputStream; >> import java.io.IOException; >> import java.util.ArrayList; >> import java.util.LinkedList; >> import java.util.List; >> >> public class Test { >> public static void main(String[] args) throws Exception { >> test2(); >> } >> >> public static void test2() throws Exception { >> final byte[] data = >> "Somestringcontainingcrlf\r\nandsomemoretextbehind\r\n".getBytes(); >> >> long start = System.currentTimeMillis(); >> for (int i = 0; i < 500000; i++) { >> if (findPos(data) == -1) { >> throw new Exception("Error"); >> } >> } >> System.out.println("Done: " + (System.currentTimeMillis() - >> start)); >> } >> >> public static int findPos(byte[] data) { >> boolean cr_found = false; >> int pos = 0; >> for (byte b : data) { >> ++pos; >> if (b == '\r') { >> cr_found = true; >> } else if (cr_found && b == '\n') { >> return pos; >> } else { >> cr_found = false; >> } >> } >> return -1; >> } >> >> } >> >> >> ------------------------------------------------------------------------ >> >> -module(test). >> -compile(export_all). >> >> testwrapper(Data, I) -> >> statistics(runtime), >> statistics(wall_clock), >> test(Data, I), >> {_, Time1} = statistics(runtime), >> {_, Time2} = statistics(wall_clock), >> io:format("Erlang Elapsed ~p (runtime) ~p (wall clock) >> milliseconds~n", [Time1, Time2]). >> >> test(_Data, 0) -> >> done; >> test(Data, I) -> >> {true, _Pos} = find_crlf_or_end(Data, 0), >> test(Data, I - 1). >> >> % Find the position of the first '\r\n' >> find_crlf_or_end(<<$\r, $\n, _/binary>>, Pos) -> >> {true, Pos}; >> find_crlf_or_end(<<_, Rest/binary>>, Pos) -> >> find_crlf_or_end(Rest, Pos + 1); >> find_crlf_or_end(<<>>, Pos) -> >> {true, Pos}. >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions > From mj@REDACTED Thu Jun 26 16:09:44 2008 From: mj@REDACTED (Mikkel Jensen) Date: Thu, 26 Jun 2008 16:09:44 +0200 Subject: [erlang-questions] Bug in xmerl Message-ID: It seems there is a bug in xmerl when loading elements that contain numeric character references followed by UTF-8 characters. Example: ? newline ? 1> element(1, xmerl_scan:string("\303\251 \303\251", [{encoding, 'utf-8'}])). {xmlElement,a,a,[], {xmlNamespace,[],[]}, [],1,[], [{xmlText,[{a,1}],1,[],"\303\251",text}, {xmlText,[{a,1}],2,[],[10,195,131,194,169],text}], [],"/",undeclared} Xmerl splits the parsed value around the newline character (strange but ok). However, the first part is encoded correctly while the second part is garbled! It's worth noticing that attribute values are encoded correctly: 2> element(1, xmerl_scan:string("", [{encoding, 'utf-8'}])). {xmlElement,a,a,[], {xmlNamespace,[],[]}, [],1, [{xmlAttribute,b,[],[],[],[],1,[],"\303\251 \303\251",false}], [],[],"/",undeclared} Can someone confirm if this is a bug? - Mikkel -------------- next part -------------- An HTML attachment was scrubbed... URL: From litaocheng@REDACTED Thu Jun 26 16:28:24 2008 From: litaocheng@REDACTED (litao cheng) Date: Thu, 26 Jun 2008 22:28:24 +0800 Subject: [erlang-questions] surprise! binary vs list in file parse. Message-ID: I had read joel's article:Parsing text and binary files with Erlang. In the article, the author show how to parse a comma-delimited text file. I write the code for practice, the module source is act.erl. By the way, I see in the bottom of joel' article, a buddy give a comment, he says, he use binary to instead of some list, It seems so efficient, so I write the second code, it's act_2.erl. Finally, I want to test how much the act_2 faster than act. So suprise, In my Compute, The result is : 96> timer_avg:tc(act, parse, ["test.txt"], 100). Max: 15991 Min: 1 Avg: 2339.05 100> timer_avg:tc(act_2, parse, ["test.txt"], 100). Max: 15997 Min: 1 Avg: 4839.03 the timer_avg is a moudule evaluates apply(Module, Function, Arguments) N times and measures the elapsed real time, about Max, Min, Avg. (use timer:tc/3). who will give me some explain? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: act.erl Type: application/octet-stream Size: 914 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: act_2.erl Type: application/octet-stream Size: 798 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: timer_avg.erl Type: application/octet-stream Size: 489 bytes Desc: not available URL: From vladdu55@REDACTED Thu Jun 26 16:39:38 2008 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Thu, 26 Jun 2008 16:39:38 +0200 Subject: [erlang-questions] surprise! binary vs list in file parse. In-Reply-To: References: Message-ID: <95be1d3b0806260739n3deb634l5e133d8683bf7502@mail.gmail.com> Hi 2008/6/26 litao cheng : > I had read joel's article:Parsing text and binary files with Erlang. In the > article, the author show how to parse a comma-delimited text file. > I write the code for practice, the module source is act.erl. By the way, I > see in the bottom of joel' article, a buddy give a comment, he says, he use > binary to instead of some list, It seems so efficient, so I write the second > code, it's act_2.erl. > > Finally, I want to test how much the act_2 faster than act. So suprise, In > my Compute, The result is : > 96> timer_avg:tc(act, parse, ["test.txt"], 100). > Max: 15991 > Min: 1 > Avg: 2339.05 > > 100> timer_avg:tc(act_2, parse, ["test.txt"], 100). > Max: 15997 > Min: 1 > Avg: 4839.03 > > the timer_avg is a moudule evaluates apply(Module, Function, Arguments) N > times and measures the elapsed real time, about Max, Min, Avg. (use > timer:tc/3). > > who will give me some explain? > You might want to retry the tests, starting them instead with spawn(timer_avg, tc, [act, parse, ["test.txt"], 100]). spawn(timer_avg, tc, [act_2, parse, ["test.txt"], 100]). This way you don't get any random garbage collections to interfere. regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu Jun 26 17:02:46 2008 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 26 Jun 2008 17:02:46 +0200 Subject: [erlang-questions] surprise! binary vs list in file parse. In-Reply-To: References: Message-ID: <56a0a2840806260802ne5b3e94yd040906409aa16f2@mail.gmail.com> 2008/6/26 litao cheng : > I had read joel's article:Parsing text and binary files with Erlang. In the > article, the author show how to parse a comma-delimited text file. > I write the code for practice, the module source is act.erl. By the way, I > see in the bottom of joel' article, a buddy give a comment, he says, he use > binary to instead of some list, It seems so efficient, so I write the second > code, it's act_2.erl. In addition to Vlads suggestion, you might want to know that there are considerable improvements to binaries in R12B compared to R11B. Many "stock erlang" systems in Linux distributions is still R11B. If I read http://www.erlang.org/doc/efficiency_guide/binaryhandling.html#4 correctly, you end up copying your binary at each append in R11B. My guess is that you can keep the fields in Heap binaries in R12B since they tend to be small and you always keep a linear access to the binary reference. On the other hand, I am not the internals expert on erlangs binary handling. From litaocheng@REDACTED Thu Jun 26 17:10:43 2008 From: litaocheng@REDACTED (litao cheng) Date: Thu, 26 Jun 2008 23:10:43 +0800 Subject: [erlang-questions] surprise! binary vs list in file parse. In-Reply-To: <95be1d3b0806260739n3deb634l5e133d8683bf7502@mail.gmail.com> References: <95be1d3b0806260739n3deb634l5e133d8683bf7502@mail.gmail.com> Message-ID: yes, I retry the test like you say. the result is same. thank you! 2008/6/26 Vlad Dumitrescu : > Hi > > 2008/6/26 litao cheng : > > I had read joel's article:Parsing text and binary files with Erlang. In the >> article, the author show how to parse a comma-delimited text file. >> I write the code for practice, the module source is act.erl. By the way, >> I see in the bottom of joel' article, a buddy give a comment, he says, he >> use binary to instead of some list, It seems so efficient, so I write the >> second code, it's act_2.erl. >> >> Finally, I want to test how much the act_2 faster than act. So suprise, >> In my Compute, The result is : >> 96> timer_avg:tc(act, parse, ["test.txt"], 100). >> Max: 15991 >> Min: 1 >> Avg: 2339.05 >> >> 100> timer_avg:tc(act_2, parse, ["test.txt"], 100). >> Max: 15997 >> Min: 1 >> Avg: 4839.03 >> >> the timer_avg is a moudule evaluates apply(Module, Function, Arguments)N times and measures the elapsed real time, about Max, Min, Avg. (use >> timer:tc/3). >> >> who will give me some explain? >> > > You might want to retry the tests, starting them instead with > spawn(timer_avg, tc, [act, parse, ["test.txt"], 100]). > spawn(timer_avg, tc, [act_2, parse, ["test.txt"], 100]). > > This way you don't get any random garbage collections to interfere. > > regards, > Vlad > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dnew@REDACTED Thu Jun 26 17:25:40 2008 From: dnew@REDACTED (Darren New) Date: Thu, 26 Jun 2008 08:25:40 -0700 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> Message-ID: <4863B4F4.8090903@san.rr.com> Richard A. O'Keefe wrote: > On 26 Jun 2008, at 9:57 am, Ulf Wiger wrote: >> Haskell has channels, and so does .NET (mailbox objects). And Hermes, whose channels are closer to what I think is being proposed (mailboxes with typed messages on them that you can receive from only if you're holding the mailbox), and whose purpose is exactly the same as that of Erlang's. > if channels are "objects", they can go ANYWHERE. That was one of the problems with Hermes implementation. The implementors found it difficult to get right what happens when you pass the receiving end of a channel back and forth between processes while many others are sending to it. > But in Nystrom's proposal, "only the creator of a channel may > receive messages from it" (p 5). So the channels have to be > created *inside* the new buffer process. How then do other > processes get their "hands" on them? And in Hermes, this was a *good* thing. It kept people you didn't want sending you stuff from sending you stuff. You could give stdin and stdout channels to an untrusted process without giving them access to the "shut down the I/O subsystem" channel. -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From dnew@REDACTED Thu Jun 26 17:36:23 2008 From: dnew@REDACTED (Darren New) Date: Thu, 26 Jun 2008 08:36:23 -0700 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: References: <18529.883.586609.71953@hamberg.it.uu.se> Message-ID: <4863B777.908@san.rr.com> Richard A. O'Keefe wrote: > I hope that we can rely on Erlang garbage collection to > collect any process suspended in a receive construct and not registered > and not referenced by any other process. But we really can't, can we? That would imply that a GC happening on one node in California could send a message to Philadelphia to let that node know there's one less reference to the process ID. It would also imply that term_to_binary serializing a process ID and then dropping the variable holding the actual PID would allow the process to exit. So process A wrapping a PID of process X in a binary, dropping it into processes B's mailbox, then exiting, might cause process X to exit before process B read the binary out of its mailbox and turned it back into a term. I don't think the semantics of Erlang are such that anyone expects processes to exit just because nobody can send them a message. What would be the exit reason? Could you put catch around the receive and keep running? (That *is* how Hermes worked: to kill a runaway process, you just snipped off its output channels by having the processes listening exit, and the process could no longer affect its environment. But Hermes had a completely different strategy from Erlang for handling failures.) -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From christoph_schueler@REDACTED Thu Jun 26 07:20:43 2008 From: christoph_schueler@REDACTED (=?ISO-8859-1?Q?Christoph_Sch=FCler?=) Date: Thu, 26 Jun 2008 07:20:43 +0200 Subject: [erlang-questions] Beginner: Windows UDP multicast receive In-Reply-To: <000c01c8d61d$01922610$04b67230$@gregory@infiniumgroup.com> References: <000c01c8d61d$01922610$04b67230$@gregory@infiniumgroup.com> Message-ID: <4863272B.9080307@web.de> Scott Gregory schrieb: > Help?! > > Scott > > > {add_membership,{{224,0,66,66},{192,168,6,49}}}]), > loop(Socket). > > > Your Code is missing a Controlling-Process, you have to use 'gen_udp:controlling_process/2' to get this work. Also you should not use 'add_membership' in 'gen_udp:open', because you won't get any IGMP-Messages. Try this: %%% CODE-START -module(udp_mcast). -export([open/2,start/2]). -export([stop/1,receiver/0]). open(Ip,Port) -> Opts=[{reuseaddr,true}, {multicast_loop,false}, {multicast_if,{0,0,0,0}}, {multicast_ttl,4}], {ok,S}=gen_udp:open(Port,Opts), inet:setopts(S,[{add_membership,{Ip,{0,0,0,0}}}]), S. close(S) -> gen_udp:close(S). start(Ip,Port) -> S=open(Ip,Port), Pid=spawn(?MODULE,receiver,[]), gen_udp:controlling_process(S,Pid), {S,Pid}. stop({S,Pid}) -> close(S), Pid ! stop. receiver() -> receive {udp, Socket, IP, InPortNo, Packet} -> io:format("~n~nFrom: ~p~nIP: ~p~nData: ~p~n",[IP,InPortNo,Packet]), receiver(); stop -> true; AnythingElse -> io:format("RECEIVED: ~p~n",[AnythingElse]), receiver() end. %%% CODE-END If there are any UPNP-Devices in your Network (e.g. DSL-Router), you may use the Module as following: --- 1> {S,Pid}=udp_mcast:start({239,255,255,250},1900). {#Port<0.142>,<0.51.0>} From: {192,168,0,1} IP: 4747 Data: "NOTIFY * HTTP/1.1\r\nHOST: 239.255.255.250:1900\r\nCACHE-CONTROL: max-age=1800\r\nLOCATION: http://192.168.0.1:80/desc.xml\r\nNT: upnp:rootdevice\r\nNTS: ssdp:alive\r\nSERVER: IGD-HTTP/1.1 UPnP/1.0 UPnP-Device-Host/1.0\r\nUSN: uuid:00-13-46-52-83-7F-0100A8C00::upnp:rootdevice\r\n\r\n" . . . --- Note: Under Windows, disable the SSDP-Service and have some Patience... hope this helps! Chris From tsuraan@REDACTED Thu Jun 26 18:05:25 2008 From: tsuraan@REDACTED (tsuraan) Date: Thu, 26 Jun 2008 11:05:25 -0500 Subject: [erlang-questions] When is code_change triggered? Message-ID: <84fb38e30806260905qb6426f2l26e63888a294d937@mail.gmail.com> When I started playing with gen_server a few years ago, I recalled that doing a c(module) in the REPL would immediately update the code of my running servers. Now, I cannot get that to happen, so I'm trying to figure out what I'm doing wrong. My test server is minimal: -module(test). -export([start_link/0, start/0]). -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]). start_link() -> gen_server:start_link({ local, ?MODULE }, ?MODULE, [], []). start() -> gen_server:start({ local, ?MODULE }, ?MODULE, [], []). init([]) -> process_flag(trap_exit, true), { ok, {}}. handle_call(Msg, From, State) -> io:format("Unexpected Call ~p From ~p~n", [Msg, From]), { reply, { error, badcall }, State }. handle_cast(Msg, State) -> io:format("Unexpected Cast ~p~n", [Msg]), { noreply, State }. handle_info(Msg, State) -> io:format("Unexpected Info ~p~n", [Msg]), { noreply, State }. terminate(Reason, _State) -> io:format("Terminating ~p for reason ~p~n", [ self(), Reason ]), ok. code_change(OldVsn, State, Extra) -> io:format("Upgrading from ~p with extra data ~p~n", [OldVsn, Extra]), { ok, State }. I can run this with test:start(), but subsequent c(test) never print the message "Upgrading from ...". Can anyone tell what I'm doing wrong, or is my memory just bad and c(module) has never caused running instances of that module to be upgraded? From christoph_schueler@REDACTED Thu Jun 26 09:20:11 2008 From: christoph_schueler@REDACTED (=?ISO-8859-15?Q?Christoph_Sch=FCler?=) Date: Thu, 26 Jun 2008 09:20:11 +0200 Subject: [erlang-questions] Beginner: Windows UDP multicast receive Message-ID: <4863432B.3010705@web.de> Scott Gregory schrieb: > Help?! > > Scott > > > {add_membership,{{224,0,66,66},{192,168,6,49}}}]), > loop(Socket). > > > Your Code is missing a Controlling-Process, you have to use 'gen_udp:controlling_process/2' to get this work. Also you should not use 'add_membership' in 'gen_udp:open', because you won't get any IGMP-Messages. Try this: %%% CODE-START -module(udp_mcast). -export([open/2,start/2]). -export([stop/1,receiver/0]). open(Ip,Port) -> Opts=[{reuseaddr,true}, {multicast_loop,false}, {multicast_if,{0,0,0,0}}, {multicast_ttl,4}], {ok,S}=gen_udp:open(Port,Opts), inet:setopts(S,[{add_membership,{Ip,{0,0,0,0}}}]), S. close(S) -> gen_udp:close(S). start(Ip,Port) -> S=open(Ip,Port), Pid=spawn(?MODULE,receiver,[]), gen_udp:controlling_process(S,Pid), {S,Pid}. stop({S,Pid}) -> close(S), Pid ! stop. receiver() -> receive {udp, Socket, IP, InPortNo, Packet} -> io:format("~n~nFrom: ~p~nPort: ~p~nData: ~p~n",[IP,InPortNo,Packet]), receiver(); stop -> true; AnythingElse -> io:format("RECEIVED: ~p~n",[AnythingElse]), receiver() end. %%% CODE-END If there are any UPNP-Devices in your Network (e.g. DSL-Router), you may use the Module as following: --- 1> {S,Pid}=udp_mcast:start({239,255,255,250},1900). {#Port<0.142>,<0.51.0>} From: {192,168,0,1} IP: 4747 Data: "NOTIFY * HTTP/1.1\r\nHOST: 239.255.255.250:1900\r\nCACHE-CONTROL: max-age=1800\r\nLOCATION: http://192.168.0.1:80/desc.xml\r\nNT: upnp:rootdevice\r\nNTS: ssdp:alive\r\nSERVER: IGD-HTTP/1.1 UPnP/1.0 UPnP-Device-Host/1.0\r\nUSN: uuid:00-13-46-52-83-7F-0100A8C00::upnp:rootdevice\r\n\r\n" . . . --- Note: Under Windows, disable the SSDP-Service and have some patience... hope this helps! Chris From svenolof@REDACTED Thu Jun 26 18:26:00 2008 From: svenolof@REDACTED (Sven-Olof Nystr|m) Date: Thu, 26 Jun 2008 18:26:00 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: References: <18529.883.586609.71953@hamberg.it.uu.se> Message-ID: <18531.49944.89467.105800@harpo.it.uu.se> Richard A. O'Keefe writes: > I spent last night reading "Ideas for a new Erlang". It's not clear > whether it would be better to write all the responses in one message > or to split them into several messages, each with its own Subject:. > Let's keep it as one to start with. OK, I'm very interested in hearing your comments. > > SELECTIVE RECEIVE VS MAILBOX OBJECTS > ==================================== > > p1. "Consider, for example, selective receive (no other programming > language requires anything similar)," > > Untrue. Ada and Occam should not just spring to mind but jump up > and down screaming loudly "pick me, pick ME!" [quoted material deleted] > The quoted material is from the Ada'05 (actually ISO something:2007) > reference manual, however the construct was already present in Ada 81. Aren't we talking about a different type of machinery here? If I'm not mistaken, Ada uses synchronous communication which means that the sender of a message will be suspended until the receiver takes it. Without receive, there would be way to examine a message without releasing the sender. I assume that this is the main motivation for Ada's selective accept. Suppose we wanted to translate some Erlang application to Ada. One would probably implement the mailboxes as a separate data structure. Erlang's selective receive would be replaced by code that scanned this data structure. You would not translate code that used Erlang's selective recive to code that uses Ada's. On the other hand, I agree that there is a strong superficial similarity between Ada's receive and Erlang's. It is quite likely that Erlang's designers were influenced by it. (They seem pretty unclear about this, see for example a thread in this mailing list: ) > The placement of the guard *before* the accept tells you something > relevant. In Ada (and Occam), the guards serve only to selectively > enable or disable which messages will be looked for *before* they are > considered for accepting, not to select or postpone messages based on > their content after they arrive as in Erlang. However, nothing in the > Erlang language requires the guards in a 'receive' to depend on the > message contents, so the Erlang selective receive can be used to do the > Ada (and Occam) thing. > > The classic example is a bounded buffer. > > buffer(Status, Contents) -> > receive > {put,Msg} when Status /= full -> > {Status1, Contents1} = add(Contents, Msg) > ; {get,Who} when Status /= empty -> > {Status1, Contents1, Msg} = pop(Contents), > Who ! Msg > end, > buffer(Status1, Contents1). > > Without selective receive (or in Ada, selective accept), this would > require three receives: one for the full case, one for the empty case, > and one for the in-between case. In general, this lets us replace up > to 2**n-1 receives with a single receive having n guarded alternatives. This is a rather interesting example. I must admit that I have not seen anything like it in Erlang. On the other hand, I have trouble understanding the underlying motivation. When the buffer is full, incoming 'put' messages will be kept in the mailbox instead of stored in the buffer, so they will still require approximately the same amount of memory on the heap. For an outside observer, there will be no difference between this buffer and one that uses an unbounded data structure, execpt that it is easier to overflow the mailbox of the bounded buffer. Also, as the 'put' messages start to accumulate in the mailbox, finding a matching 'get' message will become more costly. > p3. "It is strange that Erlang has a feature that is on one hand so > general and powerful, but yet programmers are not supposed to > take advantage of it!" > > It would be strange, IF it were true. But it isn't. Erlang programmers > ARE supposed to take advantage of selective receive. Just as selective > accept is indispensable in Ada and Occam, it is indispensable in Erlang. > There is no other way, in any of these three languages, to wait until > one of N things happens. Actually, there is no way in Erlang to even receive a message without using selective receive. > > Note that "simplifying" Erlang receives would do nothing whatever to > address the problem of mailbox overflow. Supposing Ch to be a channel, > L = lists:seq(1,1000), > lists:foreach(fun (X) -> > lists:foreach(fun (Y) -> > lists:foreach(fun (Z) -> > Ch ! ok > end, L) > end, L) > end, L) > is going to be trouble even if there is only unconditional-receive-from- > channel and no selective receive at all. Sure. There are many ways to run out of memory. > > It would be absurd to cripple the language in order to avoid a problem > and then find the problem snickering on your shoulder as lively and as > malevolent as ever. As I say above, I don't find your bounded buffer example very convincing. Do you have other examples? > p3. "To the best of my knowledge, selective receive has only two uses:" > > The bounded buffer example above shows that this is not so. > > p4. Channels as first class objects. > > But they already are. There are three proposed functions and an > operator. > We'll represent a channel as a process. The only even slightly tricky > thing is that we need to distinguish between other messages sent to the > current process and replies sent from a channel. We'll use the rather > subtle fact that a reply can only arrive when we've asked for one, and > that when we ask for one, we'll wait until we get it, so there never > will > be one in the mailbox at other times. > [simple implementation snipped] > > So anyone who wants to program in this style already may. In fact it > is useful to distinguish between a single-shot channel and a multi-shot > channel. I have thought about this. In many cases, one would only send a single message on a channel. Would there be anything to gain from having a separate type of channel for this case? Similar ideas have been mentioned elsewhere, for example the logic variables of concurrent logic languages and I-structures of data flow languages. > I hope that we can rely on Erlang garbage collection to > collect any process suspended in a receive construct and not registered > and not referenced by any other process. If we can, the distinction is > only useful to people and not really needed by the computer. Not sure that I follow your reasoning. Garbage collecting processes is a tricky issue. As you say, channels may help in certain cases. > The thing that strikes me forcibly is that I have seen > all this before. John Reppy's Concurrent ML. It's > supported in SML/NJ and Mlton and perhaps other ML > implementations. What Nystrom calls a "channel" is > what CML calls a "mailbox". (What CML calls a "channel" > is what Occam calls a "channel".) And comparing the CML > code I've seen to Erlang makes Erlang look stunningly > simple *in use*. True, many languages have a channel concept. In my opinion, one great advantage of Erlang over other languages that use channels is that in Erlang, each process has a "standard" channel that it normally takes input from. I hope that I was not unclear about this, but that is a feature of Erlang that I had not intended to change. The problem with using the single channel for all types of communication is that the mailbox ends up containing messages that are not related to the service provided by the process. In your bounded buffer example, one would normally only expect to see 'put' and 'get' messages in the mailbox. Any other messages would indicate a bug somewhere. However, if one inserted some calls to io:format() in the code, those calls would cause other messages to be sent to the process. > > > GUARDS > ====== > > > p6. "The complex selective receive primitive of old Erlang > has forced the Erlang designers to introduce a more > restricted type of expressions called _guards_." > > This is fiction, and it's fiction with a major error in it. That is terrible. > Guards are *NOT* a restricted type of expression. I meant expression in a general sense. I know that Erlang expressions and Erlang guards form two different classes of expressions. > This was much clearer in old Erlang. > Sadly, it has been obscured by the disastrous design bungle > of giving guard predicates aliases that can be used in > expressions and by allowing 'or' and 'and' -- which should > be avoided AT ALL TIMES AND IN ALL PLACES -- in both contexts. [Interesting comments on guards deleted.] > The whole point of section 3.4 is > "SINCE guards are restricted expressions, > they ought to be unrestricted expressions" > but since the assumption is flat-out false, the conclusion is > unwarranted. > > The limitations of guards have NO intrinsic connection with > receive, although they turn out to be very pleasant for receive. > Erlang guards have their roots in concurrent logic programming > languages such as Strand88, GHC, FCP, and Parlog. Sure, there is a historical connection. Still, it is hard to see how selective receive could work if ordinary expressions were allowed as guards. > Whenever I see people demanding that guards should be complexified, I hope you didn't think that I suggested that? > I find myself wondering "do I know something about readability and > maintainability that they don't, or is it the other way around?" > The way I see it, if your selection process is complex enough that > you *want* to write complex guards with user defined functions, > then what you *need* is either > - abstract patterns, or > - a rewrite. > > Let me offer an example I saw today, in > http://www.infoq.com/articles/erlang-dsl [removed example of how case analysis can be simplified] > > > LINKED MODULES > ============== > > p6. "Erlang's record facility is rather unusual (at least in the > world of functional programming languages)." > > If you are willing to count the Lisp family as functional, then > this is historically quite untrue. I've lost count of the number > of times I've written Scheme code like > > (define (make-node Op Left Right) > (vector 'node Op Left Right)) > (define (node? Node) > (and (vector? Node) > (= (vector-length Node) 4) > (eq? (vector-ref Node 0) 'node))) > (define-macro (node-op Node) > `(vector-ref ,Node 1)) > (define-macro (node-left Node)) > `(vector-ref ,Node 2)) > (define-macro (set-node-left! Node New-Left) > `(vector-set! ,Node 2 ,New-Left)) > ... > OK. The difference is of course that Erlang's macros are lexical, while Lisp's work on the internal representation. > The problems of -include have been known for a long time. > My "Delenda est preprocessor" paper was written some time > in July 1998 (at least, that's the date on the only copy > I can find). Sure. I suspect that many people recognized it as a bad idea almost immediately. > > p7. "the obvious alternative would be to expand a record definition > into a set of function definitions". > > Well, no, because you can't use function calls as patterns. > That's why abstract patterns are needed. True. So we we need a mechanism to recognize certain patterns and replace the pattern match with function calls that test and extracts subtrees. > > The work I did for Fergus O'Brien back in 1998 -- I miss him, > and I especially miss getting paid for thinking about Erlang! -- > included a proposal that I'll now reconstruct for you, with > two small changes. > > Two new directives replace -import(Mod, Funs). > > -import(Module, When, Funs). > -use(Module, When, Funs). > > Module : an atom > When : 'early' or 'late' > Funs : a list of Name/Arity (or #Name/Arity) pairs. > That could also work. > A -use directive creates a dependency (recorded in the .beam > file) between this module and Module. It also notifies the > compiler that this module may call any of the functions (and/or > abstract patterns) listed in Funs, but if so, a module prefix > is required. > > If When is 'late', the dependency is a load-time one: before > EXECUTING this module, Module should be LOADED. Cyclic late > dependencies are perfectly OK; all the members of a cycle > should be loaded before running anything from any of them. > This dependency exists even when Funs is empty; you might be > using some kind of dynamic call to call functions from Module. > The only thing that the compiler may assume about Module is that > loading will have ensured that the named functions all exist, > even if only as loader-generated stubs that raise an undefined > function exception. > > If When is 'early', the dependency is a compile-time one: > before COMPILING this module, Module should be COMPILED, and > so if Module is changed, this module also needs recompiling. > Cyclic early dependencies are not allowed. The compiler may > assume about Module anything whatever that it discovers in the > .beam file. (Arguably interface information should be in > a separate file, not unlike the .hi files used by GHC. > I propose ".ern" (for .ERlang iNfo), in honour of the late > Ern Malley.) > > The difference between -import and -use is that -import > allows (but does not require) the listed functions to be called > without a module prefix, provided there is no clash with any > other imported function or any function defined in this module. > > There may be more than one -use or -import for a Module, and > there may be both -import and -use directives for the same > Module. There may be both early and late dependencies on a > module, but if so, 'early' wins. > > early/late is split out as a separate argument so that it can > be determined by a macro. Looks like a variation of my proposal. (OK, technically my proposal is a variation of yours :-) I would be happy to see either implemented. Still, I'm not certain that this resolves all issues regarding inlining of function calls across module boundaries. It would be nice if those who rely on hot-code loading in their work could say something about which precautions they take to avoid consistency problems when updating a module. For example, what is done if a record definition needs to be changed? > > > BINDINGS VIA A TYPE SYSTEM > ========================== > > p8. "If we compare with the equivalent code in ... SML" > > But why choose SML as the comparison? Haskell's > > y*y where x = 42, y = x+x > > requires only one keyword ('where') instead of SML's four. > I'm not arguing for adding 'where' to Erlang, just pointing > out that comparisons against a limited choice of comparands > can be misleading. Yes. I used SML as I know that language well. Either way, the problem is really the nesting introduced by let. (I assume where is similar.) > > As a matter of fact, I rather like this proposal, and think Nice to hear. > it might be a nice style check to have. Although it _is_ a > bit dodgy that > X = 1 exports X > X = 1, ok does NOT export X > and it's a bit of a worry that > X = Y = f() > is a type error in the proposed scheme. I found another > problem last night, but it was 1am and I can't read what I > wrote. Yes. This type of thing need to be tried out in practice before one can say with certainity that it is practical. > > I think this proposal should be finished and written up > separately and EEPed as a style check. > > > A MINI LANGUAGE > =============== > > The reference to the CELL is a little inaccurate, not > that it matters. (The heart of a CELL is a two-way > CMT version of the PowerPC, with AltiVec.) OK. > This is an interesting idea, but there isn't really a > proposal here yet. It will be very interesting indeed > to discover what Apple's OpenCL looks like; in the > mean time, there's CUDA. > > The trick with something like this is that there really > isn't a whole lot of point in it unless you go native. > OpenCL, of course, relies on LLVM. Of course. > > I suspect that > - some sort of opaque numeric array values (not unlike > APL) > - calling out to high performance libraries like > ATLAS, the CBLAS, GSL, perflib, &c > might be a more effective way to take advantage of > the vector processing facilities in modern machines. A slightly different approach. Are you saying that the APL type arrays should appear as Erlang data structures? > Whoo, wouldn't that be fun! A hybrid of Erlang and APL2! > MACROS > ====== > > p14 "I don't know of any language that combines Lisp-style > macros with a syntax that isn't Lisp-like." > > `C (Tick-C), and arguably, Template Haskell. > 'forms' in WonderPop (the Pop-2 implementation for TOPS-10 > that Robert Rae worked on) are closer to Scheme's macros. Also, there have been attempts to retrofit Lisp with a more conventional syntax. > From my experience with Lisp, there are macros that could > have been written using Scheme syntax forms, and there are > macros that make essential use of assignment. Using Nystrom's > own example, you really _can't_ translate a "def-nt" form into > (part of) a LALR parser in isolation; the information has to be > set aside until you have seen all of the rules for that grammar. > That setting aside involves some sort of data base or other > mutable variable. Yes. The CL-Yacc library requires that the defgrammar form comes after all rules. The LR library could provide a function that sets up the grammar and leave it to the developer to decide whether to do this at runtime or compile-time. Either way, we need a way to store the rules before processing them. > > I note that the closures proposal for Java includes bending > Java syntax so that things like > with_lock (Some_Lock) { statements... } > which are traditionally handled in Lisp using macros > are in fact procedure calls, so that inlining does practically > everything you want. Here's the trick] > [for] Primary (FormalParameters : [Expressions]) Statement > ==> > Primary(Expressions, {FormalParameters => Statement}); > > [for] Primary ([Expressions]) Statement > ==> > Primary(Expressons, {=> Statement}); > > where {formals => statements [expression]} will be Java's syntax > for a lambda-expression. It is true that many of CL's macros could actually be expressed as higher-order functions, except that the syntax becomes awkward. It will be interesting to see how this works out. Thank you for your comments! Sven-Olof From paul-trapexit@REDACTED Thu Jun 26 18:43:36 2008 From: paul-trapexit@REDACTED (Paul Mineiro) Date: Thu, 26 Jun 2008 09:43:36 -0700 (PDT) Subject: [erlang-questions] When is code_change triggered? In-Reply-To: <84fb38e30806260905qb6426f2l26e63888a294d937@mail.gmail.com> References: <84fb38e30806260905qb6426f2l26e63888a294d937@mail.gmail.com> Message-ID: We had the same at question here in Hazzard County several months ago. Hot code loading is a low-level feature built into the runtime; there's space for two versions of a module and using module-prefixed calls implies a call into the latest version (and a local call implies continue to use the same module version). Before you hot code load a module, you have to first nuke the oldest version of the module (and any lingering processes). This is the purpose of code:purge/1 and code:soft_purge/1. You can then load a new module using code:load_file/1, code:load_abs/1, code:load_binary/3, etc. After loading the formerly newest version of the code will now occupy the oldest version slot. Code change is a high-level feature which is implemented by convention in the OTP libraries. The gen_X family has an inner loop which responds to special "system messages" which cause the process to transfer control to a seperate receive loop which only processes special upgrade messages, which ultimately cause the code_change/3 callback to be called. In this manner you will not process further normal gen_X messages until the new code has been loaded and the code_change handler called. This sequence is orchestrated by release_handler via calls like release_handler:install_release/1 and release_handler:upgrade_app/2. The final piece of information is that calling c(Module) in the shell is shorthand for invoking the compiler and then invoking the various code methods for loading the new version of the module. It is *not*, however, shorthand for invoking the release_handler; that wouldn't make sense for many reasons. (The shortest path to seeing your code_change/3 handler called is, imho, rather long). Hope that helps, -- p p.z. after figuring this out we wrote erlrc: http://code.google.com/p/erlrc/ On Thu, 26 Jun 2008, tsuraan wrote: > When I started playing with gen_server a few years ago, I recalled > that doing a c(module) in the REPL would immediately update the code > of my running servers. Now, I cannot get that to happen, so I'm > trying to figure out what I'm doing wrong. My test server is minimal: > > -module(test). > -export([start_link/0, start/0]). > -export([init/1, handle_call/3, handle_cast/2, handle_info/2, > terminate/2, code_change/3]). > > start_link() -> > gen_server:start_link({ local, ?MODULE }, ?MODULE, [], []). > > start() -> > gen_server:start({ local, ?MODULE }, ?MODULE, [], []). > > init([]) -> > process_flag(trap_exit, true), > { ok, {}}. > > handle_call(Msg, From, State) -> > io:format("Unexpected Call ~p From ~p~n", [Msg, From]), > { reply, { error, badcall }, State }. > > handle_cast(Msg, State) -> > io:format("Unexpected Cast ~p~n", [Msg]), > { noreply, State }. > > handle_info(Msg, State) -> > io:format("Unexpected Info ~p~n", [Msg]), > { noreply, State }. > > terminate(Reason, _State) -> > io:format("Terminating ~p for reason ~p~n", [ self(), Reason ]), > ok. > > code_change(OldVsn, State, Extra) -> > io:format("Upgrading from ~p with extra data ~p~n", [OldVsn, Extra]), > { ok, State }. > > I can run this with test:start(), but subsequent c(test) never print > the message "Upgrading from ...". Can anyone tell what I'm doing > wrong, or is my memory just bad and c(module) has never caused running > instances of that module to be upgraded? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > In an artificial world, only extremists live naturally. -- Paul Graham From ulf@REDACTED Thu Jun 26 18:38:21 2008 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 26 Jun 2008 18:38:21 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <4863B777.908@san.rr.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <4863B777.908@san.rr.com> Message-ID: <8209f740806260938o6cb373e9v7a33706c76b7d846@mail.gmail.com> 2008/6/26 Darren New : > Richard A. O'Keefe wrote: >> I hope that we can rely on Erlang garbage collection to >> collect any process suspended in a receive construct and not registered >> and not referenced by any other process. > > ... > It would also imply that term_to_binary serializing a process ID and > then dropping the variable holding the actual PID would allow the > process to exit. So process A wrapping a PID of process X in a binary, > dropping it into processes B's mailbox, then exiting, might cause > process X to exit before process B read the binary out of its mailbox > and turned it back into a term. I think we can stipulate that when serializing a pid, perhaps storing it on disk, then re-creating it and trying to use it, all bets are off anyway.* Still, it would of course be quite difficult to introduce the semantics that a process exits if it's in a blocking receive and there are no references to it - even if it would turn out to have been a good idea in the first place. * I haven't checked the GC implementation, but I would assume that a pid is available to be reused when the process has exited and there are no known references to the pid (serialized references don't count). BR, Ulf W From tsuraan@REDACTED Thu Jun 26 19:01:05 2008 From: tsuraan@REDACTED (tsuraan) Date: Thu, 26 Jun 2008 12:01:05 -0500 Subject: [erlang-questions] When is code_change triggered? In-Reply-To: References: <84fb38e30806260905qb6426f2l26e63888a294d937@mail.gmail.com> Message-ID: <84fb38e30806261001m3d2a29c9x92dee32445249d56@mail.gmail.com> > The final piece of information is that calling c(Module) in the shell is > shorthand for invoking the compiler and then invoking the various code > methods for loading the new version of the module. It is *not*, however, > shorthand for invoking the release_handler; that wouldn't make sense > for many reasons. (The shortest path to seeing your code_change/3 handler > called is, imho, rather long). It seems like there ought to be a message that one could send to the gen_X servers to tell them that there's new code they should be using. There isn't such a thing though? I can live with restarting my erl shells to test code fixes, but it just doesn't seem like it should be necessary. I'll have a look at the erlrc when I get a chance. Maybe there's inspiration to be found there. From juanjo@REDACTED Thu Jun 26 19:26:01 2008 From: juanjo@REDACTED (Juan Jose Comellas) Date: Thu, 26 Jun 2008 14:26:01 -0300 Subject: [erlang-questions] Asterisk manager parser question In-Reply-To: <4ecde87b0806260305t7786b523xae50ec396aa8b58c@mail.gmail.com> References: <4ecde87b0806260305t7786b523xae50ec396aa8b58c@mail.gmail.com> Message-ID: <1c3be50f0806261026u1a479a6bt68aae6275d3d3bfb@mail.gmail.com> I did something similar but for the FreeSWITCH mod_event_socket interface (if you're looking into using Asterisk for anything serious, you owe yourself to check what FreeSWITCH has to offer). The code that handles all of this is a bit big for an email, but the code that parses the buffer used to read from the socket is what you're probably more interested in. If you want more help with this you can contact me off-list. parse_headers(Headers, Buffer) when is_list(Headers), is_binary(Buffer) -> parse_headers(Headers, Buffer, undefined, 0). parse_headers(Headers, Buffer, _SepOffset, Offset) when Offset > size(Buffer) -> {incomplete, Headers, Buffer}; parse_headers(Headers, Buffer, SepOffset, Offset) -> case Buffer of % Match the end of a text line. <> -> handle_header_line(Headers, Buffer, Line, Tail, SepOffset); <> -> handle_header_line(Headers, Buffer, Line, Tail, SepOffset); % Match the first separator between the header name and value. <<_Line:Offset/binary, $:, _Tail/binary>> when (SepOffset =:= undefined) -> parse_headers(Headers, Buffer, Offset, Offset + 1); _ -> parse_headers(Headers, Buffer, SepOffset, Offset + 1) end. handle_header_line(Headers, _Buffer, <<>>, Tail, undefined) -> case Headers of [] -> % Empty lines at the beginning of the header lines are not considered errors. {incomplete, Headers, Tail}; _ -> % An empty line after finding more than one non-empty header means we're % done parsing headers. {ok, lists:reverse(Headers), Tail} end; handle_header_line(Headers, Buffer, Line, _Tail, undefined) when size(Line) > 0 -> % Return an error if the separator was not found. {error, Headers, Buffer}; handle_header_line(Headers, Buffer, Line, Tail, SepOffset) -> case Line of <> -> parse_headers([{Name, Value} | Headers], Tail); <> -> parse_headers([{Name, Value} | Headers], Tail); _ -> {error, Headers, Buffer} end. 2008/6/26 Jon Gretar Borgthorsson : > Hi.. > I've been playing around with erlang for a month and playing around with > gen_tcp and more. Now I've decided to redo a script that I have that parses > events from the asterisk management api. The old script tended to be a bit > unstable and I think that Erlang would be perfect for the job. > > However I'm having a little problem solving how to read the information in > an elegant way. > > Basically for those who don't know the Asterisk manager API then it's > basically a tcp port you connect to and read events as they come. Each event > is formatted like this: > > Event: Agentlogoff > Agent: > Logintime: > Uniqueid: > > > And events are seperated by 2 returns. What I was thinking of doing is > something that listens to the port and sends events clause in whole to > another process that analyzes it. What I'm having problems with is just the > first part. Recognizing the event clouses in some elegant way and sending > them on to the parsing function. I have to try to save the message up in a > buffer and when it senses 2 returns then creates a process to parse it and > clears the buffer to read the next event. I'm sure that there is some > elegant way to do this but since this is my first real erlang project then > I'm a bit lost on how the best way is. Almost everything else I have figured > out and it's just this last problem that needs solving. > > Anyone have a few free minutes to assist me on this? > > > Regards > - Jon > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dnew@REDACTED Thu Jun 26 19:41:49 2008 From: dnew@REDACTED (Darren New) Date: Thu, 26 Jun 2008 10:41:49 -0700 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <8209f740806260938o6cb373e9v7a33706c76b7d846@mail.gmail.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <4863B777.908@san.rr.com> <8209f740806260938o6cb373e9v7a33706c76b7d846@mail.gmail.com> Message-ID: <4863D4DD.8020509@san.rr.com> Ulf Wiger wrote: > I think we can stipulate that when serializing a pid, perhaps storing it > on disk, then re-creating it and trying to use it, all bets are off anyway.* I would certainly hope not. One has to serialize a pid to send it to a different node. I would hope that creating a process on my node and then sending you the only copy of the pid returned by spawn() doesn't cause the process to exit if it's waiting for your node to send it a message. > Still, it would of course be quite difficult to introduce the semantics > that a process exits if it's in a blocking receive and there are no > references to it - even if it would turn out to have been a good idea > in the first place. That's a good idea. You just have to define what "references to it" means more carefully. It's too easy for Erlang to have references that aren't "known" by any of the runtimes. There are hidden nodes (which may or may not affect things), for example, even if you discount serialized values. If I'm linked to a process that is blocked in such a receive, does that blocked process terminate? Does it get GCed without terminating? Does that stop other processes for which it had references transitively? If I lose a connection to the only node that has a copy of my pid, should I exit? What if the connection comes back? > * I haven't checked the GC implementation, but I would assume that > a pid is available to be reused when the process has exited and there > are no known references to the pid (serialized references don't count). I would think that's a bad idea, actually. Especially since it's so easy to have an "unknown reference" in Erlang. Anything in a TCP buffer is going to be an "unknown reference". You could, of course, answer all these questions. I'm just saying that GCing processes in Erlang that haven't exited seems unusually difficult to get right. > * I haven't checked the GC implementation, but I would assume that > a pid is available to be reused when the process has exited and there > are no known references to the pid (serialized references don't count). Well, it looks like you can only have 2^20 unique references per node (or three crashes) before you start having problems, and the documentation for EXT_PID isn't complete enough to actually say how many PIDs you can have per node. (Hint to authors: documenting some field as "this field is included to make things better" isn't informative. Similarly, stating how big a field is without saying why you selected a "new" representation of something isn't informative. :-) Of course, since sending to a non-existant PID is undetectable by the sender without going through all kinds of monitor() dances, making things reliable in the face of network partitions would seem to need a fair amount of code. I don't think I'd use the distribution primitives over the public internet to talk between nodes. I have enough trouble keeping my stuff running in the face of ISPs disappearing for dozens of minutes at a time without the software assuming lack of error is success. :-) -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From ulf@REDACTED Thu Jun 26 19:52:46 2008 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 26 Jun 2008 19:52:46 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <4863D4DD.8020509@san.rr.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <4863B777.908@san.rr.com> <8209f740806260938o6cb373e9v7a33706c76b7d846@mail.gmail.com> <4863D4DD.8020509@san.rr.com> Message-ID: <8209f740806261052g56ed1f94lad2f1d0fb073ed2b@mail.gmail.com> 2008/6/26 Darren New : > Ulf Wiger wrote: >> >> I think we can stipulate that when serializing a pid, perhaps storing it >> on disk, then re-creating it and trying to use it, all bets are off >> anyway.* > > I would certainly hope not. One has to serialize a pid to send it to a > different node. I would hope that creating a process on my node and then > sending you the only copy of the pid returned by spawn() doesn't cause the > process to exit if it's waiting for your node to send it a message. In that case, the remote holder of the pid cannot rely on the pid referring to the same process if it's used some time later. It should monitor the process in order to detect whether it dies. The process doesn't exit just because there are no known references to it, but once a process /has/ died, and all known (local) references are gone, some other process may reuse that pid. This is why storing pids persistently is a very bad idea. BR, Ulf W From dnew@REDACTED Thu Jun 26 20:12:10 2008 From: dnew@REDACTED (Darren New) Date: Thu, 26 Jun 2008 11:12:10 -0700 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <8209f740806261052g56ed1f94lad2f1d0fb073ed2b@mail.gmail.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <4863B777.908@san.rr.com> <8209f740806260938o6cb373e9v7a33706c76b7d846@mail.gmail.com> <4863D4DD.8020509@san.rr.com> <8209f740806261052g56ed1f94lad2f1d0fb073ed2b@mail.gmail.com> Message-ID: <4863DBFA.6000401@san.rr.com> Ulf Wiger wrote: > 2008/6/26 Darren New : >> Ulf Wiger wrote: >>> I think we can stipulate that when serializing a pid, perhaps storing it >>> on disk, then re-creating it and trying to use it, all bets are off >>> anyway.* >> I would certainly hope not. One has to serialize a pid to send it to a >> different node. I would hope that creating a process on my node and then >> sending you the only copy of the pid returned by spawn() doesn't cause the >> process to exit if it's waiting for your node to send it a message. > > In that case, the remote holder of the pid cannot rely on the pid referring > to the same process if it's used some time later. How long is "some time"? We're conflating "reusing the PID" and "GCing a process which we don't think anyone will ever wake up". If I GC the process as soon as there's no local or remote reference to the PID, I might wind up GCing the process while the data representing the PID is on the wire on its way to the node going to use it. > It should monitor the process in order to detect whether it dies. Again, not what I'm talking about. Erlang gives me ways to mostly do this unreliably. (If Erlang had a reliable way to do it, we wouldn't need mnesia:set_master_nodes, for example.) However, I'm talking about when there's no failure at all. You can't GC a process just because there's no unpacked/unserialized copies of a PID for the process, because serialization is an intrinsic part of how Erlang works with pids. Even without failures of nodes or links or use of term_to_binary, you might still GC a process while other processes "have" references to it, simply due to timing of deserialization of the internode protocol. > The process doesn't exit just because there are no known references > to it, Right. That was what I was talking about. I was pointing out that programming as if this feature may be right around the corner is probably unwise. > but once a process /has/ died, and all known (local) references are > gone, some other process may reuse that pid. This is why storing pids > persistently is a very bad idea. Except that storing pids "persistently" in variables is how one uses them. No matter where you store them, in variables or in disk files, you have to get rid of them when you get a monitor or link message saying the process exited, and you have to deal with the fact that you might get those messages when the process has, indeed, not yet exited and indeed doesn't know those messages have been sent. I'm not sure that storing a pid in a table is particularly more difficult than storing a pid in the variables of a process that's not expected to ever exit. -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From svenolof@REDACTED Thu Jun 26 21:18:00 2008 From: svenolof@REDACTED (Sven-Olof Nystr|m) Date: Thu, 26 Jun 2008 21:18:00 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> Message-ID: <18531.60264.991238.276228@harpo.it.uu.se> Richard A. O'Keefe writes: > > On 26 Jun 2008, at 9:57 am, Ulf Wiger wrote: > > Haskell has channels, and so does .NET (mailbox objects). > > And Concurrent ML, amongst others, had them even earlier. > > There is > > therefore an opportunity to compare programs and try to determine > > whether programming with channels makes for more or less > > readable code than erlang's selective receive. > > It seems to me that programming with Erlang's receive is > far simpler to do, to read, and to reason about than > "channels" (strictly speaking, mailboxes). The reason > is not far to seek: > > if channels are "objects", they can go ANYWHERE. > > There's another one. Although Nystrom presented two > receive forms, the text made it clear that two more were > required. But he had forgotten the need for something > like BSD select()/System V poll()/Ada selective accept/ > other languages' multiwaits. That means another two > forms. > > nystrom_receive() % default channel > nystrom_receive(Timeout) > nystrom_receive_from(Channel) > nystrom_receive_from(Channel, Timeout) > nystrom_receive_from_any(Channel_Set) > nystrom_receive_from_any(Channel_Set, Timeout) True. But they are all simpler than selective receive. > > But it gets worse. > > if channels are "objects", they can go anywhere, > BUT THEY DON'T GET THERE WITHOUT BEING CARRIED! In Erlang today, if one wants to keep track of communication with a particular process one needs to carry the pid. > > You have to explictly pass them around, especially loops. > So to handle the simple bounded buffer you would find > yourself writing stuff like this: > > buffer(Status, Contents, GetChan, PutChan) -> > Channels = case Status > of full -> [GetChan] > ; empty -> [PutChan] > ; _ -> [GetChan,PutChan] > end, > case nystrom_receive_from_any(Channels) > of {GetChan,Who} -> > {Status1,Contents1,Msg} = pop(Contents), > Who ! Msg > ; {PutChan,Msg} -> > {Status1,Contents1} = add(Contents, Msg) > end, > buffer(Status1, Contents1, GetChan, PutChan). I've commented on your bounded_buffer example in my previous mail. I'm not sure what you are doing here. It might help if you could show the example using selective receive. > > That's assuming that multireceive returns a {Channel,Message} > pair. Another approach would be to pass a list of {Channel, > Handler} pairs, when the code would look like > > buffer(Status, Contents, GetChan, PutChan) -> > GetHandler = fun (Who) -> > {Status1,Contents1,Msg} = pop(Contents), > Who ! Msg, > buffer(Status1, Contents1, GetChan, PutChan) > end, > PutHandler = fun (Msg) -> > {Status1,Contents1} = add(Contents, Msg), > buffer(Status1, Contents1, GetChan, PutChan) > end, > nystrom_receive_from_any( > case Status > of full -> [{GetChan,GetHandler}] > ; empty -> [{PutChan,PutHandler}] > ; _ -> [{GetChan,GetHandler}, > {PutChan,PutHandler}] > end). > > How this is in any way simpler than a selective receive > entirely escapes me. This example escapes me, too. > > There's another problem with this. > I can easily fire up a bounded buffer process in Erlang: > > Buffer = spawn (fun () -> > buffer(empty, empty_buffer_contents()) > end), > > because spawn/1 delivers the Pid *outside* the process. > But new_channel/0 delivers its result *inside* the > process. This isn't so in Concurrent ML, where mailboxes > will talk to anybody. In CML, the parent process would > create the mailboxes GetChan and PutChan: > > GetChan = Mailbox.mailbox(), > PutChan = Mailbox.mailbox(), > Buffer = spawn (fun () -> > buffer(empty, empty_buffer_contents(), > GetChan, PutChan) > end), > > But in Nystrom's proposal, "only the creator of a channel may > receive messages from it" (p 5). So the channels have to be > created *inside* the new buffer process. How then do other > processes get their "hands" on them? > > Ahah! It's all so EASY without selective receive! > > GetChanChan = new_channel(), > PutChanChan = new_channel(), > Buffer = spawn (fun () -> > GetChan = new_channel(), > PutChan = new_channel(), > GetChanChan ! GetChan, > PutChanChan ! PutChan, > buffer(empty, empty_buffer_contents(), > GetChan, PutChan) > end), > GetChan = ne_receive(GetChanChan), > PutChan = ne_receive(PutChanChan), > ... Let me just say that I find your bounded buffer example profoundly unconvincing. The buffer may be bounded, but there is nothing to prevent another process from filling the mailbox with "put" messages. > By the way, recall that the problem that this "simplification" > is supposed to solve is this: "it is considered bad style to > leave too many messages in the mailbox". Let me quote from the > Concurrent ML documentation for the Mailbox structure: Well, that was never the main problem. Selective receive is complex and unnecessary. That was my main point. Programmers sometimes manage to get into problems with it, but it was always my impression that there were strategies to avoid these problems. I thought however, that the fact that one had to make certain precautions to avoid getting into trouble strengthened the argument against selective receive. > Note that mailbox buffers are unbounded, which means that > there is no flow control to prevent a producer from greatly > outstripping a consumer, and thus exhausting memory. > > If your mailbox is getting full, it's not another questionably > simpler language construct you need. It's FLOW CONTROL, which > is a higher level protocol issue. > One thing I particularly like about 'receive' in Erlang is the > fact that it is visually hard to miss. Things that look like > function calls are much easier to lose sight of in the thick > undergrowth of things that look like function calls. In fact it > won't even do to just look for 'ne_receive' (or 'nystrom_receive*'). > (It _is_ possible to hide 'receive', but it's a whole lot harder.) That is true. One might want to look for something that is easier to find in a block of code. Sven-Olof From pablo.polvorin@REDACTED Thu Jun 26 22:01:35 2008 From: pablo.polvorin@REDACTED (Pablo Polvorin) Date: Thu, 26 Jun 2008 17:01:35 -0300 Subject: [erlang-questions] surprise! binary vs list in file parse. In-Reply-To: References: <95be1d3b0806260739n3deb634l5e133d8683bf7502@mail.gmail.com> Message-ID: <1ffe809c0806261301h488df5adu4af3713f15fd6503@mail.gmail.com> On my machine, which hasn't the R12B improvements, I'm getting the same results than you. But after a little rewrite, things go better: 2> timer_avg:tc(act,parse,["test.txt"],1000). Max: 26200 Min: 3066 Avg: 3853.10 ok 3> timer_avg:tc(act_2,parse,["test.txt"],1000). Max: 28206 Min: 5495 Avg: 5968.27 ok 4> timer_avg:tc(act_3,parse,["test.txt"],1000). Max: 19227 Min: 3045 Avg: 3215.01 ok the code I used for act_3 is like this: parse(Bin) when is_binary(Bin) -> parse(Bin, 0,[], []). parse(Bin,CurrentOffset,CurrentLine,Lines) -> case Bin of <> -> parse(Rest,0,[Field|CurrentLine],Lines); <> -> parse(Rest,0,[],[lists:reverse([Field|CurrentLine])|Lines]); <<_Field:CurrentOffset/binary,_Char,_Rest/binary>> -> parse(Bin,CurrentOffset+1,CurrentLine,Lines); <<>> -> {ok,lists:reverse(Lines)}; _ -> {error,bad_file} end. explicitly keep track of the current offset to avoid copying binaries on each append. I think that on newer erlang releases, the performance of act_2 should be similar, but I don't have it installed here to test. See comments on http://www.erlang.org/pipermail/erlang-questions/2008-June/036166.html 2008/6/26 litao cheng : > yes, I retry the test like you say. > the result is same. > thank you! > > 2008/6/26 Vlad Dumitrescu : > > Hi >> >> 2008/6/26 litao cheng : >> >> I had read joel's article:Parsing text and binary files with Erlang. In >>> the article, the author show how to parse a comma-delimited text file. >>> I write the code for practice, the module source is act.erl. By the way, >>> I see in the bottom of joel' article, a buddy give a comment, he says, he >>> use binary to instead of some list, It seems so efficient, so I write the >>> second code, it's act_2.erl. >>> >>> Finally, I want to test how much the act_2 faster than act. So suprise, >>> In my Compute, The result is : >>> 96> timer_avg:tc(act, parse, ["test.txt"], 100). >>> Max: 15991 >>> Min: 1 >>> Avg: 2339.05 >>> >>> 100> timer_avg:tc(act_2, parse, ["test.txt"], 100). >>> Max: 15997 >>> Min: 1 >>> Avg: 4839.03 >>> >>> the timer_avg is a moudule evaluates apply(Module, Function, Arguments)N times and measures the elapsed real time, about Max, Min, Avg. (use >>> timer:tc/3). >>> >>> who will give me some explain? >>> >> >> You might want to retry the tests, starting them instead with >> spawn(timer_avg, tc, [act, parse, ["test.txt"], 100]). >> spawn(timer_avg, tc, [act_2, parse, ["test.txt"], 100]). >> >> This way you don't get any random garbage collections to interfere. >> >> regards, >> Vlad >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- -- pablo http://ppolv.wordpress.com ---- -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin@REDACTED Thu Jun 26 22:22:49 2008 From: kevin@REDACTED (Kevin Scaldeferri) Date: Thu, 26 Jun 2008 13:22:49 -0700 Subject: [erlang-questions] Writing a Concurrent Binary Search Tree Data Structure in Erlang In-Reply-To: <909c265f0806251510x60ab819aqfb074b5558d7fc46@mail.gmail.com> References: <909c265f0806250222q76e6651emd164ea221f15202f@mail.gmail.com> <4d08db370806250240gbf0e527n798a296292124ef8@mail.gmail.com> <909c265f0806250246s7abbcb2at6884f50e31c5d81c@mail.gmail.com> <4d08db370806250257h63fa7cb7m2dbf1a8b682768e2@mail.gmail.com> <909c265f0806251510x60ab819aqfb074b5558d7fc46@mail.gmail.com> Message-ID: On Jun 25, 2008, at 3:10 PM, Fuad Tabba wrote: > I think I understand what you're saying here; but I still can't wrap > my head around putting that into code. Is there similar code that > you know of available online? The structure doesn't have to be a > BST, just anything that demonstrates the idea of having a structure > that stores lots persistent data to be viewed/modified by many > processes and is scalable... > > > On Wed, Jun 25, 2008 at 9:40 PM, Hynek Vychodil > wrote: > Because there is no shared state between processes in Erlang, I > would start think about process per tree node, not process per > action as usual in classic shared memory (for example C) > implementations. > I know, it sounds weird, but may be help you :-) Here's a basic outline of this idea, if I understand it right. delete() is not implemented, and it has some other deficiencies, but I hope it at least starts to demonstrate the idea of a recursive data structure represented as a collection of processes. -kevin %%%%%%%%%%%%%%%%%%%%%%%%% -module(cbst). -export([new/1, exists/2, insert/2, delete/2]). -record(node, {key, left=null, right=null}). %% %% public API %% new(Key) -> spawn(fun() -> create(Key) end). exists(Node, Key) -> Node ! {exists, Key, self()}, receive true -> true; false -> false end. insert(Node, Key) -> Node ! {insert, Key}. delete(Node, Key) -> Node ! {delete, Key}. %% %% private functions %% create(Key) -> Node = #node{key=Key}, loop(Node). loop(Node) -> receive {exists, Key, Caller}=Q -> case Node#node.key of Key -> Caller ! true; This when Key < This, Node#node.left =/= null -> Node#node.left ! Q; This when Key > This, Node#node.right =/= null -> Node#node.right ! Q; _ -> Caller ! false end, loop(Node); {insert, Key} -> case Node#node.key of Key -> loop(Node); % already in tree This when Key < This -> case Node#node.left of null -> Left = new(Key), loop(Node#node{left=Left}); Left -> Left ! {insert, Key}, loop(Node) end; This when Key > This -> case Node#node.right of null -> Right = new(Key), loop(Node#node{right=Right}); Right -> Right ! {insert, Key}, loop(Node) end end; {delete, Key} -> % left as an exercise for the reader loop(Node) end. From fuad@REDACTED Thu Jun 26 22:54:44 2008 From: fuad@REDACTED (Fuad Tabba) Date: Fri, 27 Jun 2008 08:54:44 +1200 Subject: [erlang-questions] Writing a Concurrent Binary Search Tree Data Structure in Erlang In-Reply-To: References: <909c265f0806250222q76e6651emd164ea221f15202f@mail.gmail.com> <4d08db370806250240gbf0e527n798a296292124ef8@mail.gmail.com> <909c265f0806250246s7abbcb2at6884f50e31c5d81c@mail.gmail.com> <4d08db370806250257h63fa7cb7m2dbf1a8b682768e2@mail.gmail.com> <909c265f0806251510x60ab819aqfb074b5558d7fc46@mail.gmail.com> Message-ID: <909c265f0806261354m7e3f711fk8ce3cafd0500fd3b@mail.gmail.com> Cool, that's matches my understanding of Hynek's suggestion as well. Thanks Kevin. I'm actually writing code along similar lines to test out the idea. It seems rather simple and elegant; what I don't like though is the added overhead of a process per node - I know that Erlang processes are light, but if I recall correctly from reading in Joe's book, each process has memory overhead in the order of 100 bytes. If the the data per node is large then it's not a big deal, but if the amount of data stored per node is small then that could be some serious overhead. Implementing delete() is going to be a bit tricky, need to be wary of deadlocks; but this isn't an Erlang issue, this is a general issue with writing a concurrent and shared data structure. Thanks again! Cheers, /Fuad On Fri, Jun 27, 2008 at 8:22 AM, Kevin Scaldeferri wrote: > > On Jun 25, 2008, at 3:10 PM, Fuad Tabba wrote: > > I think I understand what you're saying here; but I still can't wrap my >> head around putting that into code. Is there similar code that you know of >> available online? The structure doesn't have to be a BST, just anything that >> demonstrates the idea of having a structure that stores lots persistent data >> to be viewed/modified by many processes and is scalable... >> >> >> On Wed, Jun 25, 2008 at 9:40 PM, Hynek Vychodil >> wrote: >> Because there is no shared state between processes in Erlang, I would >> start think about process per tree node, not process per action as usual in >> classic shared memory (for example C) implementations. >> I know, it sounds weird, but may be help you :-) >> > > > > Here's a basic outline of this idea, if I understand it right. delete() is > not implemented, and it has some other deficiencies, but I hope it at least > starts to demonstrate the idea of a recursive data structure represented as > a collection of processes. > > > -kevin > > %%%%%%%%%%%%%%%%%%%%%%%%% > > -module(cbst). > -export([new/1, exists/2, insert/2, delete/2]). > > -record(node, {key, > left=null, > right=null}). > > %% > %% public API > %% > > new(Key) -> > spawn(fun() -> create(Key) end). > > exists(Node, Key) -> > Node ! {exists, Key, self()}, > receive > true -> true; > false -> false > end. > > insert(Node, Key) -> > Node ! {insert, Key}. > > delete(Node, Key) -> > Node ! {delete, Key}. > > > %% > %% private functions > %% > > create(Key) -> > Node = #node{key=Key}, > loop(Node). > > loop(Node) -> > receive > {exists, Key, Caller}=Q -> > case Node#node.key of > Key -> > Caller ! true; > This when Key < This, Node#node.left =/= null -> > Node#node.left ! Q; > This when Key > This, Node#node.right =/= null -> > Node#node.right ! Q; > _ -> > Caller ! false > end, > loop(Node); > {insert, Key} -> > case Node#node.key of > Key -> > loop(Node); % already in tree > This when Key < This -> > case Node#node.left of > null -> > Left = new(Key), > loop(Node#node{left=Left}); > Left -> > Left ! {insert, Key}, > loop(Node) > end; > This when Key > This -> > case Node#node.right of > null -> > Right = new(Key), > loop(Node#node{right=Right}); > Right -> > Right ! {insert, Key}, > loop(Node) > end > end; > {delete, Key} -> > % left as an exercise for the reader > loop(Node) > end. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anders.nygren@REDACTED Fri Jun 27 00:36:45 2008 From: anders.nygren@REDACTED (Anders Nygren) Date: Thu, 26 Jun 2008 17:36:45 -0500 Subject: [erlang-questions] How is hostname decided Message-ID: When using short names, how is the hostname decided. The reason I am asking is that I have a system based on R12B2 running on RedHat Enterprise 4 [npas@REDACTED lib]$ /opt/npas/bin/erl -sname test Erlang (BEAM) emulator version 5.6.2 [source] [64-bit] [smp:2] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.6.2 (abort with ^G) (test@REDACTED)1> inet:gethostname(). {ok,"npas"} Why is the nodename localhost, when inet:gethostname() returns npas? Looking through the sources I found what looks like the place where hostname is determined. inet_config:set_hostname() -> case inet_udp:open(0,[]) of {ok,U} -> Res = inet:gethostname(U), inet_udp:close(U), set_hostname(Res); _ -> set_hostname({ok, []}) end. But that also gives "npas" /Anders From cthulahoops@REDACTED Fri Jun 27 01:04:22 2008 From: cthulahoops@REDACTED (Adam Kelly) Date: Fri, 27 Jun 2008 00:04:22 +0100 Subject: [erlang-questions] When is code_change triggered? In-Reply-To: <84fb38e30806261001m3d2a29c9x92dee32445249d56@mail.gmail.com> References: <84fb38e30806260905qb6426f2l26e63888a294d937@mail.gmail.com> <84fb38e30806261001m3d2a29c9x92dee32445249d56@mail.gmail.com> Message-ID: <8d1798e90806261604k1280babeoeaae6bf639a01be1@mail.gmail.com> 2008/6/26 tsuraan : > I can live with restarting my erl shells to test code fixes, but it just doesn't seem like it should be > necessary. It's not necessary! Just doing "c(test)." is enough to get the new version loaded and in use. The code_change function will never be called (unless you use the release handler stuff), but your new code is loaded and working. Adam. From fuad@REDACTED Fri Jun 27 01:47:58 2008 From: fuad@REDACTED (Fuad Tabba) Date: Fri, 27 Jun 2008 11:47:58 +1200 Subject: [erlang-questions] Message Ordering and Reliability in Erlang Message-ID: <909c265f0806261647o62ef8d65xadfc0631d37573e5@mail.gmail.com> I was wondering, when sending messages in Erlang, is ordering preserved? i.e. Process A sends 1 then 2 to B; can I count on B receiving 1 before receiving 2 (not talking about selective receive). What about reliability? Assuming that there aren't faulty processes or broken links (slow is ok), is it possible for some messages to be dropped by others not? To use the same example, A send 1 then 2, is it possible for B to only receive 2? Does the answer to these questions apply all the time (i.e. processes on the same core, on different cores/processors, on completely different computers) Thanks, /Fuad -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.reilly@REDACTED Thu Jun 26 03:08:19 2008 From: matthew.reilly@REDACTED (Matthew Reilly) Date: Wed, 25 Jun 2008 18:08:19 -0700 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <1214436806.10157.366.camel@matt.sipphone.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <1214436806.10157.366.camel@matt.sipphone.com> Message-ID: <1214442499.10157.393.camel@matt.sipphone.com> On Wed, 2008-06-25 at 16:33 -0700, Matthew Reilly wrote: > On Tue, 2008-06-24 at 16:23 +0200, Sven-Olof Nystr|m wrote: > > Hello everyone, > > > > I have written down some thoughts and ideas on the future development > > of Erlang. > > > > Main topics: > > > > - some notes on implementation and backward compatibility > > > > - an alternative to Erlang's selective receive > > In terms of Erlang's selective receive, it's possible to get around the > O(N) look up issue w/o a language change (albeit with a slight change to > the VM) if you restrict which patterns you use. > > (Review of the selective receive problem) > The problem is that given code like: > receive > some_term -> some_value > end. > each time this receive is hit, the whole message queue is scanned. If > there are N unmatched messages pending, then this take O(N) each time > this bit of code is hit. > > However, if the receive pattern is static (i.e. it does not include an > already bound variable), then these duplicate scans would be redundant > -- we already know that if a message is not matched once, it won't match > the same pattern in the future. The next time we hit the same receive, > we'd only need to scan messages that have been added since we last did > the same selective receive. (See below about dealing with non-static > patterns). > > In this way, each message would be patterned matched at most once per > selective receive, and thus, regardless of the size of the message > queue, selective receives would become an amoritized O(1) task. > > The current message queue is implemented in the VM by a set of 3 > pointers: "first", "save", "last". "first" points to the first item in > the queue, "last" points to the place to add the next item, and "save" > is a temp pointer used to traverse the message queue. > > A way to implement O(1) selective receives would be to add an > (initially empty) list of saved {id,pointer} pairs to each message > queue. When a selective receive finishes scans through the message > queue, it stores a pointer to the next message would have processed > (keyed to something that uniquely identifies this receive). When the > same selective receives occurs again, it just needs to start at that > saved location (instead of at the "first" message in the list). As an > optimization, once the message queue becomes empty, this list can be > wiped out. > > This adds one pointer's worth of space to each process. Very little > overhead is added for the normal case where all receives have a default > clause. > > This is a sketch of how the VM would need to be modified to have > amortized O(1) selective receives. > > Currently a selective receive such as > func() -> > receive > {high_priority, Message} -> Message > after 0 -> no_message > end. > is implemented in bytecode such as: > {label,14}. > {loop_rec,{f,16},{x,0}}. > {test,is_tuple,{f,15},[{x,0}]}. > {test,test_arity,{f,15},[{x,0},2]}. > {get_tuple_element,{x,0},0,{x,1}}. > {get_tuple_element,{x,0},1,{x,2}}. > {test,is_eq_exact,{f,15},[{x,1},{atom,high_priority}]}. > remove_message. > {move,{x,2},{x,0}}. > return. > {label,15}. > {loop_rec_end,{f,14}}. > {label,16}. > timeout. > {move,{atom,no_message},{x,0}}. > > Here the "loop_rec" opcode returns the message pointed to by the "save" > pointer (which is normally the same as "first"). > If the message queue is empty, it jumps to the "timeout" opcode which > resets the "save" pointer to "first". > If the message queue is not empty, the message is pattern matched > against {high_priority,_}. > If it doesn't match, it calls the opcode "loop_rec_end" when sets > "save" to "save->next", and jumps back to the "loop_rec" opcode. > If it does match, it returns the second item of the tuple, and calls > the "remove_message" opcode which removes the message from the queue and > also sets "save" to "first". > > Suppose we introduce two new opcodes: > restore_queue_pos > save_queue_pos > > save_queue_pos stores the current value of "save" indexed against some > unique key. > restore_queue_pos returns either the value saved by save_queue_pos > (possibly modified by remove_message) or "first" if there was no saved > value. > The remove_message opcode needs be modified to update any saved queues > positions (in case they point to the message that is being deleted) or > remove them all if the queue becomes empty. > > The VM code for the same selected receive using these new opcodes: > > {restore_queue_pos,XXX}. > {label,14}. > {loop_rec,{f,16},{x,0}}. > {test,is_tuple,{f,15},[{x,0}]}. > {test,test_arity,{f,15},[{x,0},2]}. > {get_tuple_element,{x,0},0,{x,1}}. > {get_tuple_element,{x,0},1,{x,2}}. > {test,is_eq_exact,{f,15},[{x,1},{atom,high_priority}]}. > {save_queue_pos,XXX}. > remove_message. > {move,{x,2},{x,0}}. > return. > {label,15}. > {loop_rec_end,{f,14}}. > {label,16}. > timeout. > {move,{atom,ok},{x,0}}. > In the above, XXX represents a value unique to the selective receive and > is generated an unspecified way. > > Care would need to be taken for dynamic code reloading. E.g. when a > module is reloaded, the VM might want to reset all the values previously > stored by save_queue_pos. > > Static vs. Non-Static selective receives > This technique will fail for receives that contain bound variables. e.g. > Ref = make_ref(), > Pid ! {command, Ref}, > receive > {some_rather_unique_response_atom, Ref} -> ... > after Timeout -> ... > end > This can not use the above technique since each time this runs, the > receive is matching against a different reference. > > It's possible to re-write the code to: > func1(Pid) -> > Ref = make_ref(), > Pid ! {command,Ref}, > func2(Ref,Pid). > > func2(Ref1,Pid) -> > % This is now a static pattern since Ref2 is unbound > receive > {some_rather_unique_response_atom, Ref2} -> > if > Ref1 == Ref2 -> > ...; > true -> > % We drop messages the we wouldn't have before > func2(Ref1,Pid) > end > after Timeout -> ... > end. > Actually, there's a better way of handling that particular case. One of the most common selective receive pattern is one that has a bound variable containing a freshly generated reference. Since, semantically, references are supposed to be unique, when doing a the receive, we can ignore all the messages that existed in our queue before the reference was created (there is no proper way that any of those messages could contain the reference). Note: it's techincally possible for an existing message to contain a reference that has not yet been created (e.g. if the reference was stored in mnesia and a node restarted), but we wouldn't want to match those anyway. This could be implemented with another new opcode "last_queue_pos" which sets "save" to "last". For example, this code: func(Pid) -> MonRef = erlang:monitor(process,Pid), Ref = make_ref(), Pid ! {request, Ref}, receive {reply, Ref} -> got_response; {'DOWN',MonRef,_,_,_} -> pid_is_gone after 1000 -> timeout end. could be implemented as this bytecode: {function, func, 1, 18}. {label,17}. {func_info,{atom,small},{atom,func},1}. {label,18}. {allocate_zero,2,1}. {move,{x,0},{x,1}}. {move,{atom,process},{x,0}}. {move,{x,1},{y,1}}. {'%live',2}. last_queue_pos. {call_ext,2,{extfunc,erlang,monitor,2}}. {move,{x,0},{y,0}}. {'%live',0}. {call_ext,0,{extfunc,erlang,make_ref,0}}. {test_heap,3,1}. {put_tuple,2,{x,1}}. {put,{atom,request}}. {put,{x,0}}. {move,{x,0},{x,2}}. {move,{y,1},{x,0}}. {move,{x,2},{y,1}}. {'%live',2}. send. {label,19}. {loop_rec,{f,23},{x,0}}. {test,is_tuple,{f,22},[{x,0}]}. {select_tuple_arity,{x,0},{f,22},{list,[2,{f,20},5,{f,21}]}}. {label,20}. {get_tuple_element,{x,0},0,{x,1}}. {get_tuple_element,{x,0},1,{x,2}}. {test,is_eq_exact,{f,22},[{x,1},{atom,reply}]}. {test,is_eq_exact,{f,22},[{x,2},{y,1}]}. remove_message. {move,{atom,got_response},{x,0}}. {deallocate,2}. return. {label,21}. {get_tuple_element,{x,0},0,{x,1}}. {get_tuple_element,{x,0},1,{x,2}}. {test,is_eq_exact,{f,22},[{x,1},{atom,'DOWN'}]}. {test,is_eq_exact,{f,22},[{x,2},{y,0}]}. remove_message. {move,{atom,pid_is_gone},{x,0}}. {deallocate,2}. return. {label,22}. {loop_rec_end,{f,19}}. {label,23}. {wait_timeout,{f,19},{integer,1000}}. timeout. {move,{atom,timeout},{x,0}}. {deallocate,2}. return. The compiler just needs to know which BIFs return new references (e.g. erlang:monitor/2 and make_ref/0), and detect if the results of these functions are used in *each* branch of a receive. If so, it could do a last_queue_pos before the BIF that returns the reference, so that the receive would start with the first message that arrived after the last_queue_pos was called. > Because we now drop certain messages, this is not exactly semantically > the same as the first version. However, in general, if a message does > match this pattern, it's most likely a response to a previous call on > which we timed out. The approriate action in that case would be to drop > the message. > > > > > > > - a simple language mechanism to allow function in-lining across > > module boundaries > > > > - a new mechanism for introducing local variables with a more cleanly > > defined semantics > > > > - a mini-language to allow the efficent implementation of low-level > > algorithms > > > > The paper can be downloaded here: > > > > > > > > > > Sven-Olof Nystr?m > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://www.erlang.org/mailman/listinfo/erlang-questions > From matthew.reilly@REDACTED Thu Jun 26 01:33:25 2008 From: matthew.reilly@REDACTED (Matthew Reilly) Date: Wed, 25 Jun 2008 16:33:25 -0700 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18529.883.586609.71953@hamberg.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> Message-ID: <1214436806.10157.366.camel@matt.sipphone.com> On Tue, 2008-06-24 at 16:23 +0200, Sven-Olof Nystr|m wrote: > Hello everyone, > > I have written down some thoughts and ideas on the future development > of Erlang. > > Main topics: > > - some notes on implementation and backward compatibility > > - an alternative to Erlang's selective receive In terms of Erlang's selective receive, it's possible to get around the O(N) look up issue w/o a language change (albeit with a slight change to the VM) if you restrict which patterns you use. (Review of the selective receive problem) The problem is that given code like: receive some_term -> some_value end. each time this receive is hit, the whole message queue is scanned. If there are N unmatched messages pending, then this take O(N) each time this bit of code is hit. However, if the receive pattern is static (i.e. it does not include an already bound variable), then these duplicate scans would be redundant -- we already know that if a message is not matched once, it won't match the same pattern in the future. The next time we hit the same receive, we'd only need to scan messages that have been added since we last did the same selective receive. (See below about dealing with non-static patterns). In this way, each message would be patterned matched at most once per selective receive, and thus, regardless of the size of the message queue, selective receives would become an amoritized O(1) task. The current message queue is implemented in the VM by a set of 3 pointers: "first", "save", "last". "first" points to the first item in the queue, "last" points to the place to add the next item, and "save" is a temp pointer used to traverse the message queue. A way to implement O(1) selective receives would be to add an (initially empty) list of saved {id,pointer} pairs to each message queue. When a selective receive finishes scans through the message queue, it stores a pointer to the next message would have processed (keyed to something that uniquely identifies this receive). When the same selective receives occurs again, it just needs to start at that saved location (instead of at the "first" message in the list). As an optimization, once the message queue becomes empty, this list can be wiped out. This adds one pointer's worth of space to each process. Very little overhead is added for the normal case where all receives have a default clause. This is a sketch of how the VM would need to be modified to have amortized O(1) selective receives. Currently a selective receive such as func() -> receive {high_priority, Message} -> Message after 0 -> no_message end. is implemented in bytecode such as: {label,14}. {loop_rec,{f,16},{x,0}}. {test,is_tuple,{f,15},[{x,0}]}. {test,test_arity,{f,15},[{x,0},2]}. {get_tuple_element,{x,0},0,{x,1}}. {get_tuple_element,{x,0},1,{x,2}}. {test,is_eq_exact,{f,15},[{x,1},{atom,high_priority}]}. remove_message. {move,{x,2},{x,0}}. return. {label,15}. {loop_rec_end,{f,14}}. {label,16}. timeout. {move,{atom,no_message},{x,0}}. Here the "loop_rec" opcode returns the message pointed to by the "save" pointer (which is normally the same as "first"). If the message queue is empty, it jumps to the "timeout" opcode which resets the "save" pointer to "first". If the message queue is not empty, the message is pattern matched against {high_priority,_}. If it doesn't match, it calls the opcode "loop_rec_end" when sets "save" to "save->next", and jumps back to the "loop_rec" opcode. If it does match, it returns the second item of the tuple, and calls the "remove_message" opcode which removes the message from the queue and also sets "save" to "first". Suppose we introduce two new opcodes: restore_queue_pos save_queue_pos save_queue_pos stores the current value of "save" indexed against some unique key. restore_queue_pos returns either the value saved by save_queue_pos (possibly modified by remove_message) or "first" if there was no saved value. The remove_message opcode needs be modified to update any saved queues positions (in case they point to the message that is being deleted) or remove them all if the queue becomes empty. The VM code for the same selected receive using these new opcodes: {restore_queue_pos,XXX}. {label,14}. {loop_rec,{f,16},{x,0}}. {test,is_tuple,{f,15},[{x,0}]}. {test,test_arity,{f,15},[{x,0},2]}. {get_tuple_element,{x,0},0,{x,1}}. {get_tuple_element,{x,0},1,{x,2}}. {test,is_eq_exact,{f,15},[{x,1},{atom,high_priority}]}. {save_queue_pos,XXX}. remove_message. {move,{x,2},{x,0}}. return. {label,15}. {loop_rec_end,{f,14}}. {label,16}. timeout. {move,{atom,ok},{x,0}}. In the above, XXX represents a value unique to the selective receive and is generated an unspecified way. Care would need to be taken for dynamic code reloading. E.g. when a module is reloaded, the VM might want to reset all the values previously stored by save_queue_pos. Static vs. Non-Static selective receives This technique will fail for receives that contain bound variables. e.g. Ref = make_ref(), Pid ! {command, Ref}, receive {some_rather_unique_response_atom, Ref} -> ... after Timeout -> ... end This can not use the above technique since each time this runs, the receive is matching against a different reference. It's possible to re-write the code to: func1(Pid) -> Ref = make_ref(), Pid ! {command,Ref}, func2(Ref,Pid). func2(Ref1,Pid) -> % This is now a static pattern since Ref2 is unbound receive {some_rather_unique_response_atom, Ref2} -> if Ref1 == Ref2 -> ...; true -> % We drop messages the we wouldn't have before func2(Ref1,Pid) end after Timeout -> ... end. Because we now drop certain messages, this is not exactly semantically the same as the first version. However, in general, if a message does match this pattern, it's most likely a response to a previous call on which we timed out. The approriate action in that case would be to drop the message. > > - a simple language mechanism to allow function in-lining across > module boundaries > > - a new mechanism for introducing local variables with a more cleanly > defined semantics > > - a mini-language to allow the efficent implementation of low-level > algorithms > > The paper can be downloaded here: > > > > > Sven-Olof Nystr?m > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From ok@REDACTED Fri Jun 27 06:18:47 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 27 Jun 2008 16:18:47 +1200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18531.60264.991238.276228@harpo.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> <18531.60264.991238.276228@harpo.it.uu.se> Message-ID: <51A4415F-9973-422B-B944-FBA8922D018F@cs.otago.ac.nz> On 27 Jun 2008, at 7:18 am, Sven-Olof Nystr|m wrote: >> nystrom_receive() % default channel >> nystrom_receive(Timeout) >> nystrom_receive_from(Channel) >> nystrom_receive_from(Channel, Timeout) >> nystrom_receive_from_any(Channel_Set) >> nystrom_receive_from_any(Channel_Set, Timeout) > > True. But they are all simpler than selective receive. True, but misleading. EACH of them is simpler than selective receive. Taken *together*, ALL of them are as complex; there are just too many separated facts to keep track of. This has always been my gripe about spawn; there are just too many spawn variants, making it hard to think about them. It is ridiculous that I have to write spawn(fun () -> < end) rather than a plain fork < end with some more syntax for options. And you have to ask WHAT is simpler. A hammer is a lot simpler in itself than a nail gun, but if you have a of nails to put in, it's a lot easier to USE a nail gun than a hammer. Smalltalk has had a SharedQueue class since 1980 or earlier. Given that, I can implement an owner-restricted receiving "channel" in about 10 minutes. It's easy to BUILD. But for actually writing anything that has anything but the most trivial protocol, given me Erlang's receive any day, as being easier to USE. >> But it gets worse. >> >> if channels are "objects", they can go anywhere, >> BUT THEY DON'T GET THERE WITHOUT BEING CARRIED! > > In Erlang today, if one wants to keep track of communication with a > particular process one needs to carry the pid. Yes, but with 'channels', you have to carry MORE things. Instead of putting complexity in one place (the 'receive'), this spreads complexity all around the program. > >> >> You have to explictly pass them around, especially loops. >> So to handle the simple bounded buffer you would find >> yourself writing stuff like this: >> >> buffer(Status, Contents, GetChan, PutChan) -> >> Channels = case Status >> of full -> [GetChan] >> ; empty -> [PutChan] >> ; _ -> [GetChan,PutChan] >> end, >> case nystrom_receive_from_any(Channels) >> of {GetChan,Who} -> >> {Status1,Contents1,Msg} = pop(Contents), >> Who ! Msg >> ; {PutChan,Msg} -> >> {Status1,Contents1} = add(Contents, Msg) >> end, >> buffer(Status1, Contents1, GetChan, PutChan). > > I've commented on your bounded_buffer example in my previous mail. I'm > not sure what you are doing here. It might help if you could show the > example using selective receive. The mail you say you have commented on included the selective receive version first. It's an absolutely standard bounded buffer that accepts 'get' requests only when the buffer is not empty and 'put' requests only when the buffer is not full. Because there are two kinds of requests, under your scheme there have to be two channels. > > >> >> That's assuming that multireceive returns a {Channel,Message} >> pair. Another approach would be to pass a list of {Channel, >> Handler} pairs, when the code would look like >> >> buffer(Status, Contents, GetChan, PutChan) -> >> GetHandler = fun (Who) -> >> {Status1,Contents1,Msg} = pop(Contents), >> Who ! Msg, >> buffer(Status1, Contents1, GetChan, PutChan) >> end, >> PutHandler = fun (Msg) -> >> {Status1,Contents1} = add(Contents, Msg), >> buffer(Status1, Contents1, GetChan, PutChan) >> end, >> nystrom_receive_from_any( >> case Status >> of full -> [{GetChan,GetHandler}] >> ; empty -> [{PutChan,PutHandler}] >> ; _ -> [{GetChan,GetHandler}, >> {PutChan,PutHandler}] >> end). >> >> How this is in any way simpler than a selective receive >> entirely escapes me. > > This example escapes me, too. That's extremely odd, because it is a simple question: If a process wants at some point to receive from any one of several channels, HOW DOES IT KNOW WHICH ONE IT GOT? It won't do to say "receive from THE channel and do a case test on the result" because the whole point is that we *need* to receive selectively: if you accept a 'get' request when there is nothing in the buffer, you have got to do something with it, and sending it back to yourself is obviously the wrong thing to do. So we have to say - if the buffer is empty, the only thing I will accept is a 'put'; - if the buffer is full, the only thing I will accept is a 'get'; - otherwise, I will accept either, but I need to know which one I got. So how do you know? Does the multi-receive function - accept a list of channels and + return the INDEX of the chosen channel, after which you have to call ne_receive on that yourself + return the CHANNEL that it chose, after which you have to call ne_receive on that yourself + return the MESSAGE it accepted, without any indication of which channel it came from (this would be a bad idea because you would have to redundantly tag messages sent to particular channels) + return a {Message,Index} pair? + return a {Message,Channel} pair? + do something else? - accept a list of {Channel,Handler} pairs and + return Handler(Channel)? + return Handler(Message)? + return Handler(Channel, Message) so that the same Handler can be used with more than one channel + do something else? Whichever one you pick, somebody will find it inconvenient and implement one of the others on top of it, and then the code will be harder to comprehend that code using a plain old 'receive'. Basically, you are creating complexity-in-USE in order to address a problem which it turns out you do not solve at all. I just don't see the point. > Let me just say that I find your bounded buffer example profoundly > unconvincing. The buffer may be bounded, but there is nothing to > prevent another process from filling the mailbox with "put" messages. That's not *MY* problem. That's *YOUR* problem. There is nothing in *YOUR* proposal to prevent that. > > > >> By the way, recall that the problem that this "simplification" >> is supposed to solve is this: "it is considered bad style to >> leave too many messages in the mailbox". Let me quote from the >> Concurrent ML documentation for the Mailbox structure: > > Well, that was never the main problem. Selective receive is complex > and unnecessary. I flatly deny both claims. If you understand Erlang pattern matching, selective receive is no more complex to *understand* than your ne_receive, and it is simpler to *use*. In fact, I would argue that it is precisely the 'receive' construct in Erlang that makes Erlang such a joy to use. If I wanted extreme terseness coupled with strong type checking, I know where to find Concurrent Haskell. If I wanted flexibility in constructing complex synchronisation schemes coupled with strong type checking and a truly amazing module system, I know where to find Concurrent ML. In many ways I find Erlang syntax clumsy and verbose these days. (Don't point to my Prolog book and say "it's rather like Prolog". I was never a fan of Prolog *syntax*.) But 'receive' makes up for it all. As for necessity, I have pointed out that Ada and Occam have essentially similar constructs, and for that matter, so does CSP. Take a look at section 5.4.4 of the Occam 2.1 manual. (I am quite aware of the differences between the Occam ALT CASE construt and Erlang's receive. I am drawing attention to the similarities.) > That was my main point. Programmers sometimes manage > to get into problems with it, but it was always my impression that > there were strategies to avoid these problems. I thought however, that > the fact that one had to make certain precautions to avoid getting > into trouble strengthened the argument against selective receive. But your proposal does absolutely nothing to make mailboxes easier or safer to use. It removes the convenience and clarity and gives nothing in return. From ok@REDACTED Fri Jun 27 06:51:27 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 27 Jun 2008 16:51:27 +1200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <1214436806.10157.366.camel@matt.sipphone.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <1214436806.10157.366.camel@matt.sipphone.com> Message-ID: On 26 Jun 2008, at 11:33 am, Matthew Reilly wrote: > In terms of Erlang's selective receive, it's possible to get around > the > O(N) look up issue w/o a language change (albeit with a slight > change to > the VM) if you restrict which patterns you use. > > (Review of the selective receive problem) > The problem is that given code like: > receive > some_term -> some_value > end. > each time this receive is hit, the whole message queue is scanned. If > there are N unmatched messages pending, then this take O(N) each time > this bit of code is hit. The idea of noting which receive was done last and restarting from where that left off has been around for a while. In fact we can do even better, again with a VM change. Here's the idea. Instead of the mailbox being a simple list, make it a doubly linked list with an extra thread, plus an array of pointers to messages. Hash each message using if message is a tuple of size > 0 then if first element is an atom then hash(atom) combined with size else size else if message is an atom then hash(atom) else 0 Given receive {key,......} -> end the compiler generates code to probe just the message thread that contains tuples the right size with the right key. With multiple cases, you have to merge several message threads. This is only a sketch. You wouldn't want to bother with elaborate machinery while the mailbox was small. Note that the scheme Matthew Reilly describes is quite general if you treat the matching process in two stages. Take a prescan where all the already bound variables in the patterns have been replaced by wild-cards, and any guard tests that now contain wild-cards have been discarded (but don't throw away type tests for variables to be bound by the match). If you remember where the prescan got up to, it doesn't matter what the bound variables are, they can't possibly cause things that failed to match the prescan to match instead. So the code becomes something like if the last receive was this one, P := the stored prescan position else P := the start of the queue while P is not empty and the message at P does not match the prescan do advance P to the next message Record that the last receive was this one and that prescanning reached P. Contiune as now. No restriction on patterns is actually needed. From matthias@REDACTED Fri Jun 27 07:44:06 2008 From: matthias@REDACTED (Matthias Lang) Date: Fri, 27 Jun 2008 07:44:06 +0200 Subject: [erlang-questions] Message Ordering and Reliability in Erlang In-Reply-To: <909c265f0806261647o62ef8d65xadfc0631d37573e5@mail.gmail.com> References: <909c265f0806261647o62ef8d65xadfc0631d37573e5@mail.gmail.com> Message-ID: <20080627054406.GA3682@contorpis.lisalinda.com> On Fri, Jun 27, 2008 at 11:47:58AM +1200, Fuad Tabba wrote: > I was wondering, when sending messages in Erlang, is ordering preserved? > i.e. Process A sends 1 then 2 to B; can I count on B receiving 1 before > receiving 2 (not talking about selective receive). >From the FAQ: 10.9 Is the order of message reception guaranteed? Yes, but only within one process. If there is a live process and you send it message A and then message B, it's guaranteed that if message B arrived, message A arrived before it. On the other hand, imagine processes P, Q and R. P sends message A to Q, and then message B to R. There is no guarantee that A arrives before B. (Distributed Erlang would have a pretty tough time if this was required!) > What about reliability? Assuming that there aren't faulty processes or > broken links (slow is ok), is it possible for some messages to be dropped by > others not? To use the same example, A send 1 then 2, is it possible for B > to only receive 2? Covered in the next question in the FAQ. Start reading here: http://www.erlang.org/faq/academic.html#10.9 Matt From fuad@REDACTED Fri Jun 27 07:49:32 2008 From: fuad@REDACTED (Fuad Tabba) Date: Fri, 27 Jun 2008 17:49:32 +1200 Subject: [erlang-questions] Message Ordering and Reliability in Erlang In-Reply-To: <20080627054406.GA3682@contorpis.lisalinda.com> References: <909c265f0806261647o62ef8d65xadfc0631d37573e5@mail.gmail.com> <20080627054406.GA3682@contorpis.lisalinda.com> Message-ID: <909c265f0806262249j419b3245u1ee7032d2bccf32a@mail.gmail.com> Thanks Matt, I feel a bit silly now for not having read the FAQ :) relying on google a bit too much... On Fri, Jun 27, 2008 at 5:44 PM, Matthias Lang wrote: > On Fri, Jun 27, 2008 at 11:47:58AM +1200, Fuad Tabba wrote: > > I was wondering, when sending messages in Erlang, is ordering preserved? > > i.e. Process A sends 1 then 2 to B; can I count on B receiving 1 before > > receiving 2 (not talking about selective receive). > > From the FAQ: > > 10.9 Is the order of message reception guaranteed? > > Yes, but only within one process. > > If there is a live process and you send it message A and then message > B, it's guaranteed that if message B arrived, message A arrived before > it. > > On the other hand, imagine processes P, Q and R. P sends message A to > Q, and then message B to R. There is no guarantee that A arrives > before B. (Distributed Erlang would have a pretty tough time if this > was required!) > > > What about reliability? Assuming that there aren't faulty processes or > > broken links (slow is ok), is it possible for some messages to be dropped > by > > others not? To use the same example, A send 1 then 2, is it possible for > B > > to only receive 2? > > Covered in the next question in the FAQ. Start reading here: > > http://www.erlang.org/faq/academic.html#10.9 > > Matt > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Fri Jun 27 08:57:45 2008 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Fri, 27 Jun 2008 08:57:45 +0200 Subject: [erlang-questions] How is hostname decided In-Reply-To: References: Message-ID: <20080627065745.GA21939@erix.ericsson.se> On Thu, Jun 26, 2008 at 05:36:45PM -0500, Anders Nygren wrote: > When using short names, how is the hostname decided. > The reason I am asking is that I have a system based on R12B2 > running on RedHat Enterprise 4 > > [npas@REDACTED lib]$ /opt/npas/bin/erl -sname test > Erlang (BEAM) emulator version 5.6.2 [source] [64-bit] [smp:2] > [async-threads:0] [hipe] [kernel-poll:false] > > Eshell V5.6.2 (abort with ^G) > (test@REDACTED)1> inet:gethostname(). > {ok,"npas"} > > Why is the nodename localhost, when inet:gethostname() returns npas? > > Looking through the sources I found what looks like the place where hostname is > determined. > inet_config:set_hostname() -> > case inet_udp:open(0,[]) of > {ok,U} -> > Res = inet:gethostname(U), > inet_udp:close(U), > set_hostname(Res); > _ -> > set_hostname({ok, []}) > end. > > But that also gives "npas" Further down in inet_config:init/0 comes a call to standalone_host/0, and that might do tricks. It ensures the hostname is resolvable. What does the following OS commands tell you? $ hostname -s $ hostname -f $ getent hosts npas $ cat /etc/hosts $ cat /etc/resolv.conf $ cat /etc/nsswitch.conf And what does the following Erlang commands tell you? 1> ets:tab2list(inet_db). 2> inet:gethostbyname("npas"). You probably have a somewhat strange resolver situation. Freshly installed modern Linux machines often has. We ought to write an FAQ entry about Elang/OTP's desires about the host's resolver configuration esp. for long node names. Note that you can force a node name like this: $ /opt/npas/bin/erl -sname test@REDACTED > > /Anders > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From francesco@REDACTED Fri Jun 27 09:39:18 2008 From: francesco@REDACTED (Francesco Cesarini) Date: Fri, 27 Jun 2008 08:39:18 +0100 Subject: [erlang-questions] Erlang eXchange Streaming day 2 Message-ID: <48649926.6090803@erlang-consulting.com> Hi All, Even if experimental, it looks like the streaming went really well yesterday. Yaika, who are on location helping us out and managing the streaming, implemented improvements as we went along. Hopefully, today it should be even better. You can view the streaming at www.erlang-exchange.com and if you create a Yaika account, you can ask questions and let us have feedback. The speakers we will be streaming include (All times are UK times): 9.15, Keynote: Enterprise Integration, Steve Vinoski 10.00, Erlang & Tail-F, Klacke Wikstr?m 11.15, Presenting RabbitMQ: An Erlang Based Implementation of AMQP, Matthias Radestock 12.00, ejabberd for web 2.0 development, Mickael Remond 13.45 Building a transactional distributed data store with Erlang, Alexander Reinefeld 14.30 Couch DB at 10,000 feet, Jan Lehnardt 15.45 Using Jinterface to Bridge Erlang and Java, Dennis Byrne 17.15 Erlang Enterprise Integration Panel Discussion, Garry Bulmer 18.00 Erlang eXchange Wrap-Up & Raffle A review of yesterday is available at http://planet.trapexit.org/ Francesco -- http://www.erlang-consulting.com From vychodil.hynek@REDACTED Fri Jun 27 10:26:22 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Fri, 27 Jun 2008 10:26:22 +0200 Subject: [erlang-questions] Writing a Concurrent Binary Search Tree Data Structure in Erlang In-Reply-To: <909c265f0806261354m7e3f711fk8ce3cafd0500fd3b@mail.gmail.com> References: <909c265f0806250222q76e6651emd164ea221f15202f@mail.gmail.com> <4d08db370806250240gbf0e527n798a296292124ef8@mail.gmail.com> <909c265f0806250246s7abbcb2at6884f50e31c5d81c@mail.gmail.com> <4d08db370806250257h63fa7cb7m2dbf1a8b682768e2@mail.gmail.com> <909c265f0806251510x60ab819aqfb074b5558d7fc46@mail.gmail.com> <909c265f0806261354m7e3f711fk8ce3cafd0500fd3b@mail.gmail.com> Message-ID: <4d08db370806270126x472ded9du60576500b3f38366@mail.gmail.com> If you would like minimise overhead and or maximise performance you should use B-trees instead binary trees and you should benchmark and tune your implementation, but idea is same. What is worse, I suggest, best solution will depend of number of cores (shedulers) and load character. On Thu, Jun 26, 2008 at 10:54 PM, Fuad Tabba wrote: > Cool, that's matches my understanding of Hynek's suggestion as well. Thanks > Kevin. > > I'm actually writing code along similar lines to test out the idea. It > seems rather simple and elegant; what I don't like though is the added > overhead of a process per node - I know that Erlang processes are light, but > if I recall correctly from reading in Joe's book, each process has memory > overhead in the order of 100 bytes. If the the data per node is large then > it's not a big deal, but if the amount of data stored per node is small then > that could be some serious overhead. > > Implementing delete() is going to be a bit tricky, need to be wary of > deadlocks; but this isn't an Erlang issue, this is a general issue with > writing a concurrent and shared data structure. > > Thanks again! > > Cheers, > /Fuad > > On Fri, Jun 27, 2008 at 8:22 AM, Kevin Scaldeferri > wrote: > >> >> On Jun 25, 2008, at 3:10 PM, Fuad Tabba wrote: >> >> I think I understand what you're saying here; but I still can't wrap my >>> head around putting that into code. Is there similar code that you know of >>> available online? The structure doesn't have to be a BST, just anything that >>> demonstrates the idea of having a structure that stores lots persistent data >>> to be viewed/modified by many processes and is scalable... >>> >>> >>> On Wed, Jun 25, 2008 at 9:40 PM, Hynek Vychodil < >>> vychodil.hynek@REDACTED> wrote: >>> Because there is no shared state between processes in Erlang, I would >>> start think about process per tree node, not process per action as usual in >>> classic shared memory (for example C) implementations. >>> I know, it sounds weird, but may be help you :-) >>> >> >> >> >> Here's a basic outline of this idea, if I understand it right. delete() >> is not implemented, and it has some other deficiencies, but I hope it at >> least starts to demonstrate the idea of a recursive data structure >> represented as a collection of processes. >> >> >> -kevin >> >> %%%%%%%%%%%%%%%%%%%%%%%%% >> >> -module(cbst). >> -export([new/1, exists/2, insert/2, delete/2]). >> >> -record(node, {key, >> left=null, >> right=null}). >> >> %% >> %% public API >> %% >> >> new(Key) -> >> spawn(fun() -> create(Key) end). >> >> exists(Node, Key) -> >> Node ! {exists, Key, self()}, >> receive >> true -> true; >> false -> false >> end. >> >> insert(Node, Key) -> >> Node ! {insert, Key}. >> >> delete(Node, Key) -> >> Node ! {delete, Key}. >> >> >> %% >> %% private functions >> %% >> >> create(Key) -> >> Node = #node{key=Key}, >> loop(Node). >> >> loop(Node) -> >> receive >> {exists, Key, Caller}=Q -> >> case Node#node.key of >> Key -> >> Caller ! true; >> This when Key < This, Node#node.left =/= null -> >> Node#node.left ! Q; >> This when Key > This, Node#node.right =/= null -> >> Node#node.right ! Q; >> _ -> >> Caller ! false >> end, >> loop(Node); >> {insert, Key} -> >> case Node#node.key of >> Key -> >> loop(Node); % already in tree >> This when Key < This -> >> case Node#node.left of >> null -> >> Left = new(Key), >> loop(Node#node{left=Left}); >> Left -> >> Left ! {insert, Key}, >> loop(Node) >> end; >> This when Key > This -> >> case Node#node.right of >> null -> >> Right = new(Key), >> loop(Node#node{right=Right}); >> Right -> >> Right ! {insert, Key}, >> loop(Node) >> end >> end; >> {delete, Key} -> >> % left as an exercise for the reader >> loop(Node) >> end. >> >> > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From nem@REDACTED Fri Jun 27 10:32:59 2008 From: nem@REDACTED (Geoff Cant) Date: Fri, 27 Jun 2008 10:32:59 +0200 Subject: [erlang-questions] When is code_change triggered? In-Reply-To: (Paul Mineiro's message of "Thu, 26 Jun 2008 09:43:36 -0700 (PDT)") References: <84fb38e30806260905qb6426f2l26e63888a294d937@mail.gmail.com> Message-ID: Hi all, I puzzled about this one for a while too before stumbling onto the 'sys' module. gen_(server|fsm|event) code_change/3 callbacks are invoked by a corresponding call to sys:change_code/4. The small caveat being that the process in question must be suspended (it goes into a loop where it only processes other 'sys' messages). So to do code upgrade of the only gen_server in a system (Pid) running the callback module (Mod), you want something like: Upgrade = fun (Pid, Mod) -> c:l(Mod), sys:suspend(Pid), sys:change_code(Pid, Mod, foo, foo), sys:resume(Pid), code:purge(Mod) end. Extending that to all the Pids running a callback mod is pretty easy too. The 'foo' dummy arguments are the version number we're upgrading from and the extra information we need for the upgrade. Hope this helps, --Geoff Paul Mineiro writes: > We had the same at question here in Hazzard County several months ago. > > Hot code loading is a low-level feature built into the runtime; there's > space for two versions of a module and using module-prefixed calls implies > a call into the latest version (and a local call implies continue to use > the same module version). Before you hot code load a module, you have to > first nuke the oldest version of the module (and any lingering processes). > This is the purpose of code:purge/1 and code:soft_purge/1. You can then > load a new module using code:load_file/1, code:load_abs/1, > code:load_binary/3, etc. After loading the formerly newest version of the > code will now occupy the oldest version slot. > > Code change is a high-level feature which is implemented by convention in > the OTP libraries. The gen_X family has an inner loop which responds to > special "system messages" which cause the process to transfer control to a > seperate receive loop which only processes special upgrade messages, which > ultimately cause the code_change/3 callback to be called. In this manner > you will not process further normal gen_X messages until the new code has > been loaded and the code_change handler called. This sequence is > orchestrated by release_handler via calls like > release_handler:install_release/1 and release_handler:upgrade_app/2. > > The final piece of information is that calling c(Module) in the shell is > shorthand for invoking the compiler and then invoking the various code > methods for loading the new version of the module. It is *not*, however, > shorthand for invoking the release_handler; that wouldn't make sense > for many reasons. (The shortest path to seeing your code_change/3 handler > called is, imho, rather long). > > Hope that helps, > > -- p > > p.z. after figuring this out we wrote erlrc: > http://code.google.com/p/erlrc/ > > On Thu, 26 Jun 2008, tsuraan wrote: > >> When I started playing with gen_server a few years ago, I recalled >> that doing a c(module) in the REPL would immediately update the code >> of my running servers. Now, I cannot get that to happen, so I'm >> trying to figure out what I'm doing wrong. My test server is minimal: >> >> -module(test). >> -export([start_link/0, start/0]). >> -export([init/1, handle_call/3, handle_cast/2, handle_info/2, >> terminate/2, code_change/3]). >> >> start_link() -> >> gen_server:start_link({ local, ?MODULE }, ?MODULE, [], []). >> >> start() -> >> gen_server:start({ local, ?MODULE }, ?MODULE, [], []). >> >> init([]) -> >> process_flag(trap_exit, true), >> { ok, {}}. >> >> handle_call(Msg, From, State) -> >> io:format("Unexpected Call ~p From ~p~n", [Msg, From]), >> { reply, { error, badcall }, State }. >> >> handle_cast(Msg, State) -> >> io:format("Unexpected Cast ~p~n", [Msg]), >> { noreply, State }. >> >> handle_info(Msg, State) -> >> io:format("Unexpected Info ~p~n", [Msg]), >> { noreply, State }. >> >> terminate(Reason, _State) -> >> io:format("Terminating ~p for reason ~p~n", [ self(), Reason ]), >> ok. >> >> code_change(OldVsn, State, Extra) -> >> io:format("Upgrading from ~p with extra data ~p~n", [OldVsn, Extra]), >> { ok, State }. >> >> I can run this with test:start(), but subsequent c(test) never print >> the message "Upgrading from ...". Can anyone tell what I'm doing >> wrong, or is my memory just bad and c(module) has never caused running >> instances of that module to be upgraded? >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions From chsu79@REDACTED Fri Jun 27 11:03:53 2008 From: chsu79@REDACTED (Christian S) Date: Fri, 27 Jun 2008 11:03:53 +0200 Subject: [erlang-questions] Erlang eXchange Streaming day 2 In-Reply-To: <48649926.6090803@erlang-consulting.com> References: <48649926.6090803@erlang-consulting.com> Message-ID: > Even if experimental, it looks like the streaming went really well > yesterday. Yaika, who are on location helping us out and managing the > streaming, implemented improvements as we went along. Hopefully, today > it should be even better. Will the streams be saved and made available for download? From francesco@REDACTED Fri Jun 27 11:21:13 2008 From: francesco@REDACTED (Francesco Cesarini) Date: Fri, 27 Jun 2008 10:21:13 +0100 Subject: [erlang-questions] Erlang eXchange Streaming day 2 In-Reply-To: References: <48649926.6090803@erlang-consulting.com> Message-ID: <4864B109.8050308@erlang-consulting.com> We are recording the event with video cameras (the streaming is with web cams) and will put better quality recordings online at the Erlang eXchange site with the presentations. Francesco -- http://www.erlang-consulting.com Christian S wrote: >> Even if experimental, it looks like the streaming went really well >> yesterday. Yaika, who are on location helping us out and managing the >> streaming, implemented improvements as we went along. Hopefully, today >> it should be even better. >> > > Will the streams be saved and made available for download? > > From mats.cronqvist@REDACTED Fri Jun 27 11:22:24 2008 From: mats.cronqvist@REDACTED (Mats Cronqvist) Date: Fri, 27 Jun 2008 11:22:24 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18531.60264.991238.276228@harpo.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> <18531.60264.991238.276228@harpo.it.uu.se> Message-ID: <4864B150.7060303@gmail.com> Sven-Olof Nystr|m wrote: > Well, that was never the main problem. Selective receive is complex > and unnecessary. > over the years i've worked with perhaps 300 erlang programmers, pretty much all of which have been less accomplished computer scientists than Sven-Olof. and there are many things in Erlang that they've found complex, unnecessary, or both. the syntax, higher order functions, list comprehensions, recursion, bignums... pretty much everything in Erlang/OTP. everything, that is, except two things; selective receive and hot code loading. i can't think of anyone that has found these two things complex and unnecessary. now Sven-Olof is obviously entitled to his opinion about the meaning of complexity. but i think it's a fact that there is no great demand for a change in semantics of selective receive. mats From svenolof@REDACTED Fri Jun 27 11:26:16 2008 From: svenolof@REDACTED (Sven-Olof Nystr|m) Date: Fri, 27 Jun 2008 11:26:16 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <51A4415F-9973-422B-B944-FBA8922D018F@cs.otago.ac.nz> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> <18531.60264.991238.276228@harpo.it.uu.se> <51A4415F-9973-422B-B944-FBA8922D018F@cs.otago.ac.nz> Message-ID: <18532.45624.17663.666648@hamberg.it.uu.se> Richard A. O'Keefe writes: > The mail you say you have commented on included the > selective receive version first. It's an absolutely standard > bounded buffer that accepts 'get' requests only when the buffer > is not empty and 'put' requests only when the buffer is not full. > Because there are two kinds of requests, under your scheme there > have to be two channels. I'll focus on your bounded buffer example, since the other points in your argumentation seem to be based on it. As far as I can tell, your line of argument is: 1. The bounded buffer example is a useful and relevant example of Erlang programming. 2. The bounded buffer example is difficult or impossible to express using channels. Since I disagree on point 1, I don't see much reason to discuss your other points. Let's take a second look at your buffer example: > buffer(Status, Contents) -> > receive > {put,Msg} when Status /= full -> > {Status1, Contents1} = add(Contents, Msg) > ; {get,Who} when Status /= empty -> > {Status1, Contents1, Msg} = pop(Contents), > Who ! Msg > end, > buffer(Status1, Contents1). At first, I thought it looked pretty elegant, including the somewhat unexpected placement of the semicolon. However, a moment's reflection changed my opinion. Now, I can see why one would want to write something similar in a language with synchronous communication such as Ada. - There is no implicit buffering, so buffering has to be implemented. - There is no dynamic allocation of memory (maybe that's changed?), so buffers must be bounded. - Synchronous communication implies that if the receiver refuses to accept a message, the sender will be suspended. Thus, when the buffer is full, senders will have to wait until there is room in the buffer. In Erlang, none of these things are true. - there is already a mechanism for buffering (mailboxes). No need to implement a new one. - dynamic allocation is available - communication is asynchronous. Granted, one can simulate synchronous communication, but I don't see you doing that here. As I said in my previous reply, you haven't really implemented bounded buffers: > When the buffer is full, incoming 'put' messages will be kept in > the mailbox instead of stored in the buffer, so they will still > require approximately the same amount of memory on the heap. For an > outside observer, there will be no difference between this buffer > and one that uses an unbounded data structure, except that it is > easier to overflow the mailbox of the bounded buffer. Now, it might perhaps be interesting to compare how selective receive and channels can be used to implement either a) a correct implementation of bounded buffers or b) an implementation of unbounded buffers. If you supply one using selective receive, I'll try to give the other. If you still insist that the above is a correct implementation of bounded buffers, I suggest that you show a context where the "bounded buffer" behaves differently from an unbounded one. Sven-Olof From nicolas@REDACTED Fri Jun 27 12:42:32 2008 From: nicolas@REDACTED (Nicolas Niclausse) Date: Fri, 27 Jun 2008 12:42:32 +0200 Subject: [erlang-questions] Slave nodes without proxied I/O? In-Reply-To: <48613BEA.7060301@charpi.net> References: <73AF1508-8DE3-437F-8208-521C84A35683@ketralnis.com> <4860E042.8050803@niclux.org> <48613BEA.7060301@charpi.net> Message-ID: <4864C418.4020604@niclux.org> Nicolas Charpentier ecrivait le 24.06.2008 20:24: > Nicolas Niclausse wrote: >> David King ecrivait le 01.05.2008 20:34: >>>> By default, nodes started as slave nodes with slave:start/1 have their >>>> I/O proxied through the master node that started them. (That is, I/O >>>> operations done with the 'file' module are done on the master's >>>> filesystem, not the slaves'). >>>> Is there any way to turn this off? > >> I'm having the same problem as you: i want to start a slave without >> proxied I/O. >> >> After some digging, i discover that you only have to remove the >> "-master node()" argument in the mk_cmd fun to do that (slave.erl >> module). > I might be wrong but I have always thought that I/O was proxied because > they (master and slave nodes) was sharing the same group_leader. > If I'm right, we might be able to change the group_leader of processes > on the slave node. Well, yes, the group_leader of all the process on the slave must be a process on the first node, i guess; but how can i change this: which pid should be the new group_leader on the slave node ? I would be simpler to have an option to slave:start to choose the behavior of I/O on slave nodes, imho. -- Nicolas From ulf@REDACTED Fri Jun 27 13:23:08 2008 From: ulf@REDACTED (Ulf Wiger) Date: Fri, 27 Jun 2008 13:23:08 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <4863DBFA.6000401@san.rr.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <4863B777.908@san.rr.com> <8209f740806260938o6cb373e9v7a33706c76b7d846@mail.gmail.com> <4863D4DD.8020509@san.rr.com> <8209f740806261052g56ed1f94lad2f1d0fb073ed2b@mail.gmail.com> <4863DBFA.6000401@san.rr.com> Message-ID: <8209f740806270423n2ef90742wa22a8262c8dafe98@mail.gmail.com> 2008/6/26 Darren New : I guess I should clarify my initial comment: > Ulf Wiger wrote: >> 2008/6/26 Darren New : >>> Ulf Wiger wrote: >>>> I think we can stipulate that when serializing a pid, perhaps storing it >>>> on disk, then re-creating it and trying to use it, all bets are off >>>> anyway.* What I meant was that if we're discussing weird things that could happen if we'd allow automatic GC of a process that's blocked, not registered, and were there are no known references to the pid, we shouldn't necessarily pay too much attention to the cases were the pid has been serialized e.g. using term_to_binary(), because this is not really different from the problems that can occur already. We must always allow for the even that the process dies. If it dies (for whatever reason), and there are no references to the pid, it is ok for the runtime system to reuse the pid. Thus, using a remote pid or a pid that has been serialized, there is always a (very slight) chance that it will now refer either to a non-existing process, or some other process entirely. Now bearing in mind that we often have distributed systems, and no distributed GC, it is of course an excellent idea to not reuse pids immediately. >> In that case, the remote holder of the pid cannot rely on the pid referring >> to the same process if it's used some time later. > > How long is "some time"? This is implementation and configuration dependent, as well as dependent on "churn". Using the -P flag, the user tells the runtime system how much memory to allocate for processes. I'm not sure whether this also affects how soon pids are reused, or whether the entire address space for pids is always used. Either way, it may affect how soon pids are reused, but the time ought to be long enough that it should not be a problem for all recommended uses of pids. > We're conflating "reusing the PID" and "GCing a process which we don't > think anyone will ever wake up". If I GC the process as soon as there's > no local or remote reference to the PID, I might wind up GCing the > process while the data representing the PID is on the wire on its way to > the node going to use it. As ROK wrote, the suggestion wasn't made seriously to begin with, but as with any form of GC, one would of course have to come up with a reasonable way of determining when it is safe to remove the process. This would have to take into account normal use of pids in a distributed environment. Personally, I don't think it would be feasible to introduce such semantics anyway, since it would rarely be useful, and might sometimes cause very strange behaviour. >> It should monitor the process in order to detect whether it dies. > > Again, not what I'm talking about. Erlang gives me ways to mostly do > this unreliably. (If Erlang had a reliable way to do it, we wouldn't > need mnesia:set_master_nodes, for example.) I don't follow your reasoning here. Erlang offers a reliable way of learning whether a process is no longer reachable. The concept of mnesia master nodes is meant for a situation where full analysis of the problem cannot be performed in a generic way. If Erlang is able to tell /why/ a process is no longer reachable, it will provide that info, but there are obviously cases where it simply isn't possible to know why. In these cases, it can only tell you that the other node is not responding (for whatever reason), and thus all its processes are unreachable. Mnesia will indicate if it detects that there has been a network partitioning, but it simply cannot resolve the situation for you. It's a pathological case for which there may be acceptable application-specific solutions. That's why there's a facility to set master nodes in mnesia. There are lots of situations like that in distributed processing where the runtime system cannot possibly give you guarantees or resolve the situation automatically. Besides, erlang processes are usually part of a supervision hierarchy, which means that there will be at least one stable reference to each process (the monitoring from its closest supervisor). > > but once a process /has/ died, and all known (local) references are >> gone, some other process may reuse that pid. This is why storing pids >> persistently is a very bad idea. > > Except that storing pids "persistently" in variables is how one uses > them. No matter where you store them, in variables or in disk files, you > have to get rid of them when you get a monitor or link message saying > the process exited, and you have to deal with the fact that you might > get those messages when the process has, indeed, not yet exited and > indeed doesn't know those messages have been sent. I'm not sure that > storing a pid in a table is particularly more difficult than storing a > pid in the variables of a process that's not expected to ever exit. I disagree. To take a related example, mnesia offers different table types with different persistence requirements. A very good rule is to avoid mixing data elements with different persistence requirements in the same table. In remotely managed systems (e.g. via SNMP), you may well keep a "current alarms" table in mnesia. This table should be created such that its content doesn't survive a system restart, for the simple reason that all alarms will be re-generated after such an event, and the old alarms are useless anyway. To further the analogy, a monitoring system must also know that if the managed system restarts, all references to current alarms can, and must be, thrown away. BR, Ulf W From svenolof@REDACTED Fri Jun 27 13:49:04 2008 From: svenolof@REDACTED (Sven-Olof Nystr|m) Date: Fri, 27 Jun 2008 13:49:04 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <4864B150.7060303@gmail.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> <18531.60264.991238.276228@harpo.it.uu.se> <4864B150.7060303@gmail.com> Message-ID: <18532.54192.651817.522964@hamberg.it.uu.se> Mats Cronqvist writes: > > Selective receive is complex and unnecessary. > > > over the years i've worked with perhaps 300 erlang programmers, > pretty much all of which have been less accomplished computer scientists > than Sven-Olof. and there are many things in Erlang that they've found > complex, unnecessary, or both. the syntax, higher order functions, list > comprehensions, recursion, bignums... pretty much everything in Erlang/OTP. > everything, that is, except two things; selective receive and hot code > loading. i can't think of anyone that has found these two things complex > and unnecessary. I guess we'll have to differ on that point. I've met many people, who even after having quite a bit of experience with Erlang had trouble answering simple questions such as: if a receive matches many patterns, say receive {foo, X} -> ... ; {bar, Y} -> ... end and there are messages in the mailbox matching both patterns, which clause and which message is selected? Then there is the issue with guards, of course. Now, even with selective receive Erlang's process communication is far easier to use than most models for process communiaction programmers are likely to come in contact with, so I'm not surprised that they are pleased with Erlang's selective receive. > now Sven-Olof is obviously entitled to his opinion thanks :-) sven-olof From ulf@REDACTED Fri Jun 27 14:53:19 2008 From: ulf@REDACTED (Ulf Wiger) Date: Fri, 27 Jun 2008 14:53:19 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18531.49944.89467.105800@harpo.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> <18531.49944.89467.105800@harpo.it.uu.se> Message-ID: <8209f740806270553h56328293o463bda051ffce32d@mail.gmail.com> 2008/6/26 Sven-Olof Nystr|m : > Richard A. O'Keefe writes: > > > The thing that strikes me forcibly is that I have seen > > all this before. John Reppy's Concurrent ML. It's > > supported in SML/NJ and Mlton and perhaps other ML > > implementations. What Nystrom calls a "channel" is > > what CML calls a "mailbox". (What CML calls a "channel" > > is what Occam calls a "channel".) And comparing the CML > > code I've seen to Erlang makes Erlang look stunningly > > simple *in use*. > > True, many languages have a channel concept. > > In my opinion, one great advantage of Erlang over other languages that > use channels is that in Erlang, each process has a "standard" channel > that it normally takes input from. I hope that I was not unclear about > this, but that is a feature of Erlang that I had not intended to change. But if one of the reasons for wanting channels rather than selective receive on a single mailbox is to make it easier to reason about concurrency, wouldn't keeping an "unscoped" default channel complicate this reasoning? It's a bit like saying "global variables are bad, so we'll introduce local variables - but global variables are great, so you can have them too." BR, Ulf W From svenolof@REDACTED Fri Jun 27 15:33:54 2008 From: svenolof@REDACTED (Sven-Olof Nystr|m) Date: Fri, 27 Jun 2008 15:33:54 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <8209f740806270553h56328293o463bda051ffce32d@mail.gmail.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <18531.49944.89467.105800@harpo.it.uu.se> <8209f740806270553h56328293o463bda051ffce32d@mail.gmail.com> Message-ID: <18532.60482.6969.182774@hamberg.it.uu.se> Ulf Wiger writes: > 2008/6/26 Sven-Olof Nystr|m : > > In my opinion, one great advantage of Erlang over other languages that > > use channels is that in Erlang, each process has a "standard" channel > > that it normally takes input from. I hope that I was not unclear about > > this, but that is a feature of Erlang that I had not intended to change. > > But if one of the reasons for wanting channels rather than selective > receive on a single mailbox is to make it easier to reason about > concurrency, wouldn't keeping an "unscoped" default channel > complicate this reasoning? Maybe. It would certainly be nice to know that if we call a function in another module it cannot affect any of the channels of the current process. Thus, there would be a clear distinction between functional code and code that is involved in process communication. OTOH, in Erlang there is a one-to-one relationship between processes and mailboxes which often very natural. There is a similarity between 'self()' in Erlang and referring to the current object in object-oriented languages (some oo languages actually use the keyword 'self'). I also suspect that it would be inconvenient to add the current channel as an argument to all functions that perform process communication. > It's a bit like saying "global variables are bad, so we'll introduce > local variables - but global variables are great, so you can have > them too." Only if you think that mailboxes are like global variables :-) Sven-Olof From ulf@REDACTED Fri Jun 27 15:52:05 2008 From: ulf@REDACTED (Ulf Wiger) Date: Fri, 27 Jun 2008 15:52:05 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18532.60482.6969.182774@hamberg.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> <18531.49944.89467.105800@harpo.it.uu.se> <8209f740806270553h56328293o463bda051ffce32d@mail.gmail.com> <18532.60482.6969.182774@hamberg.it.uu.se> Message-ID: <8209f740806270652n6120b210q3755399d23f42f6b@mail.gmail.com> 2008/6/27 Sven-Olof Nystr|m : > Ulf Wiger writes: > > > It's a bit like saying "global variables are bad, so we'll introduce > > local variables - but global variables are great, so you can have > > them too." > > Only if you think that mailboxes are like global variables :-) Well, in a sense they are. (: One of the nice aspects of channels is that you cannot consume a message from a channel unless you have its handle. This means that you can be sure that a function that has not been passed a channel reference cannot remove messages from it. The default channel, however, is available to function executing in the process, so there is no explicit control of scope. BR, Ulf W From svenolof@REDACTED Fri Jun 27 16:08:19 2008 From: svenolof@REDACTED (Sven-Olof Nystr|m) Date: Fri, 27 Jun 2008 16:08:19 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18532.45624.17663.666648@hamberg.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> <18531.60264.991238.276228@harpo.it.uu.se> <51A4415F-9973-422B-B944-FBA8922D018F@cs.otago.ac.nz> <18532.45624.17663.666648@hamberg.it.uu.se> Message-ID: <18532.62547.68630.333358@hamberg.it.uu.se> I need to correct myself on one point. Those of you with no interest in Ada may want to skip this message. Sven-Olof Nystr|m writes: > Now, I can see why one would want to write something similar in a > language with synchronous communication such as Ada. [...] > - There is no dynamic allocation of memory (maybe that's changed?), > so buffers must be bounded. Even Ada'83 had dynamic memory allocation. See for example . Still, I can see good reasons for avoiding dynamic allocation of memory in typical Ada applications. Sven-Olof From devdoer2@REDACTED Fri Jun 27 18:37:25 2008 From: devdoer2@REDACTED (devdoer bird) Date: Sat, 28 Jun 2008 00:37:25 +0800 Subject: [erlang-questions] why write operation in mnesia is so slow when table fragment is in progress Message-ID: HI, I have a mensia user table ,which carry about 800,000 records .The table is distributed on 2 nodes and 4 fragments ,each fragment has 2 replications. IWhile new fragment adding is in progress in one node,the writing operation on the other node is very slow. Writing 5 records cost almost 2 mintues. Why is it so slow? How do you guys use mnesia fragment if the write opertaion is slow? -------------- next part -------------- An HTML attachment was scrubbed... URL: From devdoer2@REDACTED Fri Jun 27 18:39:34 2008 From: devdoer2@REDACTED (devdoer bird) Date: Sat, 28 Jun 2008 00:39:34 +0800 Subject: [erlang-questions] is there any logging module for erlang? In-Reply-To: References: Message-ID: OK. Thanksvery much. I'll check it then. 2008/6/25, Ahmed Ali : > > Hi, > > You can find this logging module in: > http://code.google.com/p/log4erl/. Please let me know if it didn't > work. You can find instructions for using it in "README.txt" in the > downloaded file. > > Best regards, > > Ahmed Al-Issaei > > On Sat, Jun 21, 2008 at 5:00 PM, Ahmed Ali wrote: > > Hi, > > > > Sure. Let me clean up my code and then post it somewhere in the > > internet. I'll do that within the coming week. > > > > Best regards, > > > > Ahmed Al-Issaei > > > > On Tue, Jun 17, 2008 at 5:49 PM, devdoer bird > wrote: > >> yes, that's the one I need. > >> > >> 2008/6/14, Ahmed Ali : > >>> > >>> Hi, > >>> > >>> I understand what you want. Having used log4j myself, I've tried to > >>> find something similar for my own. Eventually, I had to write it > >>> myself. This module only supports file logs but it also have support > >>> for multiple log levels, including critical (fatal) level, different > >>> loggers for different modules, size-based rotation...etc. > >>> > >>> I haven't shared the project yet, as it has some shortcomings I need > >>> to address before releasing it, however, I'll announce it soon. I hope > >>> this is what you're look for. I'll keep you updated. > >>> > >>> Best regards, > >>> > >>> Ahmed Al-Issaei > >>> > >>> 2008/6/12 devdoer bird : > >>> > Hi: > >>> > > >>> > I wonder whether any logging module for erlang exists,like python's > >>> > logging > >>> > module which support :critical level setting ,filtering ,diffrent > >>> > handlers > >>> > (stream handler,socket handler,ram handler,file handler...). > >>> > > >>> > Thanks a lot! > >>> > _______________________________________________ > >>> > erlang-questions mailing list > >>> > erlang-questions@REDACTED > >>> > http://www.erlang.org/mailman/listinfo/erlang-questions > >>> > > >> > >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul-trapexit@REDACTED Fri Jun 27 20:40:37 2008 From: paul-trapexit@REDACTED (Paul Mineiro) Date: Fri, 27 Jun 2008 11:40:37 -0700 (PDT) Subject: [erlang-questions] external term format clarification Message-ID: at http://www.erlang.org/doc/apps/erts/erl_ext_dist.html#8 the table indicates that the free vars section of the fun_ext encoding is "4 * len" whereas the text says "len". is this consistent? if not, which one is correct? -- p From dnew@REDACTED Fri Jun 27 21:37:30 2008 From: dnew@REDACTED (Darren New) Date: Fri, 27 Jun 2008 12:37:30 -0700 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18532.45624.17663.666648@hamberg.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> <18531.60264.991238.276228@harpo.it.uu.se> <51A4415F-9973-422B-B944-FBA8922D018F@cs.otago.ac.nz> <18532.45624.17663.666648@hamberg.it.uu.se> Message-ID: <4865417A.1030402@san.rr.com> Sven-Olof Nystr|m wrote: > Since I disagree on point 1, I don't see much reason to discuss your > other points. Try looking at it again, and think that you're buffering something somewhere other than memory. Say you have ten disk files, and add() writes the message to one of the disk files and pop() pulls the message from one of the disk files, and Contents is tracking which files have messages. Or say that add() puts the text on the bottom line of the screen, and you get a {get,Who} message when the user clicks on the "OK, I've seen it" at the top of the screen, like processing fast-food orders or something. Or say that add() queues some processing up on a collection of processors (say, mpeg-compressing some video with hardware acceleration) and pop() gets the results, making room for another job to run on that hardware. All that would be the same code, with the same constraints needing to be solved. That Erlang has dynamically-sized message buffers doesn't mean that a bounded buffer example is nonsensical. It just means you use it for something other than what Ada et al use it for. -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From dnew@REDACTED Fri Jun 27 21:56:51 2008 From: dnew@REDACTED (Darren New) Date: Fri, 27 Jun 2008 12:56:51 -0700 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <8209f740806270423n2ef90742wa22a8262c8dafe98@mail.gmail.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <4863B777.908@san.rr.com> <8209f740806260938o6cb373e9v7a33706c76b7d846@mail.gmail.com> <4863D4DD.8020509@san.rr.com> <8209f740806261052g56ed1f94lad2f1d0fb073ed2b@mail.gmail.com> <4863DBFA.6000401@san.rr.com> <8209f740806270423n2ef90742wa22a8262c8dafe98@mail.gmail.com> Message-ID: <48654603.1080709@san.rr.com> Ulf Wiger wrote: > What I meant was that if we're discussing weird things that could happen > if we'd allow automatic GC of a process that's blocked, not registered, > and were there are no known references to the pid, we shouldn't > necessarily pay too much attention to the cases were the pid has > been serialized e.g. using term_to_binary(), because this is not really > different from the problems that can occur already. I'm unclear what you mean by "already"? Already in the current Erlang, where a process blocked in a receive keeps running, or in a new Erlang where a process blocked in a receive can get GCed? I'm simply pointing out that a process blocked in a receive getting GCed when there are "no known references to it" would seem to be very, very common. Any time you do {Someone,Somewhere} ! {starting, spawn(fun loop/0)} you're going to risk instantly GCing the processes you just spawned, because the only place the the PID exists is in the TCP buffers in the kernel. > We must always allow for the even that the process dies. A PID blocked in a receive that nobody is sending to isn't going to die unless the node dies, which we already can find out without having a reference to the pid. If the node dies, we know all the pids on that node are invalid. > Now bearing in mind that we often have distributed systems, and > no distributed GC, it is of course an excellent idea to not reuse pids > immediately. Indeed, the system goes out of its way to let you discover that it's going to reuse pids. I don't think it's possible for the system to reuse a pid without you being able to discover it has been reused, unless the process that gets reused explicitly unlinked itself from you after you linked it. > Either way, it may affect how soon pids are reused, Except that if you're worried about it, you link to the pid and/or monitor the node. Presumedly the runtime won't reuse the pid before telling you the previous pid has exited. > the time ought to be long enough that it should not be a problem for all > recommended uses of pids. Hmm. And what are the recommended uses of pids? This I haven't seen written down anywhere. It's not too hard to deduce, and to a first approximation it sounds like "don't run the Erlang distribution primitives over an unreliable network" to me. > As ROK wrote, the suggestion wasn't made seriously to begin with, OK. I was just checking, because he really seems to understand this stuff deeply. :-) I hadn't expected my follow-up to generate any more than "yes, it would be very difficult to implement that in reality." :-) > Personally, I don't think it would be feasible to introduce such > semantics anyway, since it would rarely be useful, and might > sometimes cause very strange behaviour. I think in Erlang that's probably true. There's also a similar language called Hermes where you very specifically kill a process by closing all its outgoing channels. (The Erlang equivalent would be killing off any processes whose PIDs you've passed to it.) >>> It should monitor the process in order to detect whether it dies. >> Again, not what I'm talking about. Erlang gives me ways to mostly do >> this unreliably. (If Erlang had a reliable way to do it, we wouldn't >> need mnesia:set_master_nodes, for example.) > > I don't follow your reasoning here. Erlang offers a reliable way of learning > whether a process is no longer reachable. Yep. That doesn't tell me reliably whether the process died. If the node becomes unreachable, then becomes reachable again, I understand that the pid is still valid, is it not? > There are lots of situations like that in distributed processing where > the runtime system cannot possibly give you guarantees or resolve > the situation automatically. I don't disagree. I simply said "I can't reliably monitor a process to determine if it has died." You're agreeing in detail, it seems. >> Except that storing pids "persistently" in variables is how one uses > I disagree. I think you're interpreting "persistent" differently than I meant it. My question was more along the lines of "why is it worse to store a PID in a disk file I recreate on start-up or a term-to-binary term, than it is to store it as a local variable in a function that's expected to run indefinitely?" I.e., there's no real lifetime on PIDs even if you don't store them as anything other than PIDs. Hence, the "how long is it valid" argument doesn't really mean anything. I'm storing the PID for an indefinite amount of time. Sure, it's in volatile storage, but the node is going to run for five years without being turned off, so... -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From dnew@REDACTED Fri Jun 27 22:00:42 2008 From: dnew@REDACTED (Darren New) Date: Fri, 27 Jun 2008 13:00:42 -0700 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <8209f740806270652n6120b210q3755399d23f42f6b@mail.gmail.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <18531.49944.89467.105800@harpo.it.uu.se> <8209f740806270553h56328293o463bda051ffce32d@mail.gmail.com> <18532.60482.6969.182774@hamberg.it.uu.se> <8209f740806270652n6120b210q3755399d23f42f6b@mail.gmail.com> Message-ID: <486546EA.4040503@san.rr.com> Ulf Wiger wrote: > One of the nice aspects of channels is that you cannot consume > a message from a channel unless you have its handle. This means > that you can be sure that a function that has not been passed a > channel reference cannot remove messages from it. Except that you either make a channel reference unlike every other datum in Erlang, or you can't really guarantee this. If you can do term_to_binary on a structure holding a channel, and get back a valid reference when you unwrap it, or if you can send a channel reference in a message, then any function might have a reference to that channel. It's not something easy for the *compiler* to guarantee, certainly without global knowledge of the entire program. -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From rvirding@REDACTED Sat Jun 28 00:35:08 2008 From: rvirding@REDACTED (Robert Virding) Date: Sat, 28 Jun 2008 00:35:08 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: References: <18529.883.586609.71953@hamberg.it.uu.se> Message-ID: <3dbc6d1c0806271535oc92eeaao467de2a6b4b9c828@mail.gmail.com> Richard is absolutely right here. Guards were always seen as part of pattern matching, the bits which you could not easily write in the patterns. They have always been a sequence of *tests* which either succeed of fail, there has never been a concept of error in guards. In this sense they have never been proper expressions, even restricted ones. Boolean operations were added to expand guards and tests the changed to more resemble expressions, for example tuple(T) => is_tuple(T), but guard tests are not expressions, they are *tests*. It might have been the wrong to expand the tests with boolean operations as it means that it is more difficult to see the difference, and hence to understand it. But they are not expressions, they never have been, they are tests! Robert P.S. I have seen attempts, not in Erlang, to include the "guard" bit in the patterns and they were *very* hard to read. On 26/06/2008, Richard A. O'Keefe wrote: > GUARDS > ====== > > > p6. "The complex selective receive primitive of old Erlang > has forced the Erlang designers to introduce a more > restricted type of expressions called _guards_." > > This is fiction, and it's fiction with a major error in it. > > Guards are *NOT* a restricted type of expression. > > This was much clearer in old Erlang. > Sadly, it has been obscured by the disastrous design bungle > of giving guard predicates aliases that can be used in > expressions and by allowing 'or' and 'and' -- which should > be avoided AT ALL TIMES AND IN ALL PLACES -- in both contexts. > > It would be really really useful if there were a compiler > option to complain loudly whenever a guard predicate's alias > is used in an expression. > > The *syntax* of guards in Erlang is similar to expressions, > but the *semantics* is very different. Consider: > > Guard Expression > succeeds returns 'true' > fails returns 'false' > returns something else > raises an exception > > Some forms that would raise an exception as an expression > fail as guards. Some forms that would raise an exception > as an expression succeed as guards. The order of evaluation > of expressions is defined. The order of evaluation of > guards is not defined. If you take a guard and treat it as > an expression, expecting it to always answer 'true' or 'false', > you are in for a nasty surprise: > > some guards that fail may raise an exception > some guards that succeed may raise an exception > some guards that succeed may return a value other than 'true'. > > The whole point of section 3.4 is > "SINCE guards are restricted expressions, > they ought to be unrestricted expressions" > but since the assumption is flat-out false, the conclusion is > unwarranted. > > The limitations of guards have NO intrinsic connection with > receive, although they turn out to be very pleasant for receive. > Erlang guards have their roots in concurrent logic programming > languages such as Strand88, GHC, FCP, and Parlog. > > Whenever I see people demanding that guards should be complexified, > I find myself wondering "do I know something about readability and > maintainability that they don't, or is it the other way around?" > The way I see it, if your selection process is complex enough that > you *want* to write complex guards with user defined functions, > then what you *need* is either > - abstract patterns, or > - a rewrite. > > Let me offer an example I saw today, in > http://www.infoq.com/articles/erlang-dsl > > It doesn't involve a user defined guard, but it does show what > happens when you have a "keep-guards-simple" mindset. > > Here's the original code. Its purpose is to return a function > that sends a message if something has a value within a limit. > > to_function(Pid, Action, Quantity, Ticker, Operator, Limit) -> > fun (Ticker_, Price) -> > if > Ticker =:= Ticker_ andalso > ( ( Price < Limit andalso Operator =:= less ) orelse > ( Price > Limit andalso Operator =:= greater ) > ) -> > Pid ! {Action, Quantity, Ticker} % place an order > ; true -> > erlang:display("no rule applied") > end > end. > > I looked at that and, well, it didn't make me smile. > That complex guard *had* to go. A few seconds later: > > to_function(Pid, Action, Quantity, Ticker, less, Limit) -> > fun (T, Price) when T =:= Ticker, Price < Limit -> > Pid ! {Action, Quantity, Ticker} % place an order > ; (_, _) -> > erlang:display("no rule applied") > end; > to_function(Pid, Action, Quantity, Ticker, greater, Limit) -> > fun (T, Price) when T =:= Ticker, Price > Limit -> > Pid ! {Action, Quantity, Ticker} % place an order > ; (_, _) -> > erlang:display("no rule applied") > end. > > In the original context, this now gives us the benefit of a 'compile > time' > failure if Operator is not 'less' or 'greater', rather than a 'run time' > failure. But it gets better. This is about buying or selling stocks, > and it so happens that we always want to buy when the price is LOW, > and sell when the price is HIGH. So now we can write > > to_function(Pid, buy, Quantity, Ticker, less, Limit) -> > fun (T, Price) when T =:= Ticker, Price < Limit > -> Pid ! {buy, Quantity, Ticker} > ; (_, _) > -> erlang:display("buy rule did not apply") > end; > to_function(Pid, sell, Quantity, Ticker, greater, Limit) -> > fun (T, Price) when T =:= Ticker, Price > Limit > -> Pid ! {sell, Quantity, Ticker} > ; (_, _) > -> erlang:display("sell rule did not apply") > end. > > Now we can *see* that the returned function will buy if the price is > low or sell when the price is high, but not vice versa. This was not > possible with the original version. > > What I've done here is to trade size against readability. The next > step would be to replace Pid by Broker, but that's a different > readability issue. From ulf@REDACTED Sat Jun 28 00:37:00 2008 From: ulf@REDACTED (Ulf Wiger) Date: Sat, 28 Jun 2008 00:37:00 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <48654603.1080709@san.rr.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <4863B777.908@san.rr.com> <8209f740806260938o6cb373e9v7a33706c76b7d846@mail.gmail.com> <4863D4DD.8020509@san.rr.com> <8209f740806261052g56ed1f94lad2f1d0fb073ed2b@mail.gmail.com> <4863DBFA.6000401@san.rr.com> <8209f740806270423n2ef90742wa22a8262c8dafe98@mail.gmail.com> <48654603.1080709@san.rr.com> Message-ID: <8209f740806271537h7f34f4d0wa24fad31cfaaf6f0@mail.gmail.com> 2008/6/27 Darren New : > Ulf Wiger wrote: >> What I meant was that if we're discussing weird things that could happen >> if we'd allow automatic GC of a process that's blocked, not registered, >> and were there are no known references to the pid, we shouldn't >> necessarily pay too much attention to the cases were the pid has >> been serialized e.g. using term_to_binary(), because this is not really >> different from the problems that can occur already. > > I'm unclear what you mean by "already"? Already in the current Erlang, > where a process blocked in a receive keeps running, or in a new Erlang > where a process blocked in a receive can get GCed? Already in the current Erlang, the process in question may die, and its pid may be reused by another process. > A PID blocked in a receive that nobody is sending to isn't going to die > unless the node dies, ...or if it is linked to a process which dies, or it is explicitly killed by another process. In sufficiently complex and long-lived programs, you may well need to assume that bad and unexpected things /will/ happen. (: http://www.erlang.se/workshop/2007/proceedings/06svenss.ppt makes for nice reading, illustrating some of the pitfalls of distributed programming. BR, Ulf W From saleyn@REDACTED Sat Jun 28 01:48:24 2008 From: saleyn@REDACTED (Serge Aleynikov) Date: Fri, 27 Jun 2008 19:48:24 -0400 Subject: [erlang-questions] SNMP MIB compiler fix for R12B-2 Message-ID: <48657C48.7020602@gmail.com> The MIB compiler doesn't properly handle the REFERENCE part of a MIB definition. Judging by the implementation of the grammar it was intended to be supported but not fully implemented. Attached is a fix that's backward compatible. Passing a new option 'reference' | {reference, Boolean} to the compiler will enable the feature. Regards, Serge -------------- next part -------------- A non-text attachment was scrubbed... Name: snmpc.R12B-2.patch Type: application/octet-stream Size: 5387 bytes Desc: not available URL: From dnew@REDACTED Sat Jun 28 01:54:55 2008 From: dnew@REDACTED (Darren New) Date: Fri, 27 Jun 2008 16:54:55 -0700 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <8209f740806271537h7f34f4d0wa24fad31cfaaf6f0@mail.gmail.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <4863B777.908@san.rr.com> <8209f740806260938o6cb373e9v7a33706c76b7d846@mail.gmail.com> <4863D4DD.8020509@san.rr.com> <8209f740806261052g56ed1f94lad2f1d0fb073ed2b@mail.gmail.com> <4863DBFA.6000401@san.rr.com> <8209f740806270423n2ef90742wa22a8262c8dafe98@mail.gmail.com> <48654603.1080709@san.rr.com> <8209f740806271537h7f34f4d0wa24fad31cfaaf6f0@mail.gmail.com> Message-ID: <48657DCF.5060407@san.rr.com> Ulf Wiger wrote: > ...or if it is linked to a process which dies, or it is explicitly > killed by another process. Well, yes, there is that. Thanks for clearing that up in my mind! -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From svenolof@REDACTED Sat Jun 28 08:45:03 2008 From: svenolof@REDACTED (Sven-Olof Nystr|m) Date: Sat, 28 Jun 2008 08:45:03 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <4865417A.1030402@san.rr.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> <18531.60264.991238.276228@harpo.it.uu.se> <51A4415F-9973-422B-B944-FBA8922D018F@cs.otago.ac.nz> <18532.45624.17663.666648@hamberg.it.uu.se> <4865417A.1030402@san.rr.com> Message-ID: <18533.56815.650170.222363@harpo.it.uu.se> Darren New writes: > Sven-Olof Nystr|m wrote: > > Since I disagree on point 1, I don't see much reason to discuss your > > other points. > > Try looking at it again, and think that you're buffering something > somewhere other than memory. Say you have ten disk files, and add() > writes the message to one of the disk files and pop() pulls the message > from one of the disk files, and Contents is tracking which files have > messages. > > Or say that add() puts the text on the bottom line of the screen, and > you get a {get,Who} message when the user clicks on the "OK, I've seen > it" at the top of the screen, like processing fast-food orders or > something. > > Or say that add() queues some processing up on a collection of > processors (say, mpeg-compressing some video with hardware acceleration) > and pop() gets the results, making room for another job to run on that > hardware. Why don't you show us what the code would look like? If you want to keep track of outstanding requests, you can do it without selective receive. > All that would be the same code, with the same constraints needing to be > solved. I don't think so. In Erlang, there are better ways to solve the problem you're sketching. > That Erlang has dynamically-sized message buffers doesn't mean that a > bounded buffer example is nonsensical. It just means you use it for > something other than what Ada et al use it for. As I said, show us. Sven-Olof From dnew@REDACTED Sat Jun 28 09:59:19 2008 From: dnew@REDACTED (Darren New) Date: Sat, 28 Jun 2008 00:59:19 -0700 Subject: [erlang-questions] Ideas for a new Erlang] Message-ID: <4865EF57.7040300@san.rr.com> Sven-Olof Nystr|m wrote: > If you want to keep track of outstanding requests, you can do it > without selective receive. True. For that matter, you never need selective receive. You can always receive everything, and write your own selection code, which is basically how selective receive is implemented in the C implementation of Erlang. I mean, heck, you don't need Erlang to do *any* of the tasks it does conveniently. The trick to arguing for a change is to find situations where the change will actually make things unambiguously better, often by comparing "the usual way" with "the new way" directly. BTW, the selective receive isn't what's keeping track of the outstanding requests. But since I'm actually trying to communicate rather than merely prove you wrong, I'll instead read what you had intended to communicate. See? > > All that would be the same code, with the same constraints needing to be > > solved. > > I don't think so. In Erlang, there are better ways to solve the > problem you're sketching. No doubt better in some ways, yes. > > That Erlang has dynamically-sized message buffers doesn't mean that a > > bounded buffer example is nonsensical. It just means you use it for > > something other than what Ada et al use it for. > > As I said, show us. NMJ. Unless you want to supply the hardware. If you can't envision how the situations I described (imprecisely) could match the (incomplete) code snippet, and you need a full-fledged Erlang program including hardware drivers and UI to understand the description, then you'll have to look elsewhere for that. -- Darren New / San Diego, CA, USA (PST) Helpful housekeeping hints: Check your feather pillows for holes before putting them in the washing machine. From svenolof@REDACTED Sat Jun 28 11:17:36 2008 From: svenolof@REDACTED (Sven-Olof Nystr|m) Date: Sat, 28 Jun 2008 11:17:36 +0200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <3dbc6d1c0806271535oc92eeaao467de2a6b4b9c828@mail.gmail.com> References: <18529.883.586609.71953@hamberg.it.uu.se> <3dbc6d1c0806271535oc92eeaao467de2a6b4b9c828@mail.gmail.com> Message-ID: <18534.432.191880.248215@harpo.it.uu.se> Robert Virding writes: > On 26/06/2008, Richard A. O'Keefe wrote: > > > p6. "The complex selective receive primitive of old Erlang > > has forced the Erlang designers to introduce a more > > restricted type of expressions called _guards_." > > > > This is fiction, and it's fiction with a major error in it. > > > > Guards are *NOT* a restricted type of expression. > > Richard is absolutely right here. Guards were always seen as part of > pattern matching, the bits which you could not easily write in the > patterns. They have always been a sequence of *tests* which either > succeed of fail, there has never been a concept of error in guards. In > this sense they have never been proper expressions, even restricted > ones. Boolean operations were added to expand guards and tests the > changed to more resemble expressions, for example tuple(T) => > is_tuple(T), but guard tests are not expressions, they are *tests*. As I already explained in my reply to Richard, I meant expressions in a general sense. Expressions in C or C++ are expressions. So are expressions in Java, Common Lisp and most other programming languages. Even regular expressions are expressions. Can you give a definition of the concept "expression" that excludes guard tests of Erlang and includes all other types of expressions? > It might have been the wrong to expand the tests with boolean > operations as it means that it is more difficult to see the > difference, and hence to understand it. > > But they are not expressions, they never have been, they are tests! What I really wanted to say was that guard tests of Erlang are strange and unusual. I see nothing in your posting that contradicts that. > Robert > > P.S. I have seen attempts, not in Erlang, to include the "guard" bit > in the patterns and they were *very* hard to read. Not sure what that means. Sven-Olof From jesper.louis.andersen@REDACTED Sat Jun 28 15:44:13 2008 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 28 Jun 2008 15:44:13 +0200 Subject: [erlang-questions] Announce: etorrent 0.8 Message-ID: <56a0a2840806280644n4f9df152v7ad42e353adffcd5@mail.gmail.com> etorrent is a bittorrent client written entirely in Erlang. Version 0.8 is a technology preview. Don't expect it to be completely robust yet: I have not checked that it always survive random process kills. Here is a list of what works though: * Full compliance with the original bittorrent specification (BEP-03). * Understands the bittorrent number allocations (BEP-04) but does not yet use it for anything. * Understands compact tracker returns (BEP-23). And all that in 5300 lines of code (where 3700 is the real code, rest is documentation). v0.9 will contain a real installation procedure. For now it is a hack. Overall it the code is very succinct and I had a lot of fun improving the code to this point. The Erlang VM copes really well with this: despite some really hairy memory-abuse weaknesses in the code base, I can run some 20 torrents in 350Mb of memory and 2-3% CPU if I enable '+Ktrue +A 128' to the erl(1) call, no HiPE. It will be easy to lower the memory usage, but it has been planned for a later release. We can also shave the CPU usage a bit more, as there has been no profiling done. The source code is available through Google Code (2-Clause BSD license): http://code.google.com/p/etorrent/ If you want to try it out, start by looking at the README file which contains the description of how to run the application without using the installation procedure (Which is what I am currently using for debugging it). To check that something is really going on, check the internal mnesia tables via 'tv:start().' as I have not had time to write a nice set of functions able to print out status on the console (If someone wants to add it, mail me, it is pretty easy). and if you want to hack, then [http://repo.or.cz/w/etorrent.git] contains links to the git repository. I'm currently the sole writer and maintainer, but hackers, comments, issues and patches are welcome! I don't expect to be able to find all bugs myself :), hence this tech. preview release. Coordinate with me if you want to do something bigger. J. From tuncer.ayaz@REDACTED Sat Jun 28 21:10:29 2008 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Sat, 28 Jun 2008 21:10:29 +0200 Subject: [erlang-questions] erlang emacsmode problem In-Reply-To: <4ac8254d0806260116y48803017hf1cecfa320530fcd@mail.gmail.com> References: <286156.61083.qm@web28303.mail.ukl.yahoo.com> <4ac8254d0806260116y48803017hf1cecfa320530fcd@mail.gmail.com> Message-ID: <4ac8254d0806281210x3ada24a1o783044f549a7fd3f@mail.gmail.com> On Thu, Jun 26, 2008 at 10:16 AM, Tuncer Ayaz wrote: > On Thu, Jun 26, 2008 at 9:54 AM, Dominic Williams > wrote: >> Hi, >> >>> Double check your paths. Make sure they actually exist as written. If >>> you've been getting this error since the upgrade to R12B-3 the cause >>> of the error should be obvious. >> >> And if that was your problem, you could use some more flexible emacs-lisp >> such as this (no hard-coded tools-x.y.z version): > > I think it's obvious but in case someone is possibly missing this > little bit of info the following or something more Emacs-idiomatic > should be done before that: > (if (or (eq system-type 'windows-nt) > (eq system-type 'ms-dos)) > (setq erlang-root-dir "C:/erlang") > (setq erlang-root-dir "/usr/local/otp") > ) If you configure & build Erlang without any --prefix setting it will be installed to /usr/local/lib/erlang: (if (or (eq system-type 'windows-nt) (eq system-type 'ms-dos)) (setq erlang-root-dir "C:/erlang") (setq erlang-root-dir "/usr/local/lib/erlang") ) >> (if >> (not (boundp 'erlang-root-dir)) >> (message "Skipping erlang-mode: erlang-root-dir not defined") >> (progn >> (set 'erlang-bin (concat erlang-root-dir "/bin/")) >> (set 'erlang-lib (concat erlang-root-dir "/lib/")) >> (if >> (not (boundp 'erlang-mode-path)) >> (set 'erlang-mode-path >> (concat >> erlang-lib >> (file-name-completion "tools-" erlang-lib) >> "emacs/erlang.el"))) >> (if >> (and >> (file-readable-p erlang-mode-path) >> (file-readable-p erlang-bin)) >> (progn >> (message "Setting up erlang-mode") >> (set 'exec-path (cons erlang-bin exec-path)) >> (set 'load-path (cons >> (concat >> erlang-lib >> (file-name-completion "tools-" erlang-lib) >> "emacs") >> load-path)) >> (set 'load-path (cons (file-name-directory erlang-mode-path) load-path)) >> (require 'erlang-start)) >> (message "Skipping erlang-mode: %s and/or %s not readable" >> erlang-bin erlang-mode-path)))) >> >> Regards, >> >> Dominic Williams >> http://dominicwilliams.net >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > From talmage.news@REDACTED Sun Jun 29 07:06:25 2008 From: talmage.news@REDACTED (Mike T) Date: Sat, 28 Jun 2008 22:06:25 -0700 Subject: [erlang-questions] Question about message passing paradigm Message-ID: <6e7d53010806282206s4fc36dcei9f99107f4ee6ac30@mail.gmail.com> Hello, I was not sure where best to post this and hope someone is able to help with this question. After being bitten by the pitfalls of lock-oriented multi threading I am interested in switching to message passing oriented concurrency. I have read that erlang has a per-process ordering guarantee (that is if A sends messages 1 and 2 to B 1 will arrive before 2 at B. However, there is no guarantee that messages from C and D will not be placed in between 1 and 2. So my question is this: In my current lock oriented program design I have threads dedicated to managing different collections and actions on those collections (hashmaps), lets call them A, B, C, and D. At several points in the program one thread needs to get data from the other collections and make a decision based on the collective state of the values it gathers form those other collections. An example situation would be A needing data from B, C, and D. In that case A would attempt to lock A, B, C, and D then gather the items it needs and release the lock. How would I accomplish this same task in a message passing manner? I had though of doing: A sends a message to B, C, and D asking for the data. However, B, C, and D may have each changed independently of each other by the time they receive the request for data and/or by the time they are able to send the message to A. So, how do you deal with situations like this in a message passing paradigm? -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang-questions_efine@REDACTED Sun Jun 29 12:42:04 2008 From: erlang-questions_efine@REDACTED (Edwin Fine) Date: Sun, 29 Jun 2008 06:42:04 -0400 Subject: [erlang-questions] Question about message passing paradigm In-Reply-To: <6e7d53010806282206s4fc36dcei9f99107f4ee6ac30@mail.gmail.com> References: <6e7d53010806282206s4fc36dcei9f99107f4ee6ac30@mail.gmail.com> Message-ID: <6c2563b20806290342o2da73957t15cbbc5123e7dcc2@mail.gmail.com> I'm a bit of an Erlang newbie so the gurus may come up with something much better than this, but I can think of these possible approaches: 1. Create a gen_fsm that controls all the collections. The collections could be ETS tables or gen_servers wrapping ETS tables. Under normal use, messages are sent to the fsm to update the collections individually. When the time comes to require consistency across the collections, send a message to the fsm to get the collective state data. The fsm goes into a different state while it gathers the data. This state would reject requests to update the collections (or wait until the state changes), although reads would still be allowed. On getting the result, the state changes back to allow updates again. 2. Create a memory-only Mnesia table for each collection, and use Mnesia transactions to get the multiple values atomically. 3. Change the architecture of the current lock-oriented program to make better use of Erlang's features. Hope this helps. 2008/6/29 Mike T : > Hello, > > I was not sure where best to post this and hope someone is able to > help with this question. > > After being bitten by the pitfalls of lock-oriented multi threading I > am interested in switching to message passing oriented concurrency. > > I have read that erlang has a per-process ordering guarantee (that is > if A sends messages 1 and 2 to B 1 will arrive before 2 at B. However, > there is no guarantee that messages from C and D will not be placed in > between 1 and 2. > > So my question is this: > In my current lock oriented program design I have threads dedicated to > managing different collections and actions on those collections > (hashmaps), lets call them A, B, C, and D. At several points in the > program one thread needs to get data from the other collections and > make a decision based on the collective state of the values it gathers > form those other collections. > > An example situation would be A needing data from B, C, and D. In that > case A would attempt to lock A, B, C, and D then gather the items it > needs and release the lock. > > How would I accomplish this same task in a message passing manner? > > I had though of doing: A sends a message to B, C, and D asking for the > data. However, B, C, and D may have each changed independently of each > other by the time they receive the request for data and/or by the time > they are able to send the message to A. So, how do you deal with > situations like this in a message passing paradigm? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- The great enemy of the truth is very often not the lie -- deliberate, contrived and dishonest, but the myth, persistent, persuasive, and unrealistic. Belief in myths allows the comfort of opinion without the discomfort of thought. John F. Kennedy 35th president of US 1961-1963 (1917 - 1963) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tom.ayerst@REDACTED Sun Jun 29 13:16:34 2008 From: tom.ayerst@REDACTED (Tom Ayerst) Date: Sun, 29 Jun 2008 12:16:34 +0100 Subject: [erlang-questions] Question about message passing paradigm In-Reply-To: <6e7d53010806282206s4fc36dcei9f99107f4ee6ac30@mail.gmail.com> References: <6e7d53010806282206s4fc36dcei9f99107f4ee6ac30@mail.gmail.com> Message-ID: <3c5163930806290416w34ffb249ybe227abf21ad2de5@mail.gmail.com> Hi Mike, To go off on a slight tangent with the hope of understanding your problem better, do you have to get the data from the other collections with them locked? Can you just ask for their current state and use whatever they have at that point or are they somehow dependent on each other? Tom 2008/6/29 Mike T : > Hello, > > I was not sure where best to post this and hope someone is able to > help with this question. > > After being bitten by the pitfalls of lock-oriented multi threading I > am interested in switching to message passing oriented concurrency. > > I have read that erlang has a per-process ordering guarantee (that is > if A sends messages 1 and 2 to B 1 will arrive before 2 at B. However, > there is no guarantee that messages from C and D will not be placed in > between 1 and 2. > > So my question is this: > In my current lock oriented program design I have threads dedicated to > managing different collections and actions on those collections > (hashmaps), lets call them A, B, C, and D. At several points in the > program one thread needs to get data from the other collections and > make a decision based on the collective state of the values it gathers > form those other collections. > > An example situation would be A needing data from B, C, and D. In that > case A would attempt to lock A, B, C, and D then gather the items it > needs and release the lock. > > How would I accomplish this same task in a message passing manner? > > I had though of doing: A sends a message to B, C, and D asking for the > data. However, B, C, and D may have each changed independently of each > other by the time they receive the request for data and/or by the time > they are able to send the message to A. So, how do you deal with > situations like this in a message passing paradigm? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ahmed.nawras@REDACTED Sun Jun 29 18:17:42 2008 From: ahmed.nawras@REDACTED (Ahmed Ali) Date: Sun, 29 Jun 2008 20:17:42 +0400 Subject: [erlang-questions] ANNOUNCE: logger for erlang: Log4erl Message-ID: Hi, Log4erl version 0.7 has been released at http://code.google.com/p/log4erl. This project is aimed at replicating features of Log4* family of loggers (e.g. Log4J). Although I think error_logger that comes with the official erlang distribution is a fine piece of software, I think this new logger has its advantages. First of all, in log4erl, I'm able to write each log in a line, which helps a lot in processing log files using standard UNIX utilities (awk, sed, grep...etc). Log4erl supports customized log level along with the standard features in error_logger. Also, I think a newbie coming from other languages would find it more familiar than error_logger. You can find the list of features currently supported in the project's website above. I'm planning to add more features (ability to log to DB, possibly SNMP, multiple log handlers for each loggers....etc). Please send your requests/feedback to my email and I'll be happy to consider them. Best regards, Ahmed From csaba.hoch@REDACTED Sun Jun 29 20:38:03 2008 From: csaba.hoch@REDACTED (Csaba Hoch) Date: Sun, 29 Jun 2008 20:38:03 +0200 Subject: [erlang-questions] [clarify] edoc: using @spec tag for a fun type Message-ID: <4867D68B.1050800@gmail.com> Hi, I have an edoc question: if I define a fun type with @type, can I use it when I define a function with @spec? I mean something like this: %% @type number_fun = ((number()) -> number()). %% @spec f = number_fun() f(X) -> 2*X. But my version doesn't work. Is it possible to do something similar? Thank you, Csaba From tom.ayerst@REDACTED Sun Jun 29 22:56:53 2008 From: tom.ayerst@REDACTED (Tom Ayerst) Date: Sun, 29 Jun 2008 21:56:53 +0100 Subject: [erlang-questions] New module syntax and semantics? Message-ID: <3c5163930806291356k1ef3560dp67cd20aabbcc7b51@mail.gmail.com> Is was looking at the mochiweb code and I saw some oddities. firstly the module statement: -module(mochiweb_request, [Socket, Method, RawPath, Version, Headers]). In the use of mochiweb I can see things like Req:get(method). This looks more like an object method call, is this new? I cannot see an docs about it. Thanks for any info. Tom Ayerst In the example Req ={mochiweb_request,#Port<0.142>,'GET', "/", {1,1}, {9, {"host", {'Host',"127.0.0.1:8888"}, {"accept", {'Accept', "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"}, nil, {"accept-language", {'Accept-Language', "en-gb,en;q=0.5"}, {"accept-encoding", {'Accept-Encoding', "gzip,deflate"}, {"accept-charset", {'Accept-Charset', "ISO-8859-1,utf-8;q=0.7,*;q=0.7"}, nil,nil}, nil}, {"connection", {'Connection',"keep-alive"}, {"cache-control", {'Cache-Control',"max-age=0"}, nil,nil}, nil}}}, {"user-agent", {'User-Agent', "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9) Gecko/2008052906 Firefox/3.0"}, {"keep-alive", {'Keep-Alive',"300"}, nil,nil}, nil}}}} -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.polvorin@REDACTED Sun Jun 29 23:21:53 2008 From: pablo.polvorin@REDACTED (Pablo Polvorin) Date: Sun, 29 Jun 2008 18:21:53 -0300 Subject: [erlang-questions] New module syntax and semantics? In-Reply-To: <3c5163930806291356k1ef3560dp67cd20aabbcc7b51@mail.gmail.com> References: <3c5163930806291356k1ef3560dp67cd20aabbcc7b51@mail.gmail.com> Message-ID: <1ffe809c0806291421m63ec9612xd6916ca0b9b85b28@mail.gmail.com> seems to be using parameterized modules http://www.erlang.se/workshop/2003/paper/p29-carlsson.pdf 2008/6/29, Tom Ayerst : > Is was looking at the mochiweb code and I saw some oddities. > > firstly the module statement: > -module(mochiweb_request, [Socket, Method, RawPath, Version, Headers]). > > In the use of mochiweb I can see things like Req:get(method). > > This looks more like an object method call, is this new? I cannot see an > docs about it. > > Thanks for any info. > > Tom Ayerst > > In the example Req ={mochiweb_request,#Port<0.142>,'GET', > "/", > {1,1}, > {9, > {"host", > {'Host',"127.0.0.1:8888"}, > {"accept", > {'Accept', > > "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"}, > nil, > {"accept-language", > {'Accept-Language', > "en-gb,en;q=0.5"}, > {"accept-encoding", > {'Accept-Encoding', > "gzip,deflate"}, > {"accept-charset", > {'Accept-Charset', > > "ISO-8859-1,utf-8;q=0.7,*;q=0.7"}, > nil,nil}, > nil}, > {"connection", > {'Connection',"keep-alive"}, > {"cache-control", > > {'Cache-Control',"max-age=0"}, > nil,nil}, > nil}}}, > {"user-agent", > {'User-Agent', > "Mozilla/5.0 (Windows; U; > Windows NT 6.0; en-GB; rv:1.9) Gecko/2008052906 Firefox/3.0"}, > {"keep-alive", > {'Keep-Alive',"300"}, > nil,nil}, > nil}}}} > -- -- pablo http://ppolv.wordpress.com ---- From chsu79@REDACTED Sun Jun 29 23:24:37 2008 From: chsu79@REDACTED (Christian S) Date: Sun, 29 Jun 2008 23:24:37 +0200 Subject: [erlang-questions] New module syntax and semantics? In-Reply-To: <3c5163930806291356k1ef3560dp67cd20aabbcc7b51@mail.gmail.com> References: <3c5163930806291356k1ef3560dp67cd20aabbcc7b51@mail.gmail.com> Message-ID: The feature is called "Parameterized Modules", and it is still unofficial. See Richard Carlsson's paper on it. google can find it. There was also a follow-up paper presenting OOP-in-Erlang last year at the EUC. PS: Is the unofficialness of this feature going official, or does the vague code-upgrade properties and the hidden-state rewarding programming making it not as well received? From ulf@REDACTED Sun Jun 29 23:34:09 2008 From: ulf@REDACTED (Ulf Wiger) Date: Sun, 29 Jun 2008 23:34:09 +0200 Subject: [erlang-questions] New module syntax and semantics? In-Reply-To: <3c5163930806291356k1ef3560dp67cd20aabbcc7b51@mail.gmail.com> References: <3c5163930806291356k1ef3560dp67cd20aabbcc7b51@mail.gmail.com> Message-ID: <8209f740806291434k40e7a53ex544c1c681319be94@mail.gmail.com> That's an example of parameterized modules, which is an experimental and unsupported feature in Erlang. http://www.planeterlang.org/story.php?title=The_black_art_of_Erlangs_parameterized_modules http://www.erlang.se/workshop/2003/paper/p29-carlsson.pdf BR, Ulf W 2008/6/29 Tom Ayerst : > Is was looking at the mochiweb code and I saw some oddities. > > firstly the module statement: > -module(mochiweb_request, [Socket, Method, RawPath, Version, Headers]). > > In the use of mochiweb I can see things like Req:get(method). > > This looks more like an object method call, is this new? I cannot see an > docs about it. > > Thanks for any info. > > Tom Ayerst > > In the example Req ={mochiweb_request,#Port<0.142>,'GET', > "/", > {1,1}, > {9, > {"host", > {'Host',"127.0.0.1:8888"}, > {"accept", > {'Accept', > > "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"}, > nil, > {"accept-language", > {'Accept-Language', > "en-gb,en;q=0.5"}, > {"accept-encoding", > {'Accept-Encoding', > "gzip,deflate"}, > {"accept-charset", > {'Accept-Charset', > > "ISO-8859-1,utf-8;q=0.7,*;q=0.7"}, > nil,nil}, > nil}, > {"connection", > {'Connection',"keep-alive"}, > {"cache-control", > > {'Cache-Control',"max-age=0"}, > nil,nil}, > nil}}}, > {"user-agent", > {'User-Agent', > "Mozilla/5.0 (Windows; U; > Windows NT 6.0; en-GB; rv:1.9) Gecko/2008052906 Firefox/3.0"}, > {"keep-alive", > {'Keep-Alive',"300"}, > nil,nil}, > nil}}}} > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From ok@REDACTED Mon Jun 30 04:18:24 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 30 Jun 2008 14:18:24 +1200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18532.45624.17663.666648@hamberg.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> <18531.60264.991238.276228@harpo.it.uu.se> <51A4415F-9973-422B-B944-FBA8922D018F@cs.otago.ac.nz> <18532.45624.17663.666648@hamberg.it.uu.se> Message-ID: <4A107132-9887-4734-BA7E-A47CF8EB087F@cs.otago.ac.nz> OAs far as I can tell, your line of argument is: > > 1. The bounded buffer example is a useful and relevant example of > Erlang programming. No, that is NOT my line of argument at all. All I claim for it is that it is a SIMPLE and FAMILIAR example. > > Since I disagree on point 1, I don't see much reason to discuss your > other points. Since you are mistaken about point 1, this really doesn't carry much weight. > > > Let's take a second look at your buffer example: > >> buffer(Status, Contents) -> >> receive >> {put,Msg} when Status /= full -> >> {Status1, Contents1} = add(Contents, Msg) >> ; {get,Who} when Status /= empty -> >> {Status1, Contents1, Msg} = pop(Contents), >> Who ! Msg >> end, >> buffer(Status1, Contents1). > > At first, I thought it looked pretty elegant, including the somewhat > unexpected placement of the semicolon. What's unexpected about it? Every Erlang example I post to this list uses that placement. I've used that placement ever since Peter van Roy complained in his PhD thesis about how hard it was to tell "," and ";" apart at the end of a line. (It's also where many people in the Dijkstra camp put their semicolons, although in imperative languages with no need to distinguish ";" and "," I do not follow them.) > In Erlang, none of these things are true. > > - there is already a mechanism for buffering (mailboxes). No need to > implement a new one. That simply isn't relevant to my argument. > > > - dynamic allocation is available That simply isn't relevant to my argument. > > > - communication is asynchronous. Granted, one can simulate > synchronous communication, but I don't see you doing that here. That simply isn't relevant to my argument. My argument is simplicity itself: 1. A process is like a concurrent object or actor. 2. Just as a concurrent object or actor can be in different states, so can a process. 3. A process that is willing to accept a message in SOME states may not be willing to accept it in ALL states. [I repeat: this has NOTHING to do with buffering, NOTHING to do with dynamic allocation, and NOTHING to do with the synchronicity or otherwise of communication. It is a local fact about the process in question.] 4. The differences between states may be so large that separate 'receives' are appropriate. On the other hand, they may not. A 'state-senstive receive' can be exponentially more concise than a combination of separate receives. That's it, basically. I would remind readers that Naturam expellas furca, tamen usque recurret... (Horace, Ep.1.10) The relevance here is that although Concurrent ML _starts_ with simple mailboxes and events, a good part of CML hacking is plugging these things together with combinators to get with difficulty and obscurity what Erlang gets directly. > \As I said in my previous reply, you haven't really implemented > bounded > buffers: And it doesn't MATTER, because the example is about the fact that a process doesn't want to accept _exactly_ the same set of messages all the time. The inspiration was in fact the Santa Clause Problem. > > Now, it might perhaps be interesting to compare how selective receive > and channels can be used to implement anything at all, really. > If you still insist that the above is a correct implementation of > bounded buffers, "Still"? "Insist?" How could I "stil" do something I never started doing? Why "insist" when I never insisted? The onus is on the proposer of a major emasculation of Erlang to provide examples showing that it is a good idea. From dl-erlang@REDACTED Mon Jun 30 04:11:06 2008 From: dl-erlang@REDACTED (David Lutz) Date: Sun, 29 Jun 2008 19:11:06 -0700 Subject: [erlang-questions] Extraordinarily long garbage collection with R12B-2 Message-ID: I am still pretty much an erlang newbie, so hopefully I am missing something obvious. We have an program that was running OK with R11B-4, but now is experiencing incredibly long garbage collection times under load with R12B-2. The application is serving network requests, many clients connecting only for a short period, request/response style. We have erlang:system_monitor/2 set up to deliver messages if GC takes more than 1000ms. With R11B-4 we almost never saw this. With R12B-2 it happens fairly regularly and the timeouts are really long, most of the reports are for timeout of 4294967. Occasionally a slightly smaller timeout will be reported, but always well over 4000000. The long GC happens in both our long running gen_server processes and our ephemeral worker processes. Obviously our clients experience timeouts and other connection errors while this long GC is happening. Below is small clip from our log file showing the long GC reports. Has anybody seen this issue before? Why would R12B-2 behave so much different that R11B-4 in this respect? What steps can I try to pinpoint the issue? Thanks, D.L. =INFO REPORT==== 2008-06-29 02:02:41 UTC === Long GC in <0.13094.0>: [{timeout,4294967}, {old_heap_block_size,610}, {heap_block_size,987}, {mbuf_size,0}, {stack_size,10}, {old_heap_size,173}, {heap_size,425}] ProcessRegisteredName: [] =INFO REPORT==== 2008-06-29 02:02:43 UTC === Long GC in <0.59.0>: [{timeout,4295024}, {old_heap_block_size,377}, {heap_block_size,610}, {mbuf_size,0}, {stack_size,26}, {old_heap_size,118}, {heap_size,7}] ProcessRegisteredName: {registered_name,XXX} =INFO REPORT==== 2008-06-29 02:02:49 UTC === Long GC in <0.18816.0>: [{timeout,4294967}, {old_heap_block_size,377}, {heap_block_size,987}, {mbuf_size,0}, {stack_size,14}, {old_heap_size,33}, {heap_size,476}] ProcessRegisteredName: [] =INFO REPORT==== 2008-06-29 02:02:52 UTC === Long GC in <0.21233.0>: [{timeout,4294967}, {old_heap_block_size,1597}, {heap_block_size,987}, {mbuf_size,0}, {stack_size,31}, {old_heap_size,28}, {heap_size,878}] ProcessRegisteredName: [] =INFO REPORT==== 2008-06-29 02:02:53 UTC === Long GC in <0.22024.0>: [{timeout,4294967}, {old_heap_block_size,610}, {heap_block_size,610}, {mbuf_size,0}, {stack_size,32}, {old_heap_size,352}, {heap_size,276}] ProcessRegisteredName: [] From ok@REDACTED Mon Jun 30 04:59:48 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 30 Jun 2008 14:59:48 +1200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18532.54192.651817.522964@hamberg.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> <18531.60264.991238.276228@harpo.it.uu.se> <4864B150.7060303@gmail.com> <18532.54192.651817.522964@hamberg.it.uu.se> Message-ID: <3B3237D8-D748-43B5-B459-36399AFDA905@cs.otago.ac.nz> On 27 Jun 2008, at 11:49 pm, Sven-Olof Nystr|m wrote: > I've met many people, who even after having quite a bit of experience > with Erlang had trouble answering simple questions such as: if a > receive matches many patterns, say > > receive > {foo, X} -> ... ; > {bar, Y} -> ... > end > > and there are messages in the mailbox matching both patterns, which > clause and which message is selected? Hmm. What does the Erlang reference manual say? 6.10 Receive [1] Receives messages sent to the process using the send operator (!). [2] The patterns are sequentially matched against the first message in time order in the mailbox, then the second, and so on. [3] If a match succeeds and the optional guard sequence is true, the corresponding Body is evaluated. [4] The matching message is consumed, that is removed from the mailbox, while any other messages in the mailbox remain unchanged. Yup, it's sentence 2. Now if there are 'many people' who can't be bothered reading even the second sentence of the section that deals with the construct, I fear there must be 'many people' who will be helpless no matter WHAT constructs are offered or how they are documented. Perhaps the wording could be changed to [2'] All of the patterns are matched against the first message (the one that has been waiting longest) in the mailbox. [3'] If some pattern matches and the corresponding guard sequence succeeds, [4'] the matching message is removed from the mailbox and the corresponding body is evaluated. [2"] If the first message is not accepted, the second one will be tried, then the third, and so on. If none of the messages in the mailbox is accepted, the process will wait for new messages, checking them one at a time in the order they arrive. [4"] Messages in the mailbox that are not accepted are left in the mailbox without any change in their contents or order. If there were no claim about 'many people' with 'quite a bit of experience' not reading or not understanding the manual, I'd say to leave the manual alone. It seems clear enough to me. But since there is such a claim, maybe the revised wording above might help. >> now Sven-Olof is obviously entitled to his opinion > > thanks :-) That "obviously" is a staple of modern thought. I am no longer as persuaded as I was that it is true. "The Ethics of Belief", by William K. Clifford is uncomfortable reading, but salutory. You'll find a copy at http://www.infidels.org/library/historical/w_k_clifford/ethics_of_belief.html Because Nystrom has met 'many people' who apparently have trouble understanding the extremely strong similarity of behaviour between 'receive' and 'case', he HAS evidence for his belief that 'receive' is hard to understand. Because I struggled with Concurrent CML, but found Ada, Occam, and Erlang intuitively obvious, I have evidence that 'receive' is not hard to understand. So we have a thesis, and an antithesis, and it's time to look for the synthesis. Clearly, whether 'receive' is easy or hard to understand could depend on and presumably does depend on some factors such as - one's willingness to read the documentation - one's ability to read the documentation - one's experience with other notations such as CSP, Ada, Occam, TTCN-3 (where 'receive' can do some limited pattern matching, which when combined with 'alt' gives you a 'selective receive' that I will admit to be tricky), &c - whether one learned Erlang from a book or from a course, - in the latter case, whether the teacher understood 'receive' him/herself - and whether the teacher TAUGHT the students that 'receive' was difficult, the way many people teach that recursion is difficult, and many school maths teachers teach that "maths is too hard" - whether one has used profiling tools that report match attempts in receives, as well as match successes (_are_ there such tools?) The only evidence I have to help me decide here is that the information is not just there in the documentation, it's lying right on the surface, very hard to miss. From ok@REDACTED Mon Jun 30 05:02:13 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 30 Jun 2008 15:02:13 +1200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18532.62547.68630.333358@hamberg.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> <18531.60264.991238.276228@harpo.it.uu.se> <51A4415F-9973-422B-B944-FBA8922D018F@cs.otago.ac.nz> <18532.45624.17663.666648@hamberg.it.uu.se> <18532.62547.68630.333358@hamberg.it.uu.se> Message-ID: <988E040C-E90A-4824-BFBA-6C7F7B83A857@cs.otago.ac.nz> On 28 Jun 2008, at 2:08 am, Sven-Olof Nystr|m wrote: > I need to correct myself on one point. Let me correct your correction: the SPARK subset of Ada does _not_ include dynamic allocation, doubtless for the 'good reasons' you mentioned. From bobcalco@REDACTED Mon Jun 30 06:09:37 2008 From: bobcalco@REDACTED (Bob Calco) Date: Mon, 30 Jun 2008 00:09:37 -0400 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <3B3237D8-D748-43B5-B459-36399AFDA905@cs.otago.ac.nz> References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> <18531.60264.991238.276228@harpo.it.uu.se> <4864B150.7060303@gmail.com> <18532.54192.651817.522964@hamberg.it.uu.se> <3B3237D8-D748-43B5-B459-36399AFDA905@cs.otago.ac.nz> Message-ID: <003501c8da67$1cf60130$56e20390$@rr.com> > So we have a thesis, and an antithesis, and it's time to look for the > synthesis. That's way too Hegelian for me at this time of night.. er, morning! (12:09 AM EST) I once doubted my political philosophy background would come in handy one day. Whoddathunk it'd be today, on this subject, in this forum? :) - Bob Clearly, whether 'receive' is easy or hard to understand > could depend on and presumably does depend on some factors such as > - one's willingness to read the documentation > - one's ability to read the documentation > - one's experience with other notations such as CSP, Ada, Occam, > TTCN-3 (where 'receive' can do some limited pattern matching, > which when combined with 'alt' gives you a 'selective receive' > that I will admit to be tricky), &c > - whether one learned Erlang from a book or from a course, > - in the latter case, whether the teacher understood 'receive' > him/herself > - and whether the teacher TAUGHT the students that 'receive' was > difficult, the way many people teach that recursion is difficult, > and many school maths teachers teach that "maths is too hard" > - whether one has used profiling tools that report match attempts > in receives, as well as match successes (_are_ there such tools?) > > The only evidence I have to help me decide here is that the > information is not just there in the documentation, it's lying right > on the surface, very hard to miss. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions From ok@REDACTED Mon Jun 30 06:16:32 2008 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 30 Jun 2008 16:16:32 +1200 Subject: [erlang-questions] Ideas for a new Erlang In-Reply-To: <18534.432.191880.248215@harpo.it.uu.se> References: <18529.883.586609.71953@hamberg.it.uu.se> <3dbc6d1c0806271535oc92eeaao467de2a6b4b9c828@mail.gmail.com> <18534.432.191880.248215@harpo.it.uu.se> Message-ID: <26A7CA27-A3C9-4524-853F-40032B50D91E@cs.otago.ac.nz> On 28 Jun 2008, at 9:17 pm, Sven-Olof Nystr|m wrote: > As I already explained in my reply to Richard, I meant expressions in > a general sense. Don't. It's a good way to confuse people, especially yourself. It's just like those people who call preprocessor directives in C "statements" and then wonder why putting a semicolon on the end gets them in trouble. > Expressions in C or C++ are expressions. Yes they are. But declarations, statements, and directives are NOT. In Erlang, you would confuse yourself and others terribly if you called patterns expressions. > So are expressions in Java, > Common Lisp and most other programming languages. Even regular > expressions are expressions. Can you give a definition of the concept > "expression" that excludes guard tests of Erlang and includes all > other types of expressions? From the OED: "A collection of symbols together expressing an algebraical quantity." A compatible definition from the Wikipedia: "An expression in a programming language is a combination of values, variables, operators, and functions that are interpreted (evaluated) according to the particular rules of precedence and of association for a particular programming language, which computes and then produces (returns, in a stateful environment) another value." Wolfram's MathWorld doesn't seem to have a definition of 'expression' as such, but the pages I checked could all be summarised as "A collection of symbols which is well formed according to some grammar having a semantics that determines a value from the symbols in a given context." In Erlang, expressions are described in Chapter 6 of the reference manual. In C, C++, Java and so on, "expressions" are precisely the things accepted by the grammar rules for 'expression' and its associated static conditions. It's exactly the same in Erlang. In Erlang, those things do NOT include guards. There is no Boolean type in Erlang. Expressions in Erlang have values which are Erlang terms. Guards in Erlang have outcomes (success or failure) which historically were not represented in any way as Erlang terms. Later, as part of the unfortunate bungle, 'true' and 'false' were abused for this purpose, but the fact that these things are atoms makes the post-design bodgery obvious. Do you remember Lincoln's joke? Comedian: How many legs does a dog have, if you call a tail a leg? Straight man: five. Comedian: Wrong! _Calling_ a tail a leg doesn't _make_ it a leg! Guards and expressions don't follow the same rules. That means it isn't USEFUL to call guards expressions. > What I really wanted to say was that guard tests of Erlang are strange > and unusual. I see nothing in your posting that contradicts that. There is nothing strange or unusual about Erlang guards. They are utterly banal, within the family of concurrent declarative languages. FCP, GHC, Parlog, Strand88 all have essentially the same idea; I _think_ Andorra is similar but by the time I wanted to learn Andorra it was dead. Admittedly, within the family of Fortran-like languages (which includes C, C++, and Java), guards are not to be found. But if I wanted concurrent Fortran, well, I'd lie down until the feeling passed off. (And if I wanted concurrent C, I'd want Concurrent C, which had a construct not entirely unlike Ada's select/Erlang's receive.) > > >> Robert >> >> P.S. I have seen attempts, not in Erlang, to include the "guard" bit >> in the patterns and they were *very* hard to read. > > Not sure what that means. It means things like verify(flight(int$Num, atom$Origin, atom$Destination, numb$Cost)) <- prst('Your flight is fine.") & nl. -- IBM PROLOG Programmer's Guide, page 63. The Erlang equivalent would be something like verify(#flight{Num, Origin, Destination, Cost}) when is_integer(Num), is_atom(Origin), is_atom(Destination), is_number(Cost) -> io:write("Your flight is fine.\n"); Somewhat confusingly, p1$p2$p3$T in IBM PROLOG for 370 meant "p1(T) *or* p2(T) *or* p3(T)". Now try extending that to comparisons... From fuad@REDACTED Mon Jun 30 06:21:09 2008 From: fuad@REDACTED (Fuad Tabba) Date: Mon, 30 Jun 2008 16:21:09 +1200 Subject: [erlang-questions] Writing a Concurrent Binary Search Tree Data Structure in Erlang In-Reply-To: <909c265f0806261354m7e3f711fk8ce3cafd0500fd3b@mail.gmail.com> References: <909c265f0806250222q76e6651emd164ea221f15202f@mail.gmail.com> <4d08db370806250240gbf0e527n798a296292124ef8@mail.gmail.com> <909c265f0806250246s7abbcb2at6884f50e31c5d81c@mail.gmail.com> <4d08db370806250257h63fa7cb7m2dbf1a8b682768e2@mail.gmail.com> <909c265f0806251510x60ab819aqfb074b5558d7fc46@mail.gmail.com> <909c265f0806261354m7e3f711fk8ce3cafd0500fd3b@mail.gmail.com> Message-ID: <909c265f0806292121q671a9232q440e7eb7a2d7ea22@mail.gmail.com> Thanks for everyone's replies. I finished writing the concurrent implementation of a bst, including delete. I think it's correct, but I haven't done any extensive testing, just some tests and some reasoning (which doesn't even begin to be enough, I know). I wanted to see what you think about this implementation:- http://www.cs.auckland.ac.nz/~fuad/pbst.erl The functions that perform insert or delete don't have any feedback mechanism, which I don't think is necessary since any later call to any other function should be ordered properly, but this is making benchmarking a bit of a challenge. I can't think of a way to measure how well this performs without implementing an acknowledgement mechanism. Any ideas? I'm also not too happy with the process overhead per node; it's not a big deal if each node stores a lot of data, but it is quite significant for small nodes. Thanks, /Fuad On Fri, Jun 27, 2008 at 8:54 AM, Fuad Tabba wrote: > Cool, that's matches my understanding of Hynek's suggestion as well. Thanks > Kevin. > > I'm actually writing code along similar lines to test out the idea. It > seems rather simple and elegant; what I don't like though is the added > overhead of a process per node - I know that Erlang processes are light, but > if I recall correctly from reading in Joe's book, each process has memory > overhead in the order of 100 bytes. If the the data per node is large then > it's not a big deal, but if the amount of data stored per node is small then > that could be some serious overhead. > > Implementing delete() is going to be a bit tricky, need to be wary of > deadlocks; but this isn't an Erlang issue, this is a general issue with > writing a concurrent and shared data structure. > > Thanks again! > > Cheers, > /Fuad > > On Fri, Jun 27, 2008 at 8:22 AM, Kevin Scaldeferri > wrote: > >> >> On Jun 25, 2008, at 3:10 PM, Fuad Tabba wrote: >> >> I think I understand what you're saying here; but I still can't wrap my >>> head around putting that into code. Is there similar code that you know of >>> available online? The structure doesn't have to be a BST, just anything that >>> demonstrates the idea of having a structure that stores lots persistent data >>> to be viewed/modified by many processes and is scalable... >>> >>> >>> On Wed, Jun 25, 2008 at 9:40 PM, Hynek Vychodil < >>> vychodil.hynek@REDACTED> wrote: >>> Because there is no shared state between processes in Erlang, I would >>> start think about process per tree node, not process per action as usual in >>> classic shared memory (for example C) implementations. >>> I know, it sounds weird, but may be help you :-) >>> >> >> >> >> Here's a basic outline of this idea, if I understand it right. delete() >> is not implemented, and it has some other deficiencies, but I hope it at >> least starts to demonstrate the idea of a recursive data structure >> represented as a collection of processes. >> >> >> -kevin >> >> %%%%%%%%%%%%%%%%%%%%%%%%% >> >> -module(cbst). >> -export([new/1, exists/2, insert/2, delete/2]). >> >> -record(node, {key, >> left=null, >> right=null}). >> >> %% >> %% public API >> %% >> >> new(Key) -> >> spawn(fun() -> create(Key) end). >> >> exists(Node, Key) -> >> Node ! {exists, Key, self()}, >> receive >> true -> true; >> false -> false >> end. >> >> insert(Node, Key) -> >> Node ! {insert, Key}. >> >> delete(Node, Key) -> >> Node ! {delete, Key}. >> >> >> %% >> %% private functions >> %% >> >> create(Key) -> >> Node = #node{key=Key}, >> loop(Node). >> >> loop(Node) -> >> receive >> {exists, Key, Caller}=Q -> >> case Node#node.key of >> Key -> >> Caller ! true; >> This when Key < This, Node#node.left =/= null -> >> Node#node.left ! Q; >> This when Key > This, Node#node.right =/= null -> >> Node#node.right ! Q; >> _ -> >> Caller ! false >> end, >> loop(Node); >> {insert, Key} -> >> case Node#node.key of >> Key -> >> loop(Node); % already in tree >> This when Key < This -> >> case Node#node.left of >> null -> >> Left = new(Key), >> loop(Node#node{left=Left}); >> Left -> >> Left ! {insert, Key}, >> loop(Node) >> end; >> This when Key > This -> >> case Node#node.right of >> null -> >> Right = new(Key), >> loop(Node#node{right=Right}); >> Right -> >> Right ! {insert, Key}, >> loop(Node) >> end >> end; >> {delete, Key} -> >> % left as an exercise for the reader >> loop(Node) >> end. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bobcalco@REDACTED Mon Jun 30 06:29:04 2008 From: bobcalco@REDACTED (Bob Calco) Date: Mon, 30 Jun 2008 00:29:04 -0400 Subject: [erlang-questions] Ideas for a new Erlang References: <18529.883.586609.71953@hamberg.it.uu.se> <18530.17062.650561.451935@hamberg.it.uu.se> <95be1d3b0806251417y33ff2e5br7597757656ce259e@mail.gmail.com> <8209f740806251457w230ea12bi124f70b93c06a25c@mail.gmail.com> <478E4255-8002-46DB-93E5-C9F3F6B5E8D7@cs.otago.ac.nz> <18531.60264.991238.276228@harpo.it.uu.se> <4864B150.7060303@gmail.com> <18532.54192.651817.522964@hamberg.it.uu.se> <3B3237D8-D748-43B5-B459-36399AFDA905@cs.otago.ac.nz> Message-ID: <003601c8da69$d497f990$7dc7ecb0$@rr.com> BTW.... > > > So we have a thesis, and an antithesis, and it's time to look for the > > synthesis. > > That's way too Hegelian for me at this time of night.. er, morning! > (12:09 AM EST) > > I once doubted my political philosophy background would come in handy > one day. Whoddathunk it'd be today, on this subject, in this forum? > > :) > > - Bob > > Clearly, whether 'receive' is easy or hard to understand > > could depend on and presumably does depend on some factors such as > > - one's willingness to read the documentation > > - one's ability to read the documentation > > - one's experience with other notations such as CSP, Ada, Occam, > > TTCN-3 (where 'receive' can do some limited pattern matching, > > which when combined with 'alt' gives you a 'selective receive' > > that I will admit to be tricky), &c > > - whether one learned Erlang from a book or from a course, > > - in the latter case, whether the teacher understood 'receive' > > him/herself > > - and whether the teacher TAUGHT the students that 'receive' was > > difficult, the way many people teach that recursion is difficult, > > and many school maths teachers teach that "maths is too hard" > > - whether one has used profiling tools that report match attempts > > in receives, as well as match successes (_are_ there such tools?) > > > > The only evidence I have to help me decide here is that the > > information is not just there in the documentation, it's lying right > > on the surface, very hard to miss. I have the same opinion about the so-called 'law' of comparative advantage, which since Samuelson's textbook on economics has been TAUGHT to be "irrefutable"... yet one need only read Smith's Invisible Hand passage in the original (not as quoted by Samuelson!), and think about the problem as stated for approx. 2 seconds, to find the folly of Ricardo's little joke. But what do I know? I'm just a programmer... - Bob From erik.reitsma@REDACTED Mon Jun 30 08:45:07 2008 From: erik.reitsma@REDACTED (Erik Reitsma) Date: Mon, 30 Jun 2008 08:45:07 +0200 Subject: [erlang-questions] Question about message passing paradigm In-Reply-To: <6c2563b20806290342o2da73957t15cbbc5123e7dcc2@mail.gmail.com> References: <6e7d53010806282206s4fc36dcei9f99107f4ee6ac30@mail.gmail.com> <6c2563b20806290342o2da73957t15cbbc5123e7dcc2@mail.gmail.com> Message-ID: <110BA8ACEE682C479D0B008B6BE4AEB10696E92A@esealmw107.eemea.ericsson.se> What about: 4. Send a message to A, containing both the actual request for A, and a function FA to be executed by A. This function FA will send a message to B, containing the actual request for A, and a function FB to be executed by B. This function FB will send a message to C, containing the actual request for C, and a function FC to be executed by C. This function FC will send a message to D, containing the actual request for D, and possibly a function FD that does nothing. All the processes wait for the answer to their messages, of course. This way, A will not process anything (and therefore not change state), while waiting for B, which waits for C, which waits for D. B will not change state while waiting for C and D, and C will not change state while waiting for D. This is like locking A, then locking B, then C, then D, then releasing D, C, B and A. Beware of cyclic "locks" in this approach! *Erik. ________________________________ From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Edwin Fine Sent: Sunday, June 29, 2008 12:42 PM To: Mike T Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] Question about message passing paradigm I'm a bit of an Erlang newbie so the gurus may come up with something much better than this, but I can think of these possible approaches: 1. Create a gen_fsm that controls all the collections. The collections could be ETS tables or gen_servers wrapping ETS tables. Under normal use, messages are sent to the fsm to update the collections individually. When the time comes to require consistency across the collections, send a message to the fsm to get the collective state data. The fsm goes into a different state while it gathers the data. This state would reject requests to update the collections (or wait until the state changes), although reads would still be allowed. On getting the result, the state changes back to allow updates again. 2. Create a memory-only Mnesia table for each collection, and use Mnesia transactions to get the multiple values atomically. 3. Change the architecture of the current lock-oriented program to make better use of Erlang's features. Hope this helps. 2008/6/29 Mike T : Hello, I was not sure where best to post this and hope someone is able to help with this question. After being bitten by the pitfalls of lock-oriented multi threading I am interested in switching to message passing oriented concurrency. I have read that erlang has a per-process ordering guarantee (that is if A sends messages 1 and 2 to B 1 will arrive before 2 at B. However, there is no guarantee that messages from C and D will not be placed in between 1 and 2. So my question is this: In my current lock oriented program design I have threads dedicated to managing different collections and actions on those collections (hashmaps), lets call them A, B, C, and D. At several points in the program one thread needs to get data from the other collections and make a decision based on the collective state of the values it gathers form those other collections. An example situation would be A needing data from B, C, and D. In that case A would attempt to lock A, B, C, and D then gather the items it needs and release the lock. How would I accomplish this same task in a message passing manner? I had though of doing: A sends a message to B, C, and D asking for the data. However, B, C, and D may have each changed independently of each other by the time they receive the request for data and/or by the time they are able to send the message to A. So, how do you deal with situations like this in a message passing paradigm? _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://www.erlang.org/mailman/listinfo/erlang-questions -- The great enemy of the truth is very often not the lie -- deliberate, contrived and dishonest, but the myth, persistent, persuasive, and unrealistic. Belief in myths allows the comfort of opinion without the discomfort of thought. John F. Kennedy 35th president of US 1961-1963 (1917 - 1963) -------------- next part -------------- An HTML attachment was scrubbed... URL: From richardc@REDACTED Mon Jun 30 09:25:46 2008 From: richardc@REDACTED (Richard Carlsson) Date: Mon, 30 Jun 2008 09:25:46 +0200 Subject: [erlang-questions] [clarify] edoc: using @spec tag for a fun type In-Reply-To: <4867D68B.1050800@gmail.com> References: <4867D68B.1050800@gmail.com> Message-ID: <48688A7A.60601@it.uu.se> Csaba Hoch wrote: > if I define a fun type with @type, can I use > it when I define a function with @spec? > > I mean something like this: > > %% @type number_fun = ((number()) -> number()). > > %% @spec f = number_fun() > f(X) -> > 2*X. > > But my version doesn't work. > Is it possible to do something similar? No, this is currently not possible. I have made a note of this and will consider adding it in the future. Thanks. /Richard From vychodil.hynek@REDACTED Mon Jun 30 10:49:32 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Mon, 30 Jun 2008 10:49:32 +0200 Subject: [erlang-questions] Question about message passing paradigm In-Reply-To: <6e7d53010806282206s4fc36dcei9f99107f4ee6ac30@mail.gmail.com> References: <6e7d53010806282206s4fc36dcei9f99107f4ee6ac30@mail.gmail.com> Message-ID: <4d08db370806300149x70ac8c15q135efc8b992d3070@mail.gmail.com> Change your design from pull to push. Make servers which collect data and do something instead "request" data or actions. 2008/6/29 Mike T : > Hello, > > I was not sure where best to post this and hope someone is able to > help with this question. > > After being bitten by the pitfalls of lock-oriented multi threading I > am interested in switching to message passing oriented concurrency. > > I have read that erlang has a per-process ordering guarantee (that is > if A sends messages 1 and 2 to B 1 will arrive before 2 at B. However, > there is no guarantee that messages from C and D will not be placed in > between 1 and 2. > > So my question is this: > In my current lock oriented program design I have threads dedicated to > managing different collections and actions on those collections > (hashmaps), lets call them A, B, C, and D. At several points in the > program one thread needs to get data from the other collections and > make a decision based on the collective state of the values it gathers > form those other collections. > > An example situation would be A needing data from B, C, and D. In that > case A would attempt to lock A, B, C, and D then gather the items it > needs and release the lock. > > How would I accomplish this same task in a message passing manner? > > I had though of doing: A sends a message to B, C, and D asking for the > data. However, B, C, and D may have each changed independently of each > other by the time they receive the request for data and/or by the time > they are able to send the message to A. So, how do you deal with > situations like this in a message passing paradigm? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Mon Jun 30 11:39:57 2008 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 30 Jun 2008 11:39:57 +0200 Subject: [erlang-questions] wkipedia rendering engine Message-ID: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> Hi, I was at the erlang exchange and heard the *magnificant* talk "Building a transactional distributed data store with Erlang", by Alexander Reinefeld. I'll be blogging this as soon as I have the URL of the video of the talk. (in advance of this there was talk at the google conference on scalability http://video.google.com/videoplay?docid=-6526287646296437003&q=erlang+scalable&ei=cZ9oSLiDNIiCiwLL9fGwCA&hl=en oh and they also seem to have won the SCALE 2008 prize at the CCGrid conferense in Lyon but there is zero publicity about this AFAICS ) We (collectively) promised to help Alexander - I promised to provide him with a rendering engine (in Erlang) for the wikipedia markup language. Before I start hacking has anybody done this before? /Joe Armstrong From erlang@REDACTED Mon Jun 30 11:50:17 2008 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 30 Jun 2008 11:50:17 +0200 Subject: [erlang-questions] Project volunteers Message-ID: <9b08084c0806300250l382304edlf9b279800fb855d0@mail.gmail.com> Hi Guys, I've been at the erlang exchange and come back with a headful of ideas. I have an idea for a fun project. Make an offline stand-alone version of the wikipedia for places without internet access. Distribute to the world. I thought to use the following: - erlang - coutchDB - mochiWeb Jobs to do: - convert wikipedia dumps (mySQL format) to coutchDB - make rendering engine to convert wiki text to HTML - compress data dumps to make entiore wikipedia as small as possible - shoehorn into a low-power "one laptop for every child" computer - make distruibution package - release manager (set up groups) - write documentation /Joe Armstrong From andreengels@REDACTED Mon Jun 30 12:39:15 2008 From: andreengels@REDACTED (Andre Engels) Date: Mon, 30 Jun 2008 12:39:15 +0200 Subject: [erlang-questions] wkipedia rendering engine In-Reply-To: <6faf39c90806300338x5a7b9c20ha77ca939559196ea@mail.gmail.com> References: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> <6faf39c90806300338x5a7b9c20ha77ca939559196ea@mail.gmail.com> Message-ID: <6faf39c90806300339x367fa558l9cb919bbd693e79b@mail.gmail.com> On Mon, Jun 30, 2008 at 11:39 AM, Joe Armstrong wrote: > We (collectively) promised to help Alexander - I promised to provide him with a > rendering engine (in Erlang) for the wikipedia markup language. > > Before I start hacking has anybody done this before? What exactly do you mean by a 'rendering engine'? Translating the markup language (its name is Mediawiki, by the way) to something else? It's not a trivial task you have set yourself. There are some elements that are quite complex, for example the fact that '' is italics and ''' is bold. Notice the difference between: '''this is bold''' '''this is italic, starting with a ' '' '''this is bold '' and this part italic as well ''''' Also deciding on what point of the analysis to expand {{templates}} can lead the same code to get very different results. -- Andre Engels, andreengels@REDACTED ICQ: 6260644 -- Skype: a_engels From sonoflilit@REDACTED Mon Jun 30 12:52:11 2008 From: sonoflilit@REDACTED (Aur Saraf) Date: Mon, 30 Jun 2008 13:52:11 +0300 Subject: [erlang-questions] Project volunteers In-Reply-To: <9b08084c0806300250l382304edlf9b279800fb855d0@mail.gmail.com> References: <9b08084c0806300250l382304edlf9b279800fb855d0@mail.gmail.com> Message-ID: On Mon, Jun 30, 2008 at 12:50 PM, Joe Armstrong wrote: > Hi Guys, > > I've been at the erlang exchange and come back with a headful of ideas. > > I have an idea for a fun project. > > Make an offline stand-alone version of the wikipedia for places > without internet access. > Distribute to the world. > > I thought to use the following: > > - erlang > - coutchDB > - mochiWeb > > Jobs to do: > > - convert wikipedia dumps (mySQL format) to coutchDB > - make rendering engine to convert wiki text to HTML > - compress data dumps to make entiore wikipedia as small as possible > - shoehorn into a low-power "one laptop for every child" computer > - make distruibution package > - release manager (set up groups) > - write documentation I can't see why Erlang is the right tool for the job. Can you elaborate? -- Aur From erlang@REDACTED Mon Jun 30 12:53:43 2008 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 30 Jun 2008 12:53:43 +0200 Subject: [erlang-questions] wkipedia rendering engine In-Reply-To: <6faf39c90806300338x5a7b9c20ha77ca939559196ea@mail.gmail.com> References: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> <6faf39c90806300338x5a7b9c20ha77ca939559196ea@mail.gmail.com> Message-ID: <9b08084c0806300353s5c03cdbeh9b02d96990980e86@mail.gmail.com> On Mon, Jun 30, 2008 at 12:38 PM, Andre Engels wrote: > On Mon, Jun 30, 2008 at 11:39 AM, Joe Armstrong wrote: > >> We (collectively) promised to help Alexander - I promised to provide him with a >> rendering engine (in Erlang) for the wikipedia markup language. >> >> Before I start hacking has anybody done this before? > > What exactly do you mean by a 'rendering engine'? Translating the > markup language (its name is Mediawiki, by the way) to something else? I want a number of functions mediaWiki_to_rtf(bin()) -> rtf(). rtf_to_html(rtf()) -> html(). rtf_to_pdf(rtf()) -> pdf() etc. where rtf(), html() pdf() are abstract datav types representing (abstracted) rich text, html, and pdf() etc. The rendering engine is a wrapper round these routines to display ther result in a browser or generate PDF etc. > > It's not a trivial task you have set yourself. There are some elements > that are quite complex, for example the fact that '' is italics and > ''' is bold. Notice the difference between: > > '''this is bold''' > > '''this is italic, starting with a ' '' > > '''this is bold '' and this part italic as well ''''' > This is almost trivial :-) > Also deciding on what point of the analysis to expand {{templates}} > can lead the same code to get very different results. Glurk - you mean it's *undefined* - wow - I'll guess I'll discover this /Joe From jan@REDACTED Mon Jun 30 12:54:04 2008 From: jan@REDACTED (Jan Lehnardt) Date: Mon, 30 Jun 2008 12:54:04 +0200 Subject: [erlang-questions] Project volunteers In-Reply-To: <9b08084c0806300250l382304edlf9b279800fb855d0@mail.gmail.com> References: <9b08084c0806300250l382304edlf9b279800fb855d0@mail.gmail.com> Message-ID: <474B7098-EEAE-4DB7-972A-D7FBFF3C6184@apache.org> Hey Joe, This idea is a fun project very much indeed. And I can be of service with any CouchDB related problem. One issue I can see with CouchDB is that it trades in disk space for anything else (reliability, fault tolerance, speed). So a high- volume data project on a low-level (small disk) computer might not be the best fit. With the raw data storage CouchDB will use less than two times the amount of actual data. This might be a deal-breaker. Also, CouchDB uses so called 'views' for fast data access via b-tree indexes and those can add up to even more data in store. MacOS X comes with a facility to create compressed disk images, maybe this is available elsewhere. If we put CouchDB's (or any) data directory onto a compressed disk image, we might get quite far with the data compression. On the web-server end: CouchDB already comes with MochiWeb and lets you serve static data. What we can do here now is a small HTML+ JavaScript GUI that gets served from CouchDB that then can use Ajax to pull the actual contents from CouchDB. The catch here would be creating a proper web GUI without too much ajaxy fancyness and just using Ajax for data retrieval. That would mean though, that we need to either preprocess the Wikipdia contents into something the ajax GUI can use or it would need to do the wiki->html conversion. Probably not a good idea either way :) In any case, CouchDB can be used as dumb data store just as well with a custom MochiWeb frontend. Cheers Jan -- PS: It's "CouchDB", no 't' in there. It stands for "Cluster Of Unreliable Commodity Hardware" :-) On Jun 30, 2008, at 11:50, Joe Armstrong wrote: > Hi Guys, > > I've been at the erlang exchange and come back with a headful of > ideas. > > I have an idea for a fun project. > > Make an offline stand-alone version of the wikipedia for places > without internet access. > Distribute to the world. > > I thought to use the following: > > - erlang > - coutchDB > - mochiWeb > > Jobs to do: > > - convert wikipedia dumps (mySQL format) to coutchDB > - make rendering engine to convert wiki text to HTML > - compress data dumps to make entiore wikipedia as small as > possible > - shoehorn into a low-power "one laptop for every child" computer > - make distruibution package > - release manager (set up groups) > - write documentation > > > /Joe Armstrong > From erlang@REDACTED Mon Jun 30 12:58:38 2008 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 30 Jun 2008 12:58:38 +0200 Subject: [erlang-questions] wkipedia rendering engine In-Reply-To: <6faf39c90806300338x5a7b9c20ha77ca939559196ea@mail.gmail.com> References: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> <6faf39c90806300338x5a7b9c20ha77ca939559196ea@mail.gmail.com> Message-ID: <9b08084c0806300358h2478968cie3566ca7f7c99e59@mail.gmail.com> On Mon, Jun 30, 2008 at 12:38 PM, Andre Engels wrote: > On Mon, Jun 30, 2008 at 11:39 AM, Joe Armstrong wrote: > >> We (collectively) promised to help Alexander - I promised to provide him with a >> rendering engine (in Erlang) for the wikipedia markup language. >> Thanks - I didn't know the name of the format - seems like the processof parsing is reasonably easy -- see http://www.mediawiki.org/wiki/Markup_spec#The_Markup_Language It seems pretty amazing that there is no formal specifiation of the grammar of the markup language and that this is decided *after* there are a few quadzillion pages of markup text :-) /J >> Before I start hacking has anybody done this before? > > What exactly do you mean by a 'rendering engine'? Translating the > markup language (its name is Mediawiki, by the way) to something else? > > It's not a trivial task you have set yourself. There are some elements > that are quite complex, for example the fact that '' is italics and > ''' is bold. Notice the difference between: > > '''this is bold''' > > '''this is italic, starting with a ' '' > > '''this is bold '' and this part italic as well ''''' > > Also deciding on what point of the analysis to expand {{templates}} > can lead the same code to get very different results. > > -- > Andre Engels, andreengels@REDACTED > ICQ: 6260644 -- Skype: a_engels > -- fra@REDACTED; ingvar.akesson@REDACTED [Kopia av detta meddelande skickas till FRA f?r ?vervaknings?ndam?l. De vill ju ?nd? l?sa min e-post.] [A copy of this mail has been sent to FRA for monitoring purposes. FRA wants to read all my e-mail and have been allowed to do by the Swedish parliment - in violation of article 12 of the UN Universal Declaration of Human Rights] From andreengels@REDACTED Mon Jun 30 12:59:25 2008 From: andreengels@REDACTED (Andre Engels) Date: Mon, 30 Jun 2008 12:59:25 +0200 Subject: [erlang-questions] wkipedia rendering engine In-Reply-To: <9b08084c0806300353s5c03cdbeh9b02d96990980e86@mail.gmail.com> References: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> <6faf39c90806300338x5a7b9c20ha77ca939559196ea@mail.gmail.com> <9b08084c0806300353s5c03cdbeh9b02d96990980e86@mail.gmail.com> Message-ID: <6faf39c90806300359r446f4e0fkf0dfe44ccf34df32@mail.gmail.com> On Mon, Jun 30, 2008 at 12:53 PM, Joe Armstrong wrote: >> Also deciding on what point of the analysis to expand {{templates}} >> can lead the same code to get very different results. > > Glurk - you mean it's *undefined* - wow - I'll guess I'll discover this Might well be undefined in the sense that the only definition is "what the (php) code does is right", I'm not sure. The code seems to be in http://svn.wikimedia.org/viewvc/mediawiki/branches/stable/phase3/includes/OutputPage.php?view=log -- Andre Engels, andreengels@REDACTED ICQ: 6260644 -- Skype: a_engels From erlang@REDACTED Mon Jun 30 13:02:18 2008 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 30 Jun 2008 13:02:18 +0200 Subject: [erlang-questions] Project volunteers In-Reply-To: References: <9b08084c0806300250l382304edlf9b279800fb855d0@mail.gmail.com> Message-ID: <9b08084c0806300402l7ee270f6w2db470f834d78b1@mail.gmail.com> On Mon, Jun 30, 2008 at 12:52 PM, Aur Saraf wrote: > On Mon, Jun 30, 2008 at 12:50 PM, Joe Armstrong wrote: >> Hi Guys, >> >> I've been at the erlang exchange and come back with a headful of ideas. >> >> I have an idea for a fun project. >> >> Make an offline stand-alone version of the wikipedia for places >> without internet access. >> Distribute to the world. >> >> I thought to use the following: >> >> - erlang >> - coutchDB >> - mochiWeb >> >> Jobs to do: >> >> - convert wikipedia dumps (mySQL format) to coutchDB >> - make rendering engine to convert wiki text to HTML >> - compress data dumps to make entiore wikipedia as small as possible >> - shoehorn into a low-power "one laptop for every child" computer >> - make distruibution package >> - release manager (set up groups) >> - write documentation > > > I can't see why Erlang is the right tool for the job. Can you > elaborate? I have no idea if Erlang is the *best* tool for the job and I don't care. I would be *delighted* if other people do this in java/ruby/haskell/ whatever turns them on. I'll do mine in Erlang - I want volunteers so I post to the Erlang group. I don't want to spend more time motivating the choise of language than would take to write the program. Cheers /Joe Armstrong > > -- Aur > -- fra@REDACTED; ingvar.akesson@REDACTED [Kopia av detta meddelande skickas till FRA f?r ?vervaknings?ndam?l. De vill ju ?nd? l?sa min e-post.] [A copy of this mail has been sent to FRA for monitoring purposes. FRA wants to read all my e-mail and have been allowed to do by the Swedish parliment - in violation of article 12 of the UN Universal Declaration of Human Rights] From vychodil.hynek@REDACTED Mon Jun 30 13:12:16 2008 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Mon, 30 Jun 2008 13:12:16 +0200 Subject: [erlang-questions] wkipedia rendering engine In-Reply-To: <9b08084c0806300358h2478968cie3566ca7f7c99e59@mail.gmail.com> References: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> <6faf39c90806300338x5a7b9c20ha77ca939559196ea@mail.gmail.com> <9b08084c0806300358h2478968cie3566ca7f7c99e59@mail.gmail.com> Message-ID: <4d08db370806300412q1a1ddf9aq509c65ad93b5db5b@mail.gmail.com> On Mon, Jun 30, 2008 at 12:58 PM, Joe Armstrong wrote: > On Mon, Jun 30, 2008 at 12:38 PM, Andre Engels > wrote: > > On Mon, Jun 30, 2008 at 11:39 AM, Joe Armstrong > wrote: > > > >> We (collectively) promised to help Alexander - I promised to provide him > with a > >> rendering engine (in Erlang) for the wikipedia markup language. > >> > > Thanks - I didn't know the name of the format - seems like the > processof parsing is > reasonably easy -- see > > http://www.mediawiki.org/wiki/Markup_spec#The_Markup_Language > > > It seems pretty amazing that there is no formal specifiation of the > grammar of the markup language and that this > is decided *after* there are a few quadzillion pages of markup text :-) > > /J {joke, "It's not amazing, it's wikiworld!"}. > > > >> Before I start hacking has anybody done this before? > > > > What exactly do you mean by a 'rendering engine'? Translating the > > markup language (its name is Mediawiki, by the way) to something else? > > > > It's not a trivial task you have set yourself. There are some elements > > that are quite complex, for example the fact that '' is italics and > > ''' is bold. Notice the difference between: > > > > '''this is bold''' > > > > '''this is italic, starting with a ' '' > > > > '''this is bold '' and this part italic as well ''''' > > > > Also deciding on what point of the analysis to expand {{templates}} > > can lead the same code to get very different results. > > > > -- > > Andre Engels, andreengels@REDACTED > > ICQ: 6260644 -- Skype: a_engels > > > > > > -- > fra@REDACTED; ingvar.akesson@REDACTED > > [Kopia av detta meddelande skickas till FRA f?r ?vervaknings?ndam?l. > De vill ju ?nd? l?sa min e-post.] > > [A copy of this mail has been sent to > FRA for monitoring purposes. FRA wants to read all my e-mail and have > been allowed to do by the Swedish parliment - in violation of article > 12 of the UN Universal Declaration of Human Rights] > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Mon Jun 30 13:23:05 2008 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 30 Jun 2008 13:23:05 +0200 Subject: [erlang-questions] wkipedia rendering engine In-Reply-To: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> References: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> Message-ID: <9b08084c0806300423g4269b1c4v8822a9184cf86878@mail.gmail.com> Followup to myself. I guess the wikipedia is stored internally in the format that is presented to the users for editing ie in MetaWiki markup language. Is there a REST interface so that I can retreive the latest version of the MetaWiki markup for a specific page with, for example, a wget command. Has anybody made an erlang interface to scrape individual pages from the wikipedia - or to bulk convert the entire wikipedia to erlang terms :-) /Joe On Mon, Jun 30, 2008 at 11:39 AM, Joe Armstrong wrote: > Hi, > > I was at the erlang exchange and heard the *magnificant* talk > > "Building a transactional distributed data store with Erlang", by > Alexander Reinefeld. > > I'll be blogging this as soon as I have the URL of the video of the talk. > > (in advance of this there was talk at the google conference on scalability > > http://video.google.com/videoplay?docid=-6526287646296437003&q=erlang+scalable&ei=cZ9oSLiDNIiCiwLL9fGwCA&hl=en > > oh and they also seem to have won the SCALE 2008 prize at the > CCGrid conferense in Lyon but there is zero publicity about this AFAICS > ) > > We (collectively) promised to help Alexander - I promised to provide him with a > rendering engine (in Erlang) for the wikipedia markup language. > > Before I start hacking has anybody done this before? > > /Joe Armstrong > -- fra@REDACTED; ingvar.akesson@REDACTED [Kopia av detta meddelande skickas till FRA f?r ?vervaknings?ndam?l. De vill ju ?nd? l?sa min e-post.] [A copy of this mail has been sent to FRA for monitoring purposes. FRA wants to read all my e-mail and have been allowed to do by the Swedish parliment - in violation of article 12 of the UN Universal Declaration of Human Rights] From andreengels@REDACTED Mon Jun 30 13:33:00 2008 From: andreengels@REDACTED (Andre Engels) Date: Mon, 30 Jun 2008 13:33:00 +0200 Subject: [erlang-questions] wkipedia rendering engine In-Reply-To: <9b08084c0806300423g4269b1c4v8822a9184cf86878@mail.gmail.com> References: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> <9b08084c0806300423g4269b1c4v8822a9184cf86878@mail.gmail.com> Message-ID: <6faf39c90806300433l7b8ceadw93b381aa4b2d5cd6@mail.gmail.com> On Mon, Jun 30, 2008 at 1:23 PM, Joe Armstrong wrote: > Is there a REST interface so that I can retreive the latest version of > the MetaWiki markup for a specific page with, for example, > a wget command. What's a REST interface? There's several ways to get the MediaWiki markup of a specific page: * Go to the edit page; it contains the latest version of the markup * Go to [[Special:Export]], where you can get either the current version or all versions of a number of pages, in XML * At http://download.wikimedia.org/backup-index.html are the complete database dumps of the various wikis; the content of the page is in one of the tables -- Andre Engels, andreengels@REDACTED ICQ: 6260644 -- Skype: a_engels From jan@REDACTED Mon Jun 30 13:36:25 2008 From: jan@REDACTED (Jan Lehnardt) Date: Mon, 30 Jun 2008 13:36:25 +0200 Subject: [erlang-questions] wkipedia rendering engine In-Reply-To: <9b08084c0806300423g4269b1c4v8822a9184cf86878@mail.gmail.com> References: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> <9b08084c0806300423g4269b1c4v8822a9184cf86878@mail.gmail.com> Message-ID: <9C2A66F4-7037-4A55-9A47-DB36200D7932@apache.org> On Jun 30, 2008, at 13:23, Joe Armstrong wrote: > Is there a REST interface so that I can retreive the latest version of > the MetaWiki markup for a specific page with, for example, > a wget command. You can get bulk dumps http://en.wikipedia.org/wiki/Wikipedia:Database_download#Where_do_I_get ... Why would you do individual scraping? In order to keep up to date with changes that happened between the last dump and now()? Cheers Jan -- > Has anybody made an erlang interface to scrape individual pages from > the wikipedia - or to bulk convert the entire > wikipedia to erlang terms :-) > > /Joe > > > > On Mon, Jun 30, 2008 at 11:39 AM, Joe Armstrong > wrote: >> Hi, >> >> I was at the erlang exchange and heard the *magnificant* talk >> >> "Building a transactional distributed data store with Erlang", by >> Alexander Reinefeld. >> >> I'll be blogging this as soon as I have the URL of the video of the >> talk. >> >> (in advance of this there was talk at the google conference on >> scalability >> >> http://video.google.com/videoplay?docid=-6526287646296437003&q=erlang+scalable&ei=cZ9oSLiDNIiCiwLL9fGwCA&hl=en >> >> oh and they also seem to have won the SCALE 2008 prize at the >> CCGrid conferense in Lyon but there is zero publicity about this >> AFAICS >> ) >> >> We (collectively) promised to help Alexander - I promised to >> provide him with a >> rendering engine (in Erlang) for the wikipedia markup language. >> >> Before I start hacking has anybody done this before? >> >> /Joe Armstrong >> > > > > -- > fra@REDACTED; ingvar.akesson@REDACTED > > [Kopia av detta meddelande skickas till FRA f?r ?vervaknings?ndam?l. > De vill ju ?nd? l?sa min e-post.] > > [A copy of this mail has been sent to > FRA for monitoring purposes. FRA wants to read all my e-mail and have > been allowed to do by the Swedish parliment - in violation of article > 12 of the UN Universal Declaration of Human Rights] > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://www.erlang.org/mailman/listinfo/erlang-questions > From erlang@REDACTED Mon Jun 30 13:36:41 2008 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 30 Jun 2008 13:36:41 +0200 Subject: [erlang-questions] wkipedia rendering engine In-Reply-To: <6faf39c90806300433l7b8ceadw93b381aa4b2d5cd6@mail.gmail.com> References: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> <9b08084c0806300423g4269b1c4v8822a9184cf86878@mail.gmail.com> <6faf39c90806300433l7b8ceadw93b381aa4b2d5cd6@mail.gmail.com> Message-ID: <9b08084c0806300436j29d42923tea4660f9c3892202@mail.gmail.com> On Mon, Jun 30, 2008 at 1:33 PM, Andre Engels wrote: > On Mon, Jun 30, 2008 at 1:23 PM, Joe Armstrong wrote: > >> Is there a REST interface so that I can retreive the latest version of >> the MetaWiki markup for a specific page with, for example, >> a wget command. > > What's a REST interface? http://en.wikipedia.org/wiki/Representational_State_Transfer /J > There's several ways to get the MediaWiki > markup of a specific page: > * Go to the edit page; it contains the latest version of the markup > * Go to [[Special:Export]], where you can get either the current > version or all versions of a number of pages, in XML > * At http://download.wikimedia.org/backup-index.html are the complete > database dumps of the various wikis; the content of the page is in one > of the tables > > > -- > Andre Engels, andreengels@REDACTED > ICQ: 6260644 -- Skype: a_engels > -- fra@REDACTED; ingvar.akesson@REDACTED [Kopia av detta meddelande skickas till FRA f?r ?vervaknings?ndam?l. De vill ju ?nd? l?sa min e-post.] [A copy of this mail has been sent to FRA for monitoring purposes. FRA wants to read all my e-mail and have been allowed to do by the Swedish parliment - in violation of article 12 of the UN Universal Declaration of Human Rights] From erlang@REDACTED Mon Jun 30 13:40:28 2008 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 30 Jun 2008 13:40:28 +0200 Subject: [erlang-questions] wkipedia rendering engine In-Reply-To: <9C2A66F4-7037-4A55-9A47-DB36200D7932@apache.org> References: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> <9b08084c0806300423g4269b1c4v8822a9184cf86878@mail.gmail.com> <9C2A66F4-7037-4A55-9A47-DB36200D7932@apache.org> Message-ID: <9b08084c0806300440u8fa9494q343abcbeafca8b62@mail.gmail.com> On Mon, Jun 30, 2008 at 1:36 PM, Jan Lehnardt wrote: > On Jun 30, 2008, at 13:23, Joe Armstrong wrote: >> >> Is there a REST interface so that I can retreive the latest version of >> the MetaWiki markup for a specific page with, for example, >> a wget command. > > You can get bulk dumps > http://en.wikipedia.org/wiki/Wikipedia:Database_download#Where_do_I_get... > > Why would you do individual scraping? In order to keep up to date with > changes that happened between the last dump and now()? > To get a few test cases to test my parser on *before* download the entire thing. Also I suspect the dumps are in MySQL format with xml junk - so it might not be a trival job to extract the raw data. I (presumably) will have to install MySQL and turn some XML stuff into the raw data (just guessing here) - thought that could be a job for a volunteer :-) /Joe > Cheers > Jan > -- > >> Has anybody made an erlang interface to scrape individual pages from >> the wikipedia - or to bulk convert the entire >> wikipedia to erlang terms :-) >> >> /Joe >> >> >> >> On Mon, Jun 30, 2008 at 11:39 AM, Joe Armstrong wrote: >>> >>> Hi, >>> >>> I was at the erlang exchange and heard the *magnificant* talk >>> >>> "Building a transactional distributed data store with Erlang", by >>> Alexander Reinefeld. >>> >>> I'll be blogging this as soon as I have the URL of the video of the talk. >>> >>> (in advance of this there was talk at the google conference on >>> scalability >>> >>> >>> http://video.google.com/videoplay?docid=-6526287646296437003&q=erlang+scalable&ei=cZ9oSLiDNIiCiwLL9fGwCA&hl=en >>> >>> oh and they also seem to have won the SCALE 2008 prize at the >>> CCGrid conferense in Lyon but there is zero publicity about this AFAICS >>> ) >>> >>> We (collectively) promised to help Alexander - I promised to provide him >>> with a >>> rendering engine (in Erlang) for the wikipedia markup language. >>> >>> Before I start hacking has anybody done this before? >>> >>> /Joe Armstrong >>> >> >> >> >> -- >> fra@REDACTED; ingvar.akesson@REDACTED >> >> [Kopia av detta meddelande skickas till FRA f?r ?vervaknings?ndam?l. >> De vill ju ?nd? l?sa min e-post.] >> >> [A copy of this mail has been sent to >> FRA for monitoring purposes. FRA wants to read all my e-mail and have >> been allowed to do by the Swedish parliment - in violation of article >> 12 of the UN Universal Declaration of Human Rights] >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > -- fra@REDACTED; ingvar.akesson@REDACTED [Kopia av detta meddelande skickas till FRA f?r ?vervaknings?ndam?l. De vill ju ?nd? l?sa min e-post.] [A copy of this mail has been sent to FRA for monitoring purposes. FRA wants to read all my e-mail and have been allowed to do by the Swedish parliment - in violation of article 12 of the UN Universal Declaration of Human Rights] From schuett@REDACTED Mon Jun 30 14:13:36 2008 From: schuett@REDACTED (Thorsten Schuett) Date: Mon, 30 Jun 2008 14:13:36 +0200 Subject: [erlang-questions] wkipedia rendering engine In-Reply-To: <9b08084c0806300440u8fa9494q343abcbeafca8b62@mail.gmail.com> References: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> <9C2A66F4-7037-4A55-9A47-DB36200D7932@apache.org> <9b08084c0806300440u8fa9494q343abcbeafca8b62@mail.gmail.com> Message-ID: <200806301413.36785.schuett@zib.de> Hi all, as I am partially to blame for the noise around the wikirenderer, I will add my two cents. For our experiments, we used the XML dumps available at http://download.wikimedia.org. We have a small Java program which converts the XML dump to Erlang terms (http://www.zib.de/schuett/dumpreader.tgz). E.g. converting the bavarian dump: java -jar dumpreader.jar /home/schuett/barwiki-20080225-pages-meta-history.xml But you still have to parse the mediawiki text and convert it to HTML. For the last step we currently have two solutions: 1. Early experiments used flexbisonparse (http://svn.wikimedia.org/viewvc/mediawiki/trunk/flexbisonparse/) to convert the mediawiki text to XML and XSLT to convert the XML to HTML. 2. The current code is based on plog4u/bliki( see http://matheclipse.org/en/Java_Wikipedia_API) Thorsten On Monday 30 June 2008, Joe Armstrong wrote: > On Mon, Jun 30, 2008 at 1:36 PM, Jan Lehnardt wrote: > > On Jun 30, 2008, at 13:23, Joe Armstrong wrote: > >> Is there a REST interface so that I can retreive the latest version of > >> the MetaWiki markup for a specific page with, for example, > >> a wget command. > > > > You can get bulk dumps > > http://en.wikipedia.org/wiki/Wikipedia:Database_download#Where_do_I_get.. > >. > > > > Why would you do individual scraping? In order to keep up to date with > > changes that happened between the last dump and now()? > > To get a few test cases to test my parser on *before* download the entire > thing. > > Also I suspect the dumps are in MySQL format with xml junk - so it might > not be a trival job to extract the raw data. I (presumably) will have to > install MySQL and > turn some XML stuff into the raw data (just guessing here) - thought > that could be a job for a > volunteer :-) > > /Joe > > > Cheers > > Jan > > -- > > > >> Has anybody made an erlang interface to scrape individual pages from > >> the wikipedia - or to bulk convert the entire > >> wikipedia to erlang terms :-) > >> > >> /Joe > >> > >> On Mon, Jun 30, 2008 at 11:39 AM, Joe Armstrong wrote: > >>> Hi, > >>> > >>> I was at the erlang exchange and heard the *magnificant* talk > >>> > >>> "Building a transactional distributed data store with Erlang", by > >>> Alexander Reinefeld. > >>> > >>> I'll be blogging this as soon as I have the URL of the video of the > >>> talk. > >>> > >>> (in advance of this there was talk at the google conference on > >>> scalability > >>> > >>> > >>> http://video.google.com/videoplay?docid=-6526287646296437003&q=erlang+s > >>>calable&ei=cZ9oSLiDNIiCiwLL9fGwCA&hl=en > >>> > >>> oh and they also seem to have won the SCALE 2008 prize at the > >>> CCGrid conferense in Lyon but there is zero publicity about this AFAICS > >>> ) > >>> > >>> We (collectively) promised to help Alexander - I promised to provide > >>> him with a > >>> rendering engine (in Erlang) for the wikipedia markup language. > >>> > >>> Before I start hacking has anybody done this before? > >>> > >>> /Joe Armstrong > >> > >> -- > >> fra@REDACTED; ingvar.akesson@REDACTED > >> > >> [Kopia av detta meddelande skickas till FRA f?r ?vervaknings?ndam?l. > >> De vill ju ?nd? l?sa min e-post.] > >> > >> [A copy of this mail has been sent to > >> FRA for monitoring purposes. FRA wants to read all my e-mail and have > >> been allowed to do by the Swedish parliment - in violation of article > >> 12 of the UN Universal Declaration of Human Rights] > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://www.erlang.org/mailman/listinfo/erlang-questions From erlang@REDACTED Mon Jun 30 14:18:11 2008 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 30 Jun 2008 14:18:11 +0200 Subject: [erlang-questions] wkipedia rendering engine In-Reply-To: <728e3ef50806300458r1117255akec6196e71883be1c@mail.gmail.com> References: <9b08084c0806300239t59378f45nded85601087358d6@mail.gmail.com> <9b08084c0806300423g4269b1c4v8822a9184cf86878@mail.gmail.com> <9C2A66F4-7037-4A55-9A47-DB36200D7932@apache.org> <9b08084c0806300440u8fa9494q343abcbeafca8b62@mail.gmail.com> <728e3ef50806300458r1117255akec6196e71883be1c@mail.gmail.com> Message-ID: <9b08084c0806300518g781e9c8cj2a7f69130f60ea60@mail.gmail.com> On Mon, Jun 30, 2008 at 1:58 PM, Alain O'Dea wrote: > On Mon, Jun 30, 2008 at 9:10 AM, Joe Armstrong wrote: >> On Mon, Jun 30, 2008 at 1:36 PM, Jan Lehnardt wrote: >>> On Jun 30, 2008, at 13:23, Joe Armstrong wrote: >>>> >>>> Is there a REST interface so that I can retreive the latest version of >>>> the MetaWiki markup for a specific page with, for example, >>>> a wget command. >>> >>> You can get bulk dumps >>> http://en.wikipedia.org/wiki/Wikipedia:Database_download#Where_do_I_get... >>> >>> Why would you do individual scraping? In order to keep up to date with >>> changes that happened between the last dump and now()? >>> >> >> To get a few test cases to test my parser on *before* download the entire thing. >> >> Also I suspect the dumps are in MySQL format with xml junk - so it might not be >> a trival job to extract the raw data. I (presumably) will have to >> install MySQL and >> turn some XML stuff into the raw data (just guessing here) - thought >> that could be a job for a >> volunteer :-) >> >> /Joe >> >> >>> Cheers >>> Jan >>> -- >>> >>>> Has anybody made an erlang interface to scrape individual pages from >>>> the wikipedia - or to bulk convert the entire >>>> wikipedia to erlang terms :-) >>>> >>>> /Joe >>>> >>>> >>>> >>>> On Mon, Jun 30, 2008 at 11:39 AM, Joe Armstrong wrote: >>>>> >>>>> Hi, >>>>> >>>>> I was at the erlang exchange and heard the *magnificant* talk >>>>> >>>>> "Building a transactional distributed data store with Erlang", by >>>>> Alexander Reinefeld. >>>>> >>>>> I'll be blogging this as soon as I have the URL of the video of the talk. >>>>> >>>>> (in advance of this there was talk at the google conference on >>>>> scalability >>>>> >>>>> >>>>> http://video.google.com/videoplay?docid=-6526287646296437003&q=erlang+scalable&ei=cZ9oSLiDNIiCiwLL9fGwCA&hl=en >>>>> >>>>> oh and they also seem to have won the SCALE 2008 prize at the >>>>> CCGrid conferense in Lyon but there is zero publicity about this AFAICS >>>>> ) >>>>> >>>>> We (collectively) promised to help Alexander - I promised to provide him >>>>> with a >>>>> rendering engine (in Erlang) for the wikipedia markup language. >>>>> >>>>> Before I start hacking has anybody done this before? >>>>> >>>>> /Joe Armstrong >>>>> >>>> >>>> >>>> >>>> -- >>>> fra@REDACTED; ingvar.akesson@REDACTED >>>> >>>> [Kopia av detta meddelande skickas till FRA f?r ?vervaknings?ndam?l. >>>> De vill ju ?nd? l?sa min e-post.] >>>> >>>> [A copy of this mail has been sent to >>>> FRA for monitoring purposes. FRA wants to read all my e-mail and have >>>> been allowed to do by the Swedish parliment - in violation of article >>>> 12 of the UN Universal Declaration of Human Rights] >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://www.erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >> >> >> >> -- >> fra@REDACTED; ingvar.akesson@REDACTED >> >> [Kopia av detta meddelande skickas till FRA f?r ?vervaknings?ndam?l. >> De vill ju ?nd? l?sa min e-post.] >> >> [A copy of this mail has been sent to >> FRA for monitoring purposes. FRA wants to read all my e-mail and have >> been allowed to do by the Swedish parliment - in violation of article >> 12 of the UN Universal Declaration of Human Rights] >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://www.erlang.org/mailman/listinfo/erlang-questions >> > > There is a REST interface, but it is not exactly machine-friendly. If > you request http://en.wikipedia.org/w/index.php?title= NAME>&action=edit with a topic name put in you will get an editor > page. For example > http://en.wikipedia.org/w/index.php?title=Erlang%20(programming%20language)&action=edit > brings up the editor page for the Erlang programming language. > > The raw MediaWiki markup is in a textarea with id "wpTextbox1", but > unfortunately I have been unable to get xmerl to extract it due to the > fact that the page is HTML and not well-formed XML. Seems to work - should be easy to extract the content Why bother with xmerl just scan the text for a constant string ...