From erlang@REDACTED Tue Sep 1 09:54:24 2015 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 1 Sep 2015 09:54:24 +0200 Subject: [erlang-questions] web proxy Message-ID: Hello, I want to record all traffic between my browser and the Internet. What I want is a record of 1) the request 2) the number of bytes in the response 3) some timing data When I fetch a web page. I'd like this data (if possible) in the form of event messages being sent to an Erlang program. Why do I want this? I was horrified by reading http://blog.lmorchard.com/2015/07/22/the-verge-web-sucks/ and wanted to accurately record exactly how much data is involved when I request a web page. The I want see how much is content and how much is crap. Any ideas /Joe From paul.joseph.davis@REDACTED Tue Sep 1 09:16:19 2015 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Tue, 1 Sep 2015 02:16:19 -0500 Subject: [erlang-questions] Cost of doing +sbwt? Message-ID: Hey everyone, Lately at IBM/Cloudant we've been doing cluster upgrades from R14B01 to 17.5 (don't ask) and we've been noticing a fairly decent increase in total CPU usage. The recent upgrades have actually included a two step upgrade between three versions which I'll call A, B, and C. Version A is our most recent production release on R14B01, version B is a compatibility release with minor changes that still runs on R14B01, and version C is exactly version B code except for the upgrade to the 17.5 emulator. Each upgrade in the series involves a rolling reboot of the entire cluster which involves anywhere from three to twenty or so nodes at this point. What we've observed is that each step in the upgrade process tends to increase the CPU usage across the cluster noticeably. There's some obvious CPU issues with a rolling reboot in our clusters that we expect but the end result as been a significant jump in system CPU usage between A and C. Generally speaking we're seeing A running around 6-12% system CPU and version C running around 50% system CPU usage. When I first saw this my first thought was that it was just the busy wait scheduling that has been discussed occasionally on this list. Testing had shown a slight decrease in performance between R14B01 and 17.5 but it was roughly within in the margin of error as well as not being the most awesome representation of real user load. Given that we pushed through some less busy clusters and everything appeared to match with my expectations, namely increased CPU usage without any observable effects to the outside world. Unfortunately we had two clusters that had previously been running fairly hot and after the upgrade response latencies jumped significantly. Internal metrics showed latencies increasing by an order of magnitude as well as having significantly higher variance. On one cluster we ended up rolling back from C to A to try and undo the latency issues which ended up having little effect on total latencies. We did manage to change which parts of the database were having latency issues but overall it was an unfun operational experience. So that's the background, here are some obvservations that I've been collecting to try and narrow down the issue. Our first data point was to try and look at strace. What we noticed was that scheduler threads seemed to spend an inordinate amount of time in futex system calls. An strace run on a scheduler thread showed more than 50% of time in futex sys calls. On one of our dedicated clusters we ended up disabling the async thread pool (brilliant idea via Adam Kocoloski). While this had a super awesome impact on observed latencies for the cluster, CPU usage remained roughly the same with ~50% of CPU usage attributed to system CPU. Of note this cluster is a 9 node cluster with dual hex cores with hyperthreading for a total of 24 scheduler threads as well as a four SSD RAID 10 array for disk. My general thought here was that while it didn't necessarily fix the underlying issue, SSDs without an async thread pool made things fast enough to get back to roughly normal on this cluster. The second major data point was a similarly spec'ed cluster except that instead of SSD's it had HDDs in the same RAID configuration. Attempting to help fix latencies we made the same change to disable the async thread pool. The change appeared to make the cluster faster but there wasn't a significant change in latencies for the cluster. There was an observable increase in throughput for the entire cluster but as in all distributed systems it was a bit hard to deconvolve the various signals. Given these two data points I decided to try looking at the VM flags that might be major behavior changes between R14B01 and 17.5. The flags I decided to try setting were "+sbwt none +secio false +sws legacy" (from memory). I applied those to one node on the MT cluster and it resulted in an immediate change from roughly fifty to six-ish percent on CPU usage. Once I saw the drastic change in CPU I replaced the change to just "+sbwt none" and saw no change in CPU usage which suggested that the sbwt change was entirely responsible for the change. After letting the cluster run I also tried using "+sbwt very_low" which resulted in a noticeable increase in system CPU but nowhere near the ~50% system CPU default. The MT cluster I did the initial +sbwt change on was the multi-tenant cluster so its hard to make too many conclusions given the roughly random work load. What I did do was check that very_low increased the system CPU usage noticeably and that default put it back to 50%. During these experiments our various latency metrics seemed to respond but distributed systems, so its hard to draw direct conclusions. However, given my observations I was pretty certain that the +sbwt setting was directly affecting system CPU usage. The general theory I've been working on is that something with Erlang's elevated system CPU usage is causing my observed latencies to increase. Given that, I set +sbwt none to one of our problematic clusters. The initial response to this change was a significant increase in throughput of background tasks on the cluster. Once those cleared out the cluster latencies have settled back into the previous R14B01 levels. My general question at this point is wether scheduler busy wait has an intrinsic cost that's not reflected in the documentation. From what I've seen is that we end up with a large amount of system CPU usage that dampens user CPU usage of beam.smp which in our case affects system latencies. We have a fairly intense port/file driver usage which may affect the issue. Disabling busy wait fixes things and that confuses me. Any thoughts or similar observations would be awesomely appreciated. I think I can see how +sbwt values affect a VM but I'm fairly confused why reverting back to R14 didn't rectify things. I'm really hoping someone remembers a scheduler bug around this area for that part of the story. I'll buy many beers at the next conference for anyone that makes me think I'm not insane in all this. From tuncer.ayaz@REDACTED Tue Sep 1 12:05:10 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Tue, 1 Sep 2015 12:05:10 +0200 Subject: [erlang-questions] web proxy In-Reply-To: References: Message-ID: On Tue, Sep 1, 2015 at 9:54 AM, Joe Armstrong wrote: > Hello, > > I want to record all traffic between my browser and the Internet. > > What I want is a record of > > 1) the request > 2) the number of bytes in the response > 3) some timing data > > When I fetch a web page. > > I'd like this data (if possible) in the form of event messages being sent > to an Erlang program. > > Why do I want this? > > I was horrified by reading > http://blog.lmorchard.com/2015/07/22/the-verge-web-sucks/ > and wanted to accurately record exactly how much data is involved when > I request a web page. The I want see how much is content and how much > is crap. > > Any ideas Browsers have (developer) profiling tools, which allow you to inspect the network traffic from the client's view. That should do the trick. From manuel@REDACTED Tue Sep 1 12:06:46 2015 From: manuel@REDACTED (=?UTF-8?Q?Manuel_Dur=C3=A1n_Aguete?=) Date: Tue, 1 Sep 2015 12:06:46 +0200 Subject: [erlang-questions] web proxy In-Reply-To: References: Message-ID: Hello, the tsung recorder ? ( http://tsung.erlang-projects.org/user_manual/proxy.html#http-and-webdav) Regards. 2015-09-01 9:54 GMT+02:00 Joe Armstrong : > Hello, > > I want to record all traffic between my browser and the Internet. > > What I want is a record of > > 1) the request > 2) the number of bytes in the response > 3) some timing data > > When I fetch a web page. > > I'd like this data (if possible) in the form of event messages being sent > to an Erlang program. > > Why do I want this? > > I was horrified by reading > http://blog.lmorchard.com/2015/07/22/the-verge-web-sucks/ > and wanted to accurately record exactly how much data is involved when > I request a web page. The I want see how much is content and how much > is crap. > > Any ideas > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Tue Sep 1 14:20:16 2015 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Tue, 1 Sep 2015 14:20:16 +0200 Subject: [erlang-questions] Cost of doing +sbwt? In-Reply-To: References: Message-ID: Hi, are you using +sbt? If not, try use +sbt with default +sbwt. If you observe CPU cost but without drop of latencies, there is possibilitty, that your load characteristic with busy wait provoke some pathologcal scheduler migration (perfomed by OS) which results in your latency problems. It is just blind guess but worth to try. Hynek On Tue, Sep 1, 2015 at 9:16 AM, Paul Davis wrote: > Hey everyone, > > Lately at IBM/Cloudant we've been doing cluster upgrades from R14B01 > to 17.5 (don't ask) and we've been noticing a fairly decent increase > in total CPU usage. The recent upgrades have actually included a two > step upgrade between three versions which I'll call A, B, and C. > Version A is our most recent production release on R14B01, version B > is a compatibility release with minor changes that still runs on > R14B01, and version C is exactly version B code except for the upgrade > to the 17.5 emulator. Each upgrade in the series involves a rolling > reboot of the entire cluster which involves anywhere from three to > twenty or so nodes at this point. > > What we've observed is that each step in the upgrade process tends to > increase the CPU usage across the cluster noticeably. There's some > obvious CPU issues with a rolling reboot in our clusters that we > expect but the end result as been a significant jump in system CPU > usage between A and C. Generally speaking we're seeing A running > around 6-12% system CPU and version C running around 50% system CPU > usage. > > When I first saw this my first thought was that it was just the busy > wait scheduling that has been discussed occasionally on this list. > Testing had shown a slight decrease in performance between R14B01 and > 17.5 but it was roughly within in the margin of error as well as not > being the most awesome representation of real user load. Given that we > pushed through some less busy clusters and everything appeared to > match with my expectations, namely increased CPU usage without any > observable effects to the outside world. > > Unfortunately we had two clusters that had previously been running > fairly hot and after the upgrade response latencies jumped > significantly. Internal metrics showed latencies increasing by an > order of magnitude as well as having significantly higher variance. > > On one cluster we ended up rolling back from C to A to try and undo > the latency issues which ended up having little effect on total > latencies. We did manage to change which parts of the database were > having latency issues but overall it was an unfun operational > experience. > > So that's the background, here are some obvservations that I've been > collecting to try and narrow down the issue. > > Our first data point was to try and look at strace. What we noticed > was that scheduler threads seemed to spend an inordinate amount of > time in futex system calls. An strace run on a scheduler thread showed > more than 50% of time in futex sys calls. > > On one of our dedicated clusters we ended up disabling the async > thread pool (brilliant idea via Adam Kocoloski). While this had a > super awesome impact on observed latencies for the cluster, CPU usage > remained roughly the same with ~50% of CPU usage attributed to system > CPU. Of note this cluster is a 9 node cluster with dual hex cores with > hyperthreading for a total of 24 scheduler threads as well as a four > SSD RAID 10 array for disk. My general thought here was that while it > didn't necessarily fix the underlying issue, SSDs without an async > thread pool made things fast enough to get back to roughly normal on > this cluster. > > The second major data point was a similarly spec'ed cluster except > that instead of SSD's it had HDDs in the same RAID configuration. > Attempting to help fix latencies we made the same change to disable > the async thread pool. The change appeared to make the cluster faster > but there wasn't a significant change in latencies for the cluster. > There was an observable increase in throughput for the entire cluster > but as in all distributed systems it was a bit hard to deconvolve the > various signals. > > Given these two data points I decided to try looking at the VM flags > that might be major behavior changes between R14B01 and 17.5. The > flags I decided to try setting were "+sbwt none +secio false +sws > legacy" (from memory). I applied those to one node on the MT cluster > and it resulted in an immediate change from roughly fifty to six-ish > percent on CPU usage. > > Once I saw the drastic change in CPU I replaced the change to just > "+sbwt none" and saw no change in CPU usage which suggested that the > sbwt change was entirely responsible for the change. After letting the > cluster run I also tried using "+sbwt very_low" which resulted in a > noticeable increase in system CPU but nowhere near the ~50% system CPU > default. > > The MT cluster I did the initial +sbwt change on was the multi-tenant > cluster so its hard to make too many conclusions given the roughly > random work load. What I did do was check that very_low increased the > system CPU usage noticeably and that default put it back to 50%. > During these experiments our various latency metrics seemed to respond > but distributed systems, so its hard to draw direct conclusions. > > However, given my observations I was pretty certain that the +sbwt > setting was directly affecting system CPU usage. The general theory > I've been working on is that something with Erlang's elevated system > CPU usage is causing my observed latencies to increase. Given that, I > set +sbwt none to one of our problematic clusters. The initial > response to this change was a significant increase in throughput of > background tasks on the cluster. Once those cleared out the cluster > latencies have settled back into the previous R14B01 levels. > > My general question at this point is wether scheduler busy wait has an > intrinsic cost that's not reflected in the documentation. From what > I've seen is that we end up with a large amount of system CPU usage > that dampens user CPU usage of beam.smp which in our case affects > system latencies. We have a fairly intense port/file driver usage > which may affect the issue. Disabling busy wait fixes things and that > confuses me. > > Any thoughts or similar observations would be awesomely appreciated. I > think I can see how +sbwt values affect a VM but I'm fairly confused > why reverting back to R14 didn't rectify things. I'm really hoping > someone remembers a scheduler bug around this area for that part of > the story. > > I'll buy many beers at the next conference for anyone that makes me > think I'm not insane in all this. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon@REDACTED Tue Sep 1 15:28:13 2015 From: jon@REDACTED (Jon Schneider) Date: Tue, 1 Sep 2015 14:28:13 +0100 Subject: [erlang-questions] Cost of doing +sbwt? In-Reply-To: References: Message-ID: > CPU. Of note this cluster is a 9 node cluster with dual hex cores with > hyperthreading for a total of 24 scheduler threads as well as a four Could it be that you (or Erlang) have over-estimated the value of a hypermarketing^Whyperthread so they're treading on each others's feet and you'd benefit from fewer schedulers ? Even if you believe hyperthreads are useful having fewer schedulers would surely reduce the likelihood of locking contention. Jon From erlang@REDACTED Tue Sep 1 15:32:55 2015 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 1 Sep 2015 15:32:55 +0200 Subject: [erlang-questions] web proxy In-Reply-To: References: Message-ID: On Tue, Sep 1, 2015 at 12:05 PM, Tuncer Ayaz wrote: > On Tue, Sep 1, 2015 at 9:54 AM, Joe Armstrong wrote: >> Hello, >> >> I want to record all traffic between my browser and the Internet. >> >> What I want is a record of >> >> 1) the request >> 2) the number of bytes in the response >> 3) some timing data >> >> When I fetch a web page. >> >> I'd like this data (if possible) in the form of event messages being sent >> to an Erlang program. >> >> Why do I want this? >> >> I was horrified by reading >> http://blog.lmorchard.com/2015/07/22/the-verge-web-sucks/ >> and wanted to accurately record exactly how much data is involved when >> I request a web page. The I want see how much is content and how much >> is crap. >> >> Any ideas > > Browsers have (developer) profiling tools, which allow you to > inspect the network traffic from the client's view. That should do the > trick. I know, but I want to remotely control the browser from Erlang and get the results in Erlang not in the browser console (or whatever) - I don't want to click things inside the browser /Joe From jesper.louis.andersen@REDACTED Tue Sep 1 15:37:40 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 1 Sep 2015 15:37:40 +0200 Subject: [erlang-questions] Cost of doing +sbwt? In-Reply-To: References: Message-ID: On Tue, Sep 1, 2015 at 9:16 AM, Paul Davis wrote: > Our first data point was to try and look at strace. What we noticed > was that scheduler threads seemed to spend an inordinate amount of > time in futex system calls. An strace run on a scheduler thread showed > more than 50% of time in futex sys calls. > I may have other comments later, but this should tip you off as to what happens. A futex() syscall is made whenever a lock is contended. The uncontended case can be handled with no kernel invocation. If you spend your time here, you are contended on some resource somewhere inside the system. Like Hynek, if you are running on the bare metal and not in some puny hypervisor, then setting something like `+sbt db` is often worth it. It binds schedulers to physical cores so they don't jump around and destroys your TLBs and caches into oblivion. I'd have two paths I'd continue on this: lockcnt instrumentation in a staging environment and looking at where that contention is. Try to reproduce it. Or pray to god you are running on FreeBSD/Illumos in production in which case you can find the lock contention with a 5 line DTrace script on the production cluster :) Also, look at the current scheduler utilization! erlang:statistics(scheduler_wall_time) (read the man page, you need a system_flag too). You want to look at how much time the schedulers are spending doing useful work and how much time they are just spinning waiting for more work to come in. Though the high CPU count you are seeing more suggests a lock that is contended to me. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.platt@REDACTED Tue Sep 1 15:40:17 2015 From: pablo.platt@REDACTED (pablo platt) Date: Tue, 1 Sep 2015 16:40:17 +0300 Subject: [erlang-questions] web proxy In-Reply-To: References: Message-ID: On Tue, Sep 1, 2015 at 4:32 PM, Joe Armstrong wrote: > On Tue, Sep 1, 2015 at 12:05 PM, Tuncer Ayaz > wrote: > > On Tue, Sep 1, 2015 at 9:54 AM, Joe Armstrong wrote: > >> Hello, > >> > >> I want to record all traffic between my browser and the Internet. > >> > >> What I want is a record of > >> > >> 1) the request > >> 2) the number of bytes in the response > >> 3) some timing data > >> > >> When I fetch a web page. > >> > >> I'd like this data (if possible) in the form of event messages being > sent > >> to an Erlang program. > >> > >> Why do I want this? > >> > >> I was horrified by reading > >> http://blog.lmorchard.com/2015/07/22/the-verge-web-sucks/ > >> and wanted to accurately record exactly how much data is involved when > >> I request a web page. The I want see how much is content and how much > >> is crap. > >> > >> Any ideas > > > > Browsers have (developer) profiling tools, which allow you to > > inspect the network traffic from the client's view. That should do the > > trick. > > I know, but I want to remotely control the browser from Erlang and get > the results in Erlang not in the browser console (or whatever) - I > don't want > to click things inside the browser > What about wireshark with http filter or a pcap capture file? > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony@REDACTED Tue Sep 1 16:00:31 2015 From: tony@REDACTED (Tony Rogvall) Date: Tue, 1 Sep 2015 16:00:31 +0200 Subject: [erlang-questions] web proxy In-Reply-To: References: Message-ID: <439AB998-8D58-465A-85AF-5811A395DC91@rogvall.se> Hi Joe! You can have a look here: https://github.com/tonyrog/eth I wrote a cookie state tracker by just inspecting the HTTP/TCP packet traffic. This is an example program using ?eth? in src/eth_http_track.erl Maybe you can find something useful there. I guess you can estimate the data up/down loaded by calculate sum over packets data length, (ignoring retransmission etc) and/or check the Content-Length where possible. I think you may use enet/src/enet_tcp_fsm.erl to collect the stream, not sure what happens if you miss a packet or two? /Tony > On 1 sep 2015, at 09:54, Joe Armstrong wrote: > > Hello, > > I want to record all traffic between my browser and the Internet. > > What I want is a record of > > 1) the request > 2) the number of bytes in the response > 3) some timing data > > When I fetch a web page. > > I'd like this data (if possible) in the form of event messages being sent > to an Erlang program. > > Why do I want this? > > I was horrified by reading > http://blog.lmorchard.com/2015/07/22/the-verge-web-sucks/ > and wanted to accurately record exactly how much data is involved when > I request a web page. The I want see how much is content and how much > is crap. > > Any ideas > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From bchesneau@REDACTED Tue Sep 1 16:06:53 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Tue, 01 Sep 2015 14:06:53 +0000 Subject: [erlang-questions] web proxy In-Reply-To: References: Message-ID: On Tue, Sep 1, 2015 at 11:51 AM Joe Armstrong wrote: > Hello, > > I want to record all traffic between my browser and the Internet. > > What I want is a record of > > 1) the request > 2) the number of bytes in the response > 3) some timing data > > When I fetch a web page. > > I'd like this data (if possible) in the form of event messages being sent > to an Erlang program. > > Why do I want this? > > I was horrified by reading > http://blog.lmorchard.com/2015/07/22/the-verge-web-sucks/ > and wanted to accurately record exactly how much data is involved when > I request a web page. The I want see how much is content and how much > is crap. > > Any ideas > > /Joe > _______________________________________________ > Old, but sometimes ago I wrote sieve: https://github.com/benoitc/sieve wich could be the base for such tool. It basically allows you to catch the request and do anything you need with it. Like proxymachine from github. There is an older version but unmaintained for cowboy: https://github.com/benoitc/cowboy_revproxy It actually miss the posibility to pass the request and pass to a connected socket, but it could be added easily. - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Tue Sep 1 17:30:31 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Tue, 1 Sep 2015 11:30:31 -0400 (EDT) Subject: [erlang-questions] Tried this... Thought it interesting... But it didn't work. Message-ID: <1441121431.32149852@apps.rackspace.com> So, I ran across this tidbit in the Erlang cybersphere: List Comprehension without generators http://blog.equanimity.nl/blog/2015/03/15/erlang-one-weird-trick-goodiebag/ 1> Y = 3. 3 2> Z = [X || Y > 2]. * 1: variable 'X' is unbound OK.So I tried... -module(test). -compile(export_all). test(Y) -> [ X || Y > 2]. >3 test:test(). Error: variable 'X' is unbound Waah...is someone pulling my leg? This worked... 4> Z = [X || X <- [Y >2]]. But it seems rather pointless. I didn't check to see if the original post was dated circa April 1. Best wishes, LRP From sergej.jurecko@REDACTED Tue Sep 1 17:33:40 2015 From: sergej.jurecko@REDACTED (Sergej =?UTF-8?B?SnVyZcSNa28=?=) Date: Tue, 01 Sep 2015 17:33:40 +0200 Subject: [erlang-questions] Tried this... Thought it interesting... But it didn't work. In-Reply-To: <1441121431.32149852@apps.rackspace.com> References: <1441121431.32149852@apps.rackspace.com> Message-ID: Both X and Y must be defined outside that list comprehension. Sergej On 01/09/15 17:30, "erlang-questions-bounces@REDACTED on behalf of lloyd@REDACTED" wrote: >So, I ran across this tidbit in the Erlang cybersphere: > >List Comprehension without generators > >http://blog.equanimity.nl/blog/2015/03/15/erlang-one-weird-trick-goodiebag/ > >1> Y = 3. >3 >2> Z = [X || Y > 2]. >* 1: variable 'X' is unbound > >OK.So I tried... > >-module(test). > >-compile(export_all). > >test(Y) -> > [ X || Y > 2]. > >>3 test:test(). >Error: variable 'X' is unbound > >Waah...is someone pulling my leg? > >This worked... > >4> Z = [X || X <- [Y >2]]. > >But it seems rather pointless. > >I didn't check to see if the original post was dated circa April 1. > >Best wishes, > >LRP > > > >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions From zxq9@REDACTED Tue Sep 1 18:43:31 2015 From: zxq9@REDACTED (zxq9) Date: Wed, 02 Sep 2015 01:43:31 +0900 Subject: [erlang-questions] Tried this... Thought it interesting... But it didn't work. In-Reply-To: <1441121431.32149852@apps.rackspace.com> References: <1441121431.32149852@apps.rackspace.com> Message-ID: <15670408.uQR5RUAWqN@changa> On 2015?9?1? ??? 11:30:31 lloyd@REDACTED wrote: > So, I ran across this tidbit in the Erlang cybersphere: > > List Comprehension without generators > > http://blog.equanimity.nl/blog/2015/03/15/erlang-one-weird-trick-goodiebag/ > > 1> Y = 3. > 3 > 2> Z = [X || Y > 2]. > * 1: variable 'X' is unbound > > OK.So I tried... > > -module(test). > > -compile(export_all). > > test(Y) -> > [ X || Y > 2]. > > >3 test:test(). > Error: variable 'X' is unbound > > Waah...is someone pulling my leg? > > This worked... > > 4> Z = [X || X <- [Y >2]]. > > But it seems rather pointless. Hi there, Lloyd! This is a shortcut. It is just using the test in the list comprehension to determine whether a list of one element or an empty list is returned. Both X and Y must already be defined in the scope of the expression: 1> X = 1. 1 2> Y = 2. 2 3> Z = [X || Y > 2]. [] 4> Q = [X || Y > 1]. [1] Honestly, I don't really think this is the best use of list comprehension syntax. I can easily imagine people being confused at that, or at least requiring a few moments thought to figure out wtf is going on in that line. The fact this confounded you (and the example was unclear in the blog post, imo) is a good reason why you shouldn't do this in your code. -Craig From roberto@REDACTED Tue Sep 1 18:44:26 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Tue, 1 Sep 2015 18:44:26 +0200 Subject: [erlang-questions] TLS errors Message-ID: Dear all, I'm using SSL with Cowboy and I keep on getting these kind of errors in the logs: SSL: hello: tls_handshake.erl:167:Fatal error: insufficient security SSL: hello: tls_handshake.erl:174:Fatal error: protocol version SSL: certify: ssl_alert.erl:93:Fatal error: bad certificate SSL: hello: tls_handshake.erl:118:Fatal error: inappropriate fallback SSL: cipher: ssl_cipher.erl:292:Fatal error: bad record mac I've started seeing those after I've upgraded to Erlang 18.0.2. This wasn't happening in 17.5. Has anyone seen this, and has clues for me to pinpoint what the problem is? Also, is there anyway for me to recover the originating IP address? Any help appreciated. Thank you, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Sep 1 20:01:38 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 1 Sep 2015 20:01:38 +0200 Subject: [erlang-questions] Tried this... Thought it interesting... But it didn't work. In-Reply-To: <15670408.uQR5RUAWqN@changa> References: <1441121431.32149852@apps.rackspace.com> <15670408.uQR5RUAWqN@changa> Message-ID: On Tue, Sep 1, 2015 at 6:43 PM, zxq9 wrote: > Honestly, I don't really think this is the best use of list comprehension > syntax. I can easily imagine people being confused at that, or at least > requiring a few moments thought to figure out wtf is going on in that line. > The fact this confounded you (and the example was unclear in the blog post, > imo) is a good reason why you shouldn't do this in your code. > I must admit I sometimes do this. Consider: f(X) -> E = case X > 5 of true -> [7]; false -> [] end, [1,2,3] ++ E ++ [4,5]. This is easier written as f(X) -> [1,2,3] ++ [7 || X > 5] ++ [4,5]. but as a way to get the list comprehension to append like this is the only way I tend to use the construction. For real-world use, consider the following snippet from my Maps R18 tests: https://github.com/jlouis/maps_eqc/blob/96d06da56053e87dd33c830b293dface525be17d/src/maps_eqc.erl#L693-L696 remove_args(#state { contents = C } = State) -> frequency( [{5, ?LET(Pair, elements(C), [element(1, Pair)])} || C /= [] ] ++ [{1, ?SUCHTHAT([K], [map_key(State)], find(K, 1, C) == false)}]). The idea here is that we want to generate arguments for removing an element from a map, and C contains the current contents of the map. If the map is non-empty, C /= [], and we can pick elements from C. Otherwise, we generate a map key such that it is really not an element of the map in question (which is trivially true if C = []). The neat part is that the first variant with frequency 5 is never generated for the empty map. A case analysis in this case would tend to repeat code, so I find this somewhat more nimble. But these situations are probably the only situations on which I use this way of writing. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fernando.benavides@REDACTED Tue Sep 1 20:15:02 2015 From: fernando.benavides@REDACTED (Fernando 'Brujo' Benavides) Date: Tue, 1 Sep 2015 15:15:02 -0300 Subject: [erlang-questions] Tried this... Thought it interesting... But it didn't work. In-Reply-To: References: <1441121431.32149852@apps.rackspace.com> <15670408.uQR5RUAWqN@changa> Message-ID: <4C362377-BB80-4FFB-9BB0-0D006A824289@inaka.net> We had this same discussion before with Jesper (when I wrote this blog post), but I still think these versions clearer: remove_args(#state { contents = C } = State) -> frequency( case C of [] -> []; C -> [{5, ?LET(Pair, elements(C), [element(1, Pair)])}] end ++ [{1, ?SUCHTHAT([K], [map_key(State)], find(K, 1, C) == false)}]). f(X) -> [1,2,3] ++ e(X) ++ [4,5]. e(X) when X > 5 -> [7]; e(_) -> []. My two cents, ?\_(?)_/? Fernando "Brujo" Benavides about.me/elbrujohalcon > On Sep 1, 2015, at 15:01, Jesper Louis Andersen wrote: > > > On Tue, Sep 1, 2015 at 6:43 PM, zxq9 > wrote: > Honestly, I don't really think this is the best use of list comprehension syntax. I can easily imagine people being confused at that, or at least requiring a few moments thought to figure out wtf is going on in that line. The fact this confounded you (and the example was unclear in the blog post, imo) is a good reason why you shouldn't do this in your code. > > I must admit I sometimes do this. Consider: > > f(X) -> > E = case X > 5 of > true -> [7]; > false -> [] > end, > [1,2,3] ++ E ++ [4,5]. > > This is easier written as > > f(X) -> > [1,2,3] ++ [7 || X > 5] ++ [4,5]. > > but as a way to get the list comprehension to append like this is the only way I tend to use the construction. For real-world use, consider the following snippet from my Maps R18 tests: > > https://github.com/jlouis/maps_eqc/blob/96d06da56053e87dd33c830b293dface525be17d/src/maps_eqc.erl#L693-L696 > > remove_args(#state { contents = C } = State) -> > frequency( > [{5, ?LET(Pair, elements(C), [element(1, Pair)])} || C /= [] ] ++ > [{1, ?SUCHTHAT([K], [map_key(State)], find(K, 1, C) == false)}]). > > The idea here is that we want to generate arguments for removing an element from a map, and C contains the current contents of the map. If the map is non-empty, C /= [], and we can pick elements from C. Otherwise, we generate a map key such that it is really not an element of the map in question (which is trivially true if C = []). The neat part is that the first variant with frequency 5 is never generated for the empty map. > > A case analysis in this case would tend to repeat code, so I find this somewhat more nimble. But these situations are probably the only situations on which I use this way of writing. > > > -- > J. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Tue Sep 1 20:55:37 2015 From: zxq9@REDACTED (zxq9) Date: Wed, 02 Sep 2015 03:55:37 +0900 Subject: [erlang-questions] Tried this... Thought it interesting... But it didn't work. In-Reply-To: References: <1441121431.32149852@apps.rackspace.com> <15670408.uQR5RUAWqN@changa> Message-ID: <1969395.bKdKAYm9Vi@changa> On 2015?9?1? ??? 20:01:38 Jesper Louis Andersen wrote: > On Tue, Sep 1, 2015 at 6:43 PM, zxq9 wrote: > > > Honestly, I don't really think this is the best use of list comprehension > > syntax. I can easily imagine people being confused at that, or at least > > requiring a few moments thought to figure out wtf is going on in that line. > > The fact this confounded you (and the example was unclear in the blog post, > > imo) is a good reason why you shouldn't do this in your code. > > > > I must admit I sometimes do this. Consider: > > f(X) -> > E = case X > 5 of > true -> [7]; > false -> [] > end, > [1,2,3] ++ E ++ [4,5]. > > This is easier written as > > f(X) -> > [1,2,3] ++ [7 || X > 5] ++ [4,5]. > > but as a way to get the list comprehension to append like this is the only > way I tend to use the construction. For real-world use, consider the > following snippet from my Maps R18 tests: > > https://github.com/jlouis/maps_eqc/blob/96d06da56053e87dd33c830b293dface525be17d/src/maps_eqc.erl#L693-L696 > > remove_args(#state { contents = C } = State) -> > frequency( > [{5, ?LET(Pair, elements(C), [element(1, Pair)])} || C /= [] ] ++ > [{1, ?SUCHTHAT([K], [map_key(State)], find(K, 1, C) == false)}]). > > The idea here is that we want to generate arguments for removing an element > from a map, and C contains the current contents of the map. If the map is > non-empty, C /= [], and we can pick elements from C. Otherwise, we generate > a map key such that it is really not an element of the map in question > (which is trivially true if C = []). The neat part is that the first > variant with frequency 5 is never generated for the empty map. > > A case analysis in this case would tend to repeat code, so I find this > somewhat more nimble. But these situations are probably the only situations > on which I use this way of writing. I sort of see your point, but I tend to write far less exciting code. I would probably do something more like: remove_args(#state { contents = C } = State) -> frequency(prefix_on_empty(C) ++ [{1, ?SUCHTHAT([K], [map_key(State)], find(K, 1, C) == false)}]). prefix_on_empty([]) -> []; prefix_on_empty(C) -> [{5, ?LET(Pair, elements(C), [element(1, Pair)])}]. Note, I have no idea what a fitting name for prefix_on_empty/1 should be, but you get the idea. Anyway, interesting technique. -Craig From zxq9@REDACTED Tue Sep 1 20:56:28 2015 From: zxq9@REDACTED (zxq9) Date: Wed, 02 Sep 2015 03:56:28 +0900 Subject: [erlang-questions] Tried this... Thought it interesting... But it didn't work. In-Reply-To: <4C362377-BB80-4FFB-9BB0-0D006A824289@inaka.net> References: <1441121431.32149852@apps.rackspace.com> <4C362377-BB80-4FFB-9BB0-0D006A824289@inaka.net> Message-ID: <2289886.0A3hYideU6@changa> On 2015?9?1? ??? 15:15:02 Fernando 'Brujo' Benavides wrote: > We had this same discussion before with Jesper (when I wrote this blog post), but I still think these versions clearer: > > remove_args(#state { contents = C } = State) -> > frequency( > case C of > [] -> []; > C -> [{5, ?LET(Pair, elements(C), [element(1, Pair)])}] > end ++ [{1, ?SUCHTHAT([K], [map_key(State)], find(K, 1, C) == false)}]). > > > f(X) -> [1,2,3] ++ e(X) ++ [4,5]. > e(X) when X > 5 -> [7]; > e(_) -> []. > > My two cents, ?\_(?)_/? ...or this (though appending to the tail end of a case feels weird to me -- but it is instantly understandable). I would have spent a moment scratching my head at the original because I see it so rarely. Actually, the only times I've ever seen this have been this discussion and a similar discussion about coding anomalies a few months ago. -Craig From paul.joseph.davis@REDACTED Tue Sep 1 21:14:48 2015 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Tue, 1 Sep 2015 14:14:48 -0500 Subject: [erlang-questions] Cost of doing +sbwt? In-Reply-To: References: Message-ID: On Tue, Sep 1, 2015 at 8:37 AM, Jesper Louis Andersen wrote: > > On Tue, Sep 1, 2015 at 9:16 AM, Paul Davis > wrote: >> >> Our first data point was to try and look at strace. What we noticed >> was that scheduler threads seemed to spend an inordinate amount of >> time in futex system calls. An strace run on a scheduler thread showed >> more than 50% of time in futex sys calls. > > > I may have other comments later, but this should tip you off as to what > happens. A futex() syscall is made whenever a lock is contended. The > uncontended case can be handled with no kernel invocation. If you spend your > time here, you are contended on some resource somewhere inside the system. > > Like Hynek, if you are running on the bare metal and not in some puny > hypervisor, then setting something like `+sbt db` is often worth it. It > binds schedulers to physical cores so they don't jump around and destroys > your TLBs and caches into oblivion. > > I'd have two paths I'd continue on this: lockcnt instrumentation in a > staging environment and looking at where that contention is. Try to > reproduce it. Or pray to god you are running on FreeBSD/Illumos in > production in which case you can find the lock contention with a 5 line > DTrace script on the production cluster :) > > Also, look at the current scheduler utilization! > erlang:statistics(scheduler_wall_time) (read the man page, you need a > system_flag too). You want to look at how much time the schedulers are > spending doing useful work and how much time they are just spinning waiting > for more work to come in. Though the high CPU count you are seeing more > suggests a lock that is contended to me. > > > -- > J. Some more data on this. We don't set +sbt on either R14B01 or 17.5. As I understand it that means that schedulers are unbound before and after the upgrade. As to hyperthreading and reducing scheduler count, after we reverted a cluster back to R14B01 and kept seeing the elevated system CPU one of my theories was that the cluster had been experiencing scheduler collapse which artificially limited the number of schedulers which reduced lock contention. To try and approximate that I spent some time playing with different numbers of schedulers online. Setting the online schedulers to half or a third made a definite impact on the sys CPU metrics but it wasn't an exact match to the pre-upgrade metrics. Running +sbwt none gives a much closer pre-upgrade behavior while maintaining all 24 schedulers. >From Jesper's comment on futex that was our thought as well. The fact that we're in the kernel for futex calls should theoretically mean we're contending quite hard for locks somewhere. That was what lead to trying to disable the thread pool to see if that was a lock having issues and eventually what lead me to randomly trying +sbwt none. Here's a gist with some data from lcnt and strace that I collected. The lcnt run is before I set +sbwt none, and there's an strace from before and after. I'm going to try and get some data from the scheduler_wall_time stuff now and will update with anything I find on that. Also, does anyone have a quick pointer to where the busy wait loop is? After I look at the scheduler time I was going to go find that code and see if I couldn't come up with a better idea of what exactly might be changing with that setting. From paul.joseph.davis@REDACTED Tue Sep 1 21:16:13 2015 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Tue, 1 Sep 2015 14:16:13 -0500 Subject: [erlang-questions] Cost of doing +sbwt? In-Reply-To: References: Message-ID: Doh, a link for that gist would be handy: https://gist.github.com/davisp/3ab37e9c69522fe1badd On Tue, Sep 1, 2015 at 2:14 PM, Paul Davis wrote: > On Tue, Sep 1, 2015 at 8:37 AM, Jesper Louis Andersen > wrote: >> >> On Tue, Sep 1, 2015 at 9:16 AM, Paul Davis >> wrote: >>> >>> Our first data point was to try and look at strace. What we noticed >>> was that scheduler threads seemed to spend an inordinate amount of >>> time in futex system calls. An strace run on a scheduler thread showed >>> more than 50% of time in futex sys calls. >> >> >> I may have other comments later, but this should tip you off as to what >> happens. A futex() syscall is made whenever a lock is contended. The >> uncontended case can be handled with no kernel invocation. If you spend your >> time here, you are contended on some resource somewhere inside the system. >> >> Like Hynek, if you are running on the bare metal and not in some puny >> hypervisor, then setting something like `+sbt db` is often worth it. It >> binds schedulers to physical cores so they don't jump around and destroys >> your TLBs and caches into oblivion. >> >> I'd have two paths I'd continue on this: lockcnt instrumentation in a >> staging environment and looking at where that contention is. Try to >> reproduce it. Or pray to god you are running on FreeBSD/Illumos in >> production in which case you can find the lock contention with a 5 line >> DTrace script on the production cluster :) >> >> Also, look at the current scheduler utilization! >> erlang:statistics(scheduler_wall_time) (read the man page, you need a >> system_flag too). You want to look at how much time the schedulers are >> spending doing useful work and how much time they are just spinning waiting >> for more work to come in. Though the high CPU count you are seeing more >> suggests a lock that is contended to me. >> >> >> -- >> J. > > Some more data on this. We don't set +sbt on either R14B01 or 17.5. As > I understand it that means that schedulers are unbound before and > after the upgrade. > > As to hyperthreading and reducing scheduler count, after we reverted a > cluster back to R14B01 and kept seeing the elevated system CPU one of > my theories was that the cluster had been experiencing scheduler > collapse which artificially limited the number of schedulers which > reduced lock contention. To try and approximate that I spent some time > playing with different numbers of schedulers online. Setting the > online schedulers to half or a third made a definite impact on the sys > CPU metrics but it wasn't an exact match to the pre-upgrade metrics. > Running +sbwt none gives a much closer pre-upgrade behavior while > maintaining all 24 schedulers. > > From Jesper's comment on futex that was our thought as well. The fact > that we're in the kernel for futex calls should theoretically mean > we're contending quite hard for locks somewhere. That was what lead to > trying to disable the thread pool to see if that was a lock having > issues and eventually what lead me to randomly trying +sbwt none. > > Here's a gist with some data from lcnt and strace that I collected. > The lcnt run is before I set +sbwt none, and there's an strace from > before and after. I'm going to try and get some data from the > scheduler_wall_time stuff now and will update with anything I find on > that. > > Also, does anyone have a quick pointer to where the busy wait loop is? > After I look at the scheduler time I was going to go find that code > and see if I couldn't come up with a better idea of what exactly might > be changing with that setting. From lukas@REDACTED Tue Sep 1 21:42:10 2015 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 1 Sep 2015 21:42:10 +0200 Subject: [erlang-questions] Cost of doing +sbwt? In-Reply-To: References: Message-ID: On Tue, Sep 1, 2015 at 9:14 PM, Paul Davis wrote: > Also, does anyone have a quick pointer to where the busy wait loop is? > After I look at the scheduler time I was going to go find that code > and see if I couldn't come up with a better idea of what exactly might > be changing with that setting. This should be the code that does the waiting: https://github.com/erlang/otp/blob/master/erts/lib_src/pthread/ethr_event.c#L65-L161 The mutex implementation that calls it is in here: https://github.com/erlang/otp/blob/master/erts/lib_src/common/ethr_mutex.c The different spin options are set here: https://github.com/erlang/otp/blob/master/erts/emulator/beam/erl_process.c#L5325-L5364 There are also a couple of other places where it spind in erl_process.c, just search for spin and you'll find them :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Wed Sep 2 04:24:46 2015 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Tue, 1 Sep 2015 22:24:46 -0400 Subject: [erlang-questions] Tried this... Thought it interesting... But it didn't work. In-Reply-To: References: <1441121431.32149852@apps.rackspace.com> <15670408.uQR5RUAWqN@changa> Message-ID: Hi Jesper, Your code exhibits a depth of understanding and subtlety that I can only hope to aspire to. Can you help me understand the ?LET(... and ?SUCHTHAT(... expressions? As always, I deeply appreciate your guidance. Lloyd Sent from my iPad > On Sep 1, 2015, at 2:01 PM, Jesper Louis Andersen wrote: > > >> On Tue, Sep 1, 2015 at 6:43 PM, zxq9 wrote: >> Honestly, I don't really think this is the best use of list comprehension syntax. I can easily imagine people being confused at that, or at least requiring a few moments thought to figure out wtf is going on in that line. The fact this confounded you (and the example was unclear in the blog post, imo) is a good reason why you shouldn't do this in your code. > > I must admit I sometimes do this. Consider: > > f(X) -> > E = case X > 5 of > true -> [7]; > false -> [] > end, > [1,2,3] ++ E ++ [4,5]. > > This is easier written as > > f(X) -> > [1,2,3] ++ [7 || X > 5] ++ [4,5]. > > but as a way to get the list comprehension to append like this is the only way I tend to use the construction. For real-world use, consider the following snippet from my Maps R18 tests: > > https://github.com/jlouis/maps_eqc/blob/96d06da56053e87dd33c830b293dface525be17d/src/maps_eqc.erl#L693-L696 > > remove_args(#state { contents = C } = State) -> > frequency( > [{5, ?LET(Pair, elements(C), [element(1, Pair)])} || C /= [] ] ++ > [{1, ?SUCHTHAT([K], [map_key(State)], find(K, 1, C) == false)}]). > > The idea here is that we want to generate arguments for removing an element from a map, and C contains the current contents of the map. If the map is non-empty, C /= [], and we can pick elements from C. Otherwise, we generate a map key such that it is really not an element of the map in question (which is trivially true if C = []). The neat part is that the first variant with frequency 5 is never generated for the empty map. > > A case analysis in this case would tend to repeat code, so I find this somewhat more nimble. But these situations are probably the only situations on which I use this way of writing. > > > -- > J. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Sep 2 05:37:11 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 2 Sep 2015 15:37:11 +1200 Subject: [erlang-questions] Tried this... Thought it interesting... But it didn't work. In-Reply-To: <1441121431.32149852@apps.rackspace.com> References: <1441121431.32149852@apps.rackspace.com> Message-ID: On 2/09/2015, at 3:30 am, wrote: > So, I ran across this tidbit in the Erlang cybersphere: > > List Comprehension without generators > > http://blog.equanimity.nl/blog/2015/03/15/erlang-one-weird-trick-goodiebag/ > > 1> Y = 3. > 3 > 2> Z = [X || Y > 2]. > * 1: variable 'X' is unbound Why did you think that would work? What did you expect it to be? 1> Y = 3, A = [a || Y < 2], B = [b || Y > 2], {A,B}. {[],[b]} Perhaps more interesting: 2> [io:write(a) || Y < 2], [io:write(b) || Y > 2]. b[ok] from which we see that io:write(a) was not called, while io:write(b) was called. 3> Pid = spawn(fun () -> receive X -> io:write('OK') end end). <0.38.0> 4> [Pid ! a || Y < 2]. [] 5> [Pid ! b || Y > 2]. 'OK'[b] And this, boys and girls, means that Erlang has a one-armed 'if then ' statement. All those years I've been saying "we don't need no stinking COND" and we already had one... From ingela.andin@REDACTED Wed Sep 2 08:59:24 2015 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 2 Sep 2015 08:59:24 +0200 Subject: [erlang-questions] TLS errors In-Reply-To: References: Message-ID: Hi! This I suspect is because of of ssl in OTP 18 no longer supports legacy insecure ciphers and SSL/TLS versions by default. If needed for interoperability they can still be configured. >From the release notes: "Remove default support for SSL-3.0, due to Poodle vunrability in protocol specification. Add padding check for TLS-1.0 to remove Poodle vunrability from TLS 1.0, also add the option padding_check. This option only affects TLS-1.0 connections and if set to false it disables the block cipher padding check to be able to interoperate with legacy software. Remove default support for RC4 cipher suites, as they are consider too weak." 2015-09-01 18:44 GMT+02:00 Roberto Ostinelli : > Dear all, > I'm using SSL with Cowboy and I keep on getting these kind of errors in > the logs: > > SSL: hello: tls_handshake.erl:167:Fatal error: insufficient security > Could not find any common algorithms > SSL: hello: tls_handshake.erl:174:Fatal error: protocol version > No accetable TLS protocol version > SSL: certify: ssl_alert.erl:93:Fatal error: bad certificate > SSL: hello: tls_handshake.erl:118:Fatal error: inappropriate fallback > Prevention of Poodle > SSL: cipher: ssl_cipher.erl:292:Fatal error: bad record mac > > I've started seeing those after I've upgraded to Erlang 18.0.2. This > wasn't happening in 17.5. > > Has anyone seen this, and has clues for me to pinpoint what the problem is? > Also, is there anyway for me to recover the originating IP address? > > Any help appreciated. > > Regards Ingela Erlang/OTP team - Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From alttagil@REDACTED Wed Sep 2 13:52:13 2015 From: alttagil@REDACTED (Alex Hudich) Date: Wed, 2 Sep 2015 14:52:13 +0300 Subject: [erlang-questions] Distributed Process Spawner Message-ID: <741ECEF2-CB8F-495A-A022-0444B00FB39D@gmail.com> Hi! I am working on a piece of software which requires start of massive amount of workers (using spawn_link) but now I face with the limitation of one server for handling all the workers. All workers start with simple spawn_link then do their work and return some piece of data to a caller by sending a message. I wonder if I can unload master server and add some additional erlang nodes to perform this job. Of course I looked into pool and slave modules but it seems (beside of setting up ssh keys) does not support dynamic adding or deleting of nodes. Actually I want to know how to build some node supervising module that allows incoming connections from a new slave nodes and automatically plug them into some pool of nodes (including master) for spawning processes. Also it has to exclude down nodes and load balancing workers amongst nodes (so it has to collect some load statistics from nodes) Could you guys help me to find such component or give me advice how to build such thing? Thank you. From jesper.louis.andersen@REDACTED Wed Sep 2 14:23:57 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 2 Sep 2015 14:23:57 +0200 Subject: [erlang-questions] Cost of doing +sbwt? In-Reply-To: References: Message-ID: If memory serves, R14 can't experience scheduler collapse since it doesn't do rebalancing of work the same way as R15 and onwards. So I think this is a red herring. Have you established a baseline for the locking in R14? You are contending on the runqueue lock quite a lot, which could account for all the spinning you are seeing, but it is hard to say if this is a high or low number without some baseline you can use to relate. Also, many of the futex() calls are probably for this contention as well. There is a chance your scheduler utilization isn't that high, but you are getting into the spinning all the time. If utilization is fairly low, then the 50% CPU isn't of concern: just load the system more :) Chances are you are hunting the wrong mark as well: You have 2 or more pathologies, and they overlap in what you are seeing. Hence you get distracted by the noise generated by the other problems. It may be you have a CPU problem and on top of that, you have a latency/blocking problem in an I/O layer as well. One could account for the latency spikes, whereas the other would explain the high CPU. But if you don't know you are sitting with two problems in the first place, then their cooperation in the system confuses you. If you have elevated CPU, then a snapshot of the current thread stacks at 97hz per second[1] should tell you where things are taking time. This in itself could give you hints as to where you are spending all of your time in the system, and also what you are spinning on, if anything. [1] Old trick: Never snapshot at 100hz or something which means you can get into phase with other jobs. Pick some prime around your target. On Tue, Sep 1, 2015 at 9:42 PM, Lukas Larsson wrote: > On Tue, Sep 1, 2015 at 9:14 PM, Paul Davis > wrote: > >> Also, does anyone have a quick pointer to where the busy wait loop is? >> After I look at the scheduler time I was going to go find that code >> and see if I couldn't come up with a better idea of what exactly might >> be changing with that setting. > > > This should be the code that does the waiting: > https://github.com/erlang/otp/blob/master/erts/lib_src/pthread/ethr_event.c#L65-L161 > > The mutex implementation that calls it is in here: > https://github.com/erlang/otp/blob/master/erts/lib_src/common/ethr_mutex.c > > The different spin options are set here: > https://github.com/erlang/otp/blob/master/erts/emulator/beam/erl_process.c#L5325-L5364 > > There are also a couple of other places where it spind in erl_process.c, > just search for spin and you'll find them :) > -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Wed Sep 2 15:03:32 2015 From: sergej.jurecko@REDACTED (=?UTF-8?Q?Sergej_Jure=C4=8Dko?=) Date: Wed, 2 Sep 2015 15:03:32 +0200 Subject: [erlang-questions] Tried this... Thought it interesting... But it didn't work. In-Reply-To: <1441151164.479222416@apps.rackspace.com> References: <1441121431.32149852@apps.rackspace.com> <1441151164.479222416@apps.rackspace.com> Message-ID: Your question gave me a simple usage example that I just encountered. It is trivial. Instead of: case check_something() of ok -> ok; _ -> exit(error) end. You can write: [ exit(error) || check_something() /= ok ]. Sergej On Wed, Sep 2, 2015 at 1:46 AM, wrote: > Hi Sergei, > > But if X is defined outside the list comprehension, isn't it immutable? > > Can you show me an example that works? > > Thanks, > > Lloyd > > -----Original Message----- > From: "Sergej Jure?ko" > Sent: Tuesday, September 1, 2015 11:33am > To: lloyd@REDACTED, "erlang-questions" > > Subject: Re: [erlang-questions] Tried this... Thought it interesting... > But it didn't work. > > Both X and Y must be defined outside that list comprehension. > > > Sergej > > > > > > On 01/09/15 17:30, "erlang-questions-bounces@REDACTED on behalf of > lloyd@REDACTED" lloyd@REDACTED> wrote: > > >So, I ran across this tidbit in the Erlang cybersphere: > > > >List Comprehension without generators > > > > > http://blog.equanimity.nl/blog/2015/03/15/erlang-one-weird-trick-goodiebag/ > > > >1> Y = 3. > >3 > >2> Z = [X || Y > 2]. > >* 1: variable 'X' is unbound > > > >OK.So I tried... > > > >-module(test). > > > >-compile(export_all). > > > >test(Y) -> > > [ X || Y > 2]. > > > >>3 test:test(). > >Error: variable 'X' is unbound > > > >Waah...is someone pulling my leg? > > > >This worked... > > > >4> Z = [X || X <- [Y >2]]. > > > >But it seems rather pointless. > > > >I didn't check to see if the original post was dated circa April 1. > > > >Best wishes, > > > >LRP > > > > > > > >_______________________________________________ > >erlang-questions mailing list > >erlang-questions@REDACTED > >http://erlang.org/mailman/listinfo/erlang-questions > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Wed Sep 2 15:23:03 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 2 Sep 2015 15:23:03 +0200 Subject: [erlang-questions] Tried this... Thought it interesting... But it didn't work. In-Reply-To: References: <1441121431.32149852@apps.rackspace.com> <15670408.uQR5RUAWqN@changa> Message-ID: On Wed, Sep 2, 2015 at 4:24 AM, Lloyd R. Prentice wrote: > Your code exhibits a depth of understanding and subtlety that I can only > hope to aspire to. > > Can you help me understand the ?LET(... and ?SUCHTHAT(... expressions? > I agree with Fernando, that it is often nicer to pull these things into a function on its own, and use that function as in: f(X) -> [1,2] ++ e(X). e(X) when X < 5 -> []; e(X) -> [X]. In the QuickCheck models however, you would often end up writing many of such one-off functions, so this is why I sometimes tend to use the [X || test()] scheme. As for ?LET(...) and ?SUCHTHAT(...): They are macros which are used in QuickCheck like tools (Erlang QuickCheck, Proper, Triq, ...). If you have a generator like choose/2 which generates numbers between two endpoints: 6> triq_dom:sample( triq_dom:choose(1, 10)). [5,9,1,3,7,6,1,5,3,8,9] Some times, however, you don't want a raw generator. You want to do something to the result before you use it. The problem however is that the generator generates. So your code would have to: 1. draw a specimen from the generator. 2. apply some function to the drawn specimen 3. return the result of the application, but do so as a new generator doing this! This is what ?LET(...) does[0]. Writing ?LET(I, choose(1,10), I * 7). would return a new generator, which draws a number between 1 and 10, and returns 7 times that number. That is, it returns numbers in the range [7,14,21,28,...,70]. The ?SUCHTHAT(..) macro is used as ?SUCHTHAT(S, Gen, Pred(S)). It picks a specimen from Gen and then tests it against Pred. If Pred(S) returns true, it returns that value. If Pred(S) returns false, it tries generating a new value satisfying the predicate. I.e., generate S from Gen such that P(S) is true. ?SUCHTHAT(..) itself is a generator, doing this internally[1]. You use the ?SUCHTHAT(..) macro to reject certain values which are not desirable. In the case of trying to remove an element from a map, you want to generate an element that is not present in the map in some cases. This is to verify such removal has no effect on the map. So I use a ?SUCHTHAT(...) to make sure the generated target is not already present in the map. I expect to be able to find such a value in relatively few tries. Even considering very large maps with thousands of elements, you can easily generate a new element randomly, which isn't in there. [0] ?LET(..) is a monadic bind over the quickcheck monad. [1] The caveat of ?SUCHTHAT is that you want it to generate a valid value fairly quickly. If you are drawing random hay from a haystack to find a needle, and the haystack is large, then it is a bad idea. You have a random search problem at hand then. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine.koener@REDACTED Wed Sep 2 15:43:56 2015 From: antoine.koener@REDACTED (Antoine Koener) Date: Wed, 2 Sep 2015 15:43:56 +0200 Subject: [erlang-questions] Distributed Process Spawner In-Reply-To: <741ECEF2-CB8F-495A-A022-0444B00FB39D@gmail.com> References: <741ECEF2-CB8F-495A-A022-0444B00FB39D@gmail.com> Message-ID: Hi, Alex, May be you can look at the following code: https://github.com/Nekso/nkdist "NkDIST is an library to manage Erlang processes evenly distributed in a riak_core cluster. When you add or remove nodes from the cluster, NkDIST-based processes automatically move to another node to rebalance the cluster." On Wed, Sep 2, 2015 at 1:52 PM, Alex Hudich wrote: > Hi! > > I am working on a piece of software which requires start of massive amount > of workers (using spawn_link) but now I face with the limitation of one > server for handling all the workers. > All workers start with simple spawn_link then do their work and return > some piece of data to a caller by sending a message. > > I wonder if I can unload master server and add some additional erlang > nodes to perform this job. Of course I looked into pool and slave modules > but it seems (beside of setting up ssh keys) does not support dynamic > adding or deleting of nodes. > Actually I want to know how to build some node supervising module that > allows incoming connections from a new slave nodes and automatically plug > them into some pool of nodes (including master) for spawning processes. > Also it has to exclude down nodes and load balancing workers amongst nodes > (so it has to collect some load statistics from nodes) > > > Could you guys help me to find such component or give me advice how to > build such thing? > > Thank you. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akrupicka@REDACTED Wed Sep 2 16:19:26 2015 From: akrupicka@REDACTED (Adam Krupicka) Date: Wed, 02 Sep 2015 16:19:26 +0200 Subject: [erlang-questions] Tried this... Thought it interesting... But it didn't work. In-Reply-To: References: <1441121431.32149852@apps.rackspace.com> <1441151164.479222416@apps.rackspace.com> Message-ID: <1441203443-sup-829@arch.local> > [ exit(error) || check_something() /= ok ]. Another way to write this (not sure if more or less clearly) is: (check_something() /= ok) andalso exit(error). A. K. From rj@REDACTED Wed Sep 2 17:18:44 2015 From: rj@REDACTED (Richard Jones) Date: Wed, 2 Sep 2015 16:18:44 +0100 Subject: [erlang-questions] release handler crash during relup, sys:get_status noproc In-Reply-To: References: Message-ID: for the record, my initial suspicions were justified. i reproduced this by adding a timer:sleep in release_handler_1 to make it easier to trigger. it would only manifest if a supervisor exited (cleanly or otherwise) during release installation. workers restarting doesn't trigger it. if anyone else has experienced noproc crashes during release upgrades, i recommend trying this patch: https://github.com/erlang/otp/pull/822 On 24 August 2015 at 17:20, Richard Jones wrote: > Anyone else experienced a crash like this when doing a release upgrade? > ie, calling release_handler:install_release, with a valid relup > > {"init terminating in > do_boot",{{badmatch,{error,{'EXIT',{noproc,{sys,get_status,[<6453.14610.13>]}}}}},[{erl_eval,expr,3,[]}]}} > > I've seen this a couple of times now (erlang 17.x) when upgrading > production systems under load, even with a trivial relup. No idea what that > pid was. > > I think it might be a race in release_handler_1 where it calls > sys:get_status without a catch, when the process in question may have been > a supervision tree that had legitimately shut down since the list of pids > was fetched. > > ie: > > > https://github.com/erlang/otp/blob/OTP-17.5.6.3/lib/sasl/src/release_handler_1.erl#L589 > > which calls get_proc_state, which does: > > {status, _, {module, _}, [_, State, _, _, _]} = sys:get_status(Proc) > > I've not managed to make a test for this yet, planning to spam lots of > terminate_childs to a busy supervisor while calling > release_handler_1:get_supervised_procs to try and reproduce. > > If i'm right, it would only be triggered if parts of a supervision tree > are shutting down during a release_upgrade, which perhaps isn't very > common, depending on how dynamic the average supervision tree is in erlang > apps. > > Any feedback appreciated before I spend more time studying release handler > code :) > > RJ > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Wed Sep 2 19:12:20 2015 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Wed, 02 Sep 2015 19:12:20 +0200 Subject: [erlang-questions] Distributed Process Spawner In-Reply-To: <741ECEF2-CB8F-495A-A022-0444B00FB39D@gmail.com> Message-ID: Hi , Slave is nice for cpu computation on many machine, but why do you not do spawn_link on another node(s)? Is your limit on your main node caused by the number of local workers or by the global number of workers ? (where is the bottleneck ? In the process receiving the message sent back ?) Regards Le?2 sept. 2015 13:52, Alex Hudich a ?crit?: > > Hi! > > I am working on a piece of software which requires start of massive amount of workers (using spawn_link) but now I face with the limitation of one server for handling all the workers. > All workers start with simple spawn_link then do their work and return some piece of data to a caller by sending a message. > > I wonder if I can unload master server and add some additional erlang nodes to perform this job. Of course I looked into pool and slave modules but it seems (beside of setting up ssh keys) does not support dynamic adding or deleting of nodes. > Actually I want to know how to build some node supervising module that allows incoming connections from a new slave nodes and automatically plug them into some pool of nodes (including master) for spawning processes. Also it has to exclude down nodes and load balancing workers amongst nodes (so it has to collect some load statistics from nodes) > > > Could you guys help me to find such component or give me advice how to build such thing? > > Thank you. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mjtruog@REDACTED Wed Sep 2 19:20:47 2015 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 02 Sep 2015 10:20:47 -0700 Subject: [erlang-questions] Distributed Process Spawner In-Reply-To: <741ECEF2-CB8F-495A-A022-0444B00FB39D@gmail.com> References: <741ECEF2-CB8F-495A-A022-0444B00FB39D@gmail.com> Message-ID: <55E72FEF.80202@gmail.com> On 09/02/2015 04:52 AM, Alex Hudich wrote: > Hi! > > I am working on a piece of software which requires start of massive amount of workers (using spawn_link) but now I face with the limitation of one server for handling all the workers. > All workers start with simple spawn_link then do their work and return some piece of data to a caller by sending a message. > > I wonder if I can unload master server and add some additional erlang nodes to perform this job. Of course I looked into pool and slave modules but it seems (beside of setting up ssh keys) does not support dynamic adding or deleting of nodes. > Actually I want to know how to build some node supervising module that allows incoming connections from a new slave nodes and automatically plug them into some pool of nodes (including master) for spawning processes. Also it has to exclude down nodes and load balancing workers amongst nodes (so it has to collect some load statistics from nodes) > > > Could you guys help me to find such component or give me advice how to build such thing? > > Thank you. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > If you use the pools pg2 (http://www.erlang.org/doc/man/pg2.html) or cpg (https://github.com/okeuday/cpg/) and then use node auto-discovery with something like https://github.com/okeuday/nodefinder/ . That can provide a way of distributing traffic among processes selected from the pool as nodes are added. CloudI uses cpg with nodefinder to make this work with CloudI services, and you could use the same functionality with the cloudi_core dependency (https://github.com/CloudI/cloudi_core/) due to the cloudi_service_api module supporting the API at http://cloudi.org/api.html#CloudI . Load balancing among nodes is done internally due to the pool lookup. For monitoring the load to add or remove nodes you could add some basic monitoring. Best Regards, Michael From jesper.louis.andersen@REDACTED Wed Sep 2 20:19:03 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 2 Sep 2015 20:19:03 +0200 Subject: [erlang-questions] Distributed Process Spawner In-Reply-To: <55E72FEF.80202@gmail.com> References: <741ECEF2-CB8F-495A-A022-0444B00FB39D@gmail.com> <55E72FEF.80202@gmail.com> Message-ID: In addition to the already mentioned, Game Analytics did a scheduler called gascheduler: https://github.com/GameAnalytics/gascheduler which I haven't tried out or read the source code of. But it does sound like a fit for the problem at hand. On Wed, Sep 2, 2015 at 7:20 PM, Michael Truog wrote: > On 09/02/2015 04:52 AM, Alex Hudich wrote: > >> Hi! >> >> I am working on a piece of software which requires start of massive >> amount of workers (using spawn_link) but now I face with the limitation of >> one server for handling all the workers. >> All workers start with simple spawn_link then do their work and return >> some piece of data to a caller by sending a message. >> >> I wonder if I can unload master server and add some additional erlang >> nodes to perform this job. Of course I looked into pool and slave modules >> but it seems (beside of setting up ssh keys) does not support dynamic >> adding or deleting of nodes. >> Actually I want to know how to build some node supervising module that >> allows incoming connections from a new slave nodes and automatically plug >> them into some pool of nodes (including master) for spawning processes. >> Also it has to exclude down nodes and load balancing workers amongst nodes >> (so it has to collect some load statistics from nodes) >> >> >> Could you guys help me to find such component or give me advice how to >> build such thing? >> >> Thank you. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> If you use the pools pg2 (http://www.erlang.org/doc/man/pg2.html) or cpg > (https://github.com/okeuday/cpg/) > and then use node auto-discovery with something like > https://github.com/okeuday/nodefinder/ . That can provide > a way of distributing traffic among processes selected from the pool as > nodes are added. CloudI uses cpg with nodefinder > to make this work with CloudI services, and you could use the same > functionality with the cloudi_core dependency > (https://github.com/CloudI/cloudi_core/) due to the cloudi_service_api > module supporting the API at > http://cloudi.org/api.html#CloudI . > > Load balancing among nodes is done internally due to the pool lookup. For > monitoring the load to add or remove nodes > you could add some basic monitoring. > > Best Regards, > Michael > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.ostinelli@REDACTED Wed Sep 2 20:22:11 2015 From: roberto.ostinelli@REDACTED (Roberto Ostinelli) Date: Wed, 2 Sep 2015 20:22:11 +0200 Subject: [erlang-questions] TLS errors In-Reply-To: References: Message-ID: Thank you Ingela. This seems to be the case indeed. > On 02/set/2015, at 08:59, Ingela Andin wrote: > > Hi! > > This I suspect is because of of ssl in OTP 18 no longer supports legacy insecure ciphers and SSL/TLS versions by default. If needed > for interoperability they can still be configured. > > From the release notes: > "Remove default support for SSL-3.0, due to Poodle vunrability in protocol specification. > > Add padding check for TLS-1.0 to remove Poodle vunrability from TLS 1.0, also add the option padding_check. This option only affects TLS-1.0 connections and if set to false it disables the block cipher padding check to be able to interoperate with legacy software. > > Remove default support for RC4 cipher suites, as they are consider too weak." > > > 2015-09-01 18:44 GMT+02:00 Roberto Ostinelli : >> Dear all, >> I'm using SSL with Cowboy and I keep on getting these kind of errors in the logs: >> >> SSL: hello: tls_handshake.erl:167:Fatal error: insufficient security > > Could not find any common algorithms > > >> SSL: hello: tls_handshake.erl:174:Fatal error: protocol version > > No accetable TLS protocol version > > >> SSL: certify: ssl_alert.erl:93:Fatal error: bad certificate >> SSL: hello: tls_handshake.erl:118:Fatal error: inappropriate fallback > > Prevention of Poodle > > >> SSL: cipher: ssl_cipher.erl:292:Fatal error: bad record mac >> >> I've started seeing those after I've upgraded to Erlang 18.0.2. This wasn't happening in 17.5. >> >> Has anyone seen this, and has clues for me to pinpoint what the problem is? >> Also, is there anyway for me to recover the originating IP address? >> >> Any help appreciated. > > Regards Ingela Erlang/OTP team - Ericsson AB > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From neerajsharma.live@REDACTED Wed Sep 2 21:46:33 2015 From: neerajsharma.live@REDACTED (Neeraj Sharma) Date: Thu, 3 Sep 2015 01:16:33 +0530 Subject: [erlang-questions] Porint erlang on rumpkernel In-Reply-To: References: Message-ID: I have completed the first phase of erlang on rumprun port. I have intentionally reverted all my changes (except one for the ssl build-script) in-line with the remove-till-you-cant-take-away philosophy and additionally avoid maintenance overhead. Having said that the second phase would see those changes though. At present there is no epmd, no smp, while it does have crypto and ssl. I was in the dilemma to go via ose way or *nix, but then the comment stating ose being unsupported made that choice a bit easier though not from porting point of view. https://www.freelists.org/post/rumpkernel-users/Erlang-on-rumprun -Neeraj On Wed, Aug 26, 2015 at 4:01 PM, Neeraj Sharma wrote: > Thanks Lukas and Motiejus. > > On Wed, Aug 26, 2015 at 2:26 PM, Motiejus Jak?tys > wrote: > >> On Wed, Aug 26, 2015 at 8:25 AM, Neeraj Sharma >> wrote: >> > Hi, >> > >> > I am trying to port erlang on rumpkernel and running into some issues. >> > RumpKernel do not have fork/execve, so there are some constraints. >> Please >> > note that I am new to erlang internals, so please be a bit more verbose. >> >> Hi, Neeraj, >> >> Erlang was recently ported to OSv, which also doesn't have fork(). Might >> help: >> >> * https://www.youtube.com/watch?v=tz1TWwc9KhQ >> * https://github.com/cloudius-systems/osv-apps/tree/master/erlang >> >> Good luck! >> >> Motiejus >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Wed Sep 2 22:21:18 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Wed, 2 Sep 2015 16:21:18 -0400 (EDT) Subject: [erlang-questions] Tried this... Thought it interesting... But it didn't work. In-Reply-To: References: <1441121431.32149852@apps.rackspace.com> <15670408.uQR5RUAWqN@changa> Message-ID: <1441225278.990913344@apps.rackspace.com> Hi Jesper, When are you going to write your Erlang book? You have much to teach and I, for one, much appreciate it. All the best, Lloyd -----Original Message----- From: "Jesper Louis Andersen" Sent: Wednesday, September 2, 2015 9:23am To: "Lloyd R. Prentice" Cc: "zxq9" , "Erlang (E-mail)" Subject: Re: [erlang-questions] Tried this... Thought it interesting... But it didn't work. On Wed, Sep 2, 2015 at 4:24 AM, Lloyd R. Prentice wrote: > Your code exhibits a depth of understanding and subtlety that I can only > hope to aspire to. > > Can you help me understand the ?LET(... and ?SUCHTHAT(... expressions? > I agree with Fernando, that it is often nicer to pull these things into a function on its own, and use that function as in: f(X) -> [1,2] ++ e(X). e(X) when X < 5 -> []; e(X) -> [X]. In the QuickCheck models however, you would often end up writing many of such one-off functions, so this is why I sometimes tend to use the [X || test()] scheme. As for ?LET(...) and ?SUCHTHAT(...): They are macros which are used in QuickCheck like tools (Erlang QuickCheck, Proper, Triq, ...). If you have a generator like choose/2 which generates numbers between two endpoints: 6> triq_dom:sample( triq_dom:choose(1, 10)). [5,9,1,3,7,6,1,5,3,8,9] Some times, however, you don't want a raw generator. You want to do something to the result before you use it. The problem however is that the generator generates. So your code would have to: 1. draw a specimen from the generator. 2. apply some function to the drawn specimen 3. return the result of the application, but do so as a new generator doing this! This is what ?LET(...) does[0]. Writing ?LET(I, choose(1,10), I * 7). would return a new generator, which draws a number between 1 and 10, and returns 7 times that number. That is, it returns numbers in the range [7,14,21,28,...,70]. The ?SUCHTHAT(..) macro is used as ?SUCHTHAT(S, Gen, Pred(S)). It picks a specimen from Gen and then tests it against Pred. If Pred(S) returns true, it returns that value. If Pred(S) returns false, it tries generating a new value satisfying the predicate. I.e., generate S from Gen such that P(S) is true. ?SUCHTHAT(..) itself is a generator, doing this internally[1]. You use the ?SUCHTHAT(..) macro to reject certain values which are not desirable. In the case of trying to remove an element from a map, you want to generate an element that is not present in the map in some cases. This is to verify such removal has no effect on the map. So I use a ?SUCHTHAT(...) to make sure the generated target is not already present in the map. I expect to be able to find such a value in relatively few tries. Even considering very large maps with thousands of elements, you can easily generate a new element randomly, which isn't in there. [0] ?LET(..) is a monadic bind over the quickcheck monad. [1] The caveat of ?SUCHTHAT is that you want it to generate a valid value fairly quickly. If you are drawing random hay from a haystack to find a needle, and the haystack is large, then it is a bad idea. You have a random search problem at hand then. -- J. From neerajsharma.live@REDACTED Wed Sep 2 23:04:46 2015 From: neerajsharma.live@REDACTED (Neeraj Sharma) Date: Thu, 3 Sep 2015 02:34:46 +0530 Subject: [erlang-questions] Regarding conflicting pcre casing erlang build failure Message-ID: Hi, While build Erlang (for rump kernel though it's immaterial) there was a build failure, which traced back to erlang's build environment using host pcre version rather than its own. I found a patch ( http://lists.busybox.net/pipermail/buildroot/2015-February/118966.html), which changes the order and it does fix things. I'd like to know whether this can be taken as a formal fix or better alternative exists already? Thanks, Neeraj -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Thu Sep 3 01:21:34 2015 From: vasdeveloper@REDACTED (Kannan) Date: Thu, 3 Sep 2015 04:51:34 +0530 Subject: [erlang-questions] DB for Full Text Search Message-ID: Team, We are evaluating to select a database that natively supports "full text search". Do you have any inputs? The following are key: * Erlang integration * Document orientation * Weights for different keys or keys at different depths (in a JSON document) * Performance -- In-memory should be fine. * Scalability Thanks, Theepan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Thu Sep 3 01:36:12 2015 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Thu, 3 Sep 2015 01:36:12 +0200 Subject: [erlang-questions] DB for Full Text Search In-Reply-To: References: Message-ID: Hello, It looks like ElasticSearch is your answer here. Best Regards, Dmitry >-|-|-(*> > On 03 Sep 2015, at 01:21, Kannan wrote: > > Team, > > We are evaluating to select a database that natively supports "full text search". Do you have any inputs? The following are key: > > * Erlang integration > * Document orientation > * Weights for different keys or keys at different depths (in a JSON document) > * Performance -- In-memory should be fine. > * Scalability > > Thanks, > Theepan > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From schneider@REDACTED Thu Sep 3 09:13:05 2015 From: schneider@REDACTED (Frans Schneider) Date: Thu, 03 Sep 2015 09:13:05 +0200 Subject: [erlang-questions] Erlang DDS implementation or binding Message-ID: <55E7F301.1000106@xs4all.nl> Hi list, I am looking for an DDS Erlang implementation or binding to a DDS library. Google came up with nothing useful. Thanks, Frans From binarin@REDACTED Thu Sep 3 10:52:22 2015 From: binarin@REDACTED (Alexey Lebedeff) Date: Thu, 3 Sep 2015 11:52:22 +0300 Subject: [erlang-questions] Porint erlang on rumpkernel In-Reply-To: References: Message-ID: Hi Neeraj, You could look into https://github.com/erlang/otp/pull/815 for pure-erlang empd. Best, Alexey 2015-09-02 22:46 GMT+03:00 Neeraj Sharma : > I have completed the first phase of erlang on rumprun port. > > I have intentionally reverted all my changes (except one for the ssl > build-script) in-line with the remove-till-you-cant-take-away philosophy > and additionally avoid maintenance overhead. Having said that the second > phase would see those changes though. > > At present there is no epmd, no smp, while it does have crypto and ssl. I > was in the dilemma to go via ose way or *nix, but then the comment stating > ose being unsupported made that choice a bit easier though not from porting > point of view. > > https://www.freelists.org/post/rumpkernel-users/Erlang-on-rumprun > > > -Neeraj > > On Wed, Aug 26, 2015 at 4:01 PM, Neeraj Sharma < > neerajsharma.live@REDACTED> wrote: > >> Thanks Lukas and Motiejus. >> >> On Wed, Aug 26, 2015 at 2:26 PM, Motiejus Jak?tys >> wrote: >> >>> On Wed, Aug 26, 2015 at 8:25 AM, Neeraj Sharma >>> wrote: >>> > Hi, >>> > >>> > I am trying to port erlang on rumpkernel and running into some issues. >>> > RumpKernel do not have fork/execve, so there are some constraints. >>> Please >>> > note that I am new to erlang internals, so please be a bit more >>> verbose. >>> >>> Hi, Neeraj, >>> >>> Erlang was recently ported to OSv, which also doesn't have fork(). Might >>> help: >>> >>> * https://www.youtube.com/watch?v=tz1TWwc9KhQ >>> * https://github.com/cloudius-systems/osv-apps/tree/master/erlang >>> >>> Good luck! >>> >>> Motiejus >>> >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From puzza007@REDACTED Thu Sep 3 11:00:59 2015 From: puzza007@REDACTED (Paul Oliver) Date: Thu, 03 Sep 2015 09:00:59 +0000 Subject: [erlang-questions] DB for Full Text Search In-Reply-To: References: Message-ID: If your primary data store is elsewhere then elasticsearch will probably fulfil those requirements. See https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-0 for details about data loss with elasticsearch (at least as of 1.5.0. It might be worth testing 1.7.0 with jepsen to see if anything has changed). On Thu, Sep 3, 2015 at 1:36 AM Dmitry Kolesnikov wrote: > Hello, > > It looks like ElasticSearch is your answer here. > > Best Regards, > Dmitry > >-|-|-(*> > > > On 03 Sep 2015, at 01:21, Kannan wrote: > > > > Team, > > > > We are evaluating to select a database that natively supports "full text > search". Do you have any inputs? The following are key: > > > > * Erlang integration > > * Document orientation > > * Weights for different keys or keys at different depths (in a JSON > document) > > * Performance -- In-memory should be fine. > > * Scalability > > > > Thanks, > > Theepan > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From neerajsharma.live@REDACTED Thu Sep 3 15:10:02 2015 From: neerajsharma.live@REDACTED (Neeraj Sharma) Date: Thu, 3 Sep 2015 18:40:02 +0530 Subject: [erlang-questions] Porint erlang on rumpkernel In-Reply-To: References: Message-ID: I have invested some time already, but for the near term this will be very useful. Thanks -Neeraj On 3 Sep 2015 14:22, "Alexey Lebedeff" wrote: > Hi Neeraj, > > You could look into https://github.com/erlang/otp/pull/815 for > pure-erlang empd. > > Best, > Alexey > > > 2015-09-02 22:46 GMT+03:00 Neeraj Sharma : > >> I have completed the first phase of erlang on rumprun port. >> >> I have intentionally reverted all my changes (except one for the ssl >> build-script) in-line with the remove-till-you-cant-take-away philosophy >> and additionally avoid maintenance overhead. Having said that the second >> phase would see those changes though. >> >> At present there is no epmd, no smp, while it does have crypto and ssl. I >> was in the dilemma to go via ose way or *nix, but then the comment stating >> ose being unsupported made that choice a bit easier though not from porting >> point of view. >> >> https://www.freelists.org/post/rumpkernel-users/Erlang-on-rumprun >> >> >> -Neeraj >> >> On Wed, Aug 26, 2015 at 4:01 PM, Neeraj Sharma < >> neerajsharma.live@REDACTED> wrote: >> >>> Thanks Lukas and Motiejus. >>> >>> On Wed, Aug 26, 2015 at 2:26 PM, Motiejus Jak?tys >> > wrote: >>> >>>> On Wed, Aug 26, 2015 at 8:25 AM, Neeraj Sharma >>>> wrote: >>>> > Hi, >>>> > >>>> > I am trying to port erlang on rumpkernel and running into some issues. >>>> > RumpKernel do not have fork/execve, so there are some constraints. >>>> Please >>>> > note that I am new to erlang internals, so please be a bit more >>>> verbose. >>>> >>>> Hi, Neeraj, >>>> >>>> Erlang was recently ported to OSv, which also doesn't have fork(). >>>> Might help: >>>> >>>> * https://www.youtube.com/watch?v=tz1TWwc9KhQ >>>> * https://github.com/cloudius-systems/osv-apps/tree/master/erlang >>>> >>>> Good luck! >>>> >>>> Motiejus >>>> >>> >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Thu Sep 3 15:15:26 2015 From: ivan@REDACTED (Ivan Uemlianin) Date: Thu, 3 Sep 2015 14:15:26 +0100 Subject: [erlang-questions] rebar: how to set which C compiler to use Message-ID: <55E847EE.1090103@llaisdy.com> Dear All I have downloaded an erlang project which includes some C files, and which uses rebar as a build tool. I'd like to explicitly set which compiler rebar uses to compile the C files. Is this possible in the rebar.config? Some details: - ./rebar compile returns a load of C compiler errors - adding CFLAGS and LDFLAGS helps a bit - Compiling C files individually I notice different C compilers have very different results. The platform this is all on "cc" is gcc 4.1, while "gcc" (which gets best results) is gcc 4.8. - rebar seems to be using "cc" (judging by the errors). - Setting a CC environment variable seems not to have any effect. With thanks and best wishes Ivan -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED @llaisdy llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From fernando.benavides@REDACTED Thu Sep 3 15:26:00 2015 From: fernando.benavides@REDACTED (Fernando 'Brujo' Benavides) Date: Thu, 3 Sep 2015 10:26:00 -0300 Subject: [erlang-questions] DB for Full Text Search In-Reply-To: References: Message-ID: <7B7FB41D-6970-4C75-A1AB-ADFAA9831D81@inaka.net> And, while you?re at it, in terms of Erlang integration, it?s worth checking tirerl :) Fernando "Brujo" Benavides about.me/elbrujohalcon > On Sep 3, 2015, at 06:00, Paul Oliver wrote: > > If your primary data store is elsewhere then elasticsearch will probably fulfil those requirements. See https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-0 for details about data loss with elasticsearch (at least as of 1.5.0. It might be worth testing 1.7.0 with jepsen to see if anything has changed). > > On Thu, Sep 3, 2015 at 1:36 AM Dmitry Kolesnikov > wrote: > Hello, > > It looks like ElasticSearch is your answer here. > > Best Regards, > Dmitry > >-|-|-(*> > > > On 03 Sep 2015, at 01:21, Kannan > wrote: > > > > Team, > > > > We are evaluating to select a database that natively supports "full text search". Do you have any inputs? The following are key: > > > > * Erlang integration > > * Document orientation > > * Weights for different keys or keys at different depths (in a JSON document) > > * Performance -- In-memory should be fine. > > * Scalability > > > > Thanks, > > Theepan > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon@REDACTED Thu Sep 3 15:28:38 2015 From: jon@REDACTED (Jon Schneider) Date: Thu, 3 Sep 2015 14:28:38 +0100 Subject: [erlang-questions] rebar: how to set which C compiler to use In-Reply-To: <55E847EE.1090103@llaisdy.com> References: <55E847EE.1090103@llaisdy.com> Message-ID: <3472f29c4c0c8e2bda8dc541b849e1e0.squirrel@mail.jschneider.net> Not the official method I'm sure but you could create a little wrapper script called cc earlier in your path that does exec gcc-4.x.y "$@" Jon > - Setting a CC environment variable seems not to have any effect. From erlang@REDACTED Thu Sep 3 15:35:59 2015 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 3 Sep 2015 15:35:59 +0200 Subject: [erlang-questions] DB for Full Text Search In-Reply-To: References: Message-ID: On Thu, Sep 3, 2015 at 1:21 AM, Kannan wrote: > Team, > > We are evaluating to select a database that natively supports "full text > search". Do you have any inputs? The following are key: > > * Erlang integration > * Document orientation > * Weights for different keys or keys at different depths (in a JSON > document) > * Performance -- In-memory should be fine. > * Scalability Sorry but I have to ask: How much memory have to got? (in-memory might be fine, but it's vague, are we talking GBytes, Mbytes, TeraBytes?) How many document/second do you want to index/search? How many words per document? How big in the corpus? What do you want to retrieve (name of file(s) where words occur?) Is the index write append only or must it be updatable Is the index replicated? Security? What at the input documents (text, html, pdf, ...)? What languages are the input documents in? Without stating your requirement it is impossible to give a good answer there is an incredible spectrum of answers. /Joe > > Thanks, > Theepan > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From vasdeveloper@REDACTED Thu Sep 3 22:19:40 2015 From: vasdeveloper@REDACTED (Kannan) Date: Fri, 4 Sep 2015 01:49:40 +0530 Subject: [erlang-questions] DB for Full Text Search In-Reply-To: References: Message-ID: Thanks for all for your valuable inputs. I am looking forward for some more variations. Hi Joe, I hope the story below answers all your questions. Hardware is not a limiting factor. We are in the process of building a global business-customer discovery/social/engagement platform, out from the cloud. If we succeed, it will be a big platform. Contents of the documents will be plain text, and CRUD operations will be performed on the contents. Size - on average, 1MB each. The whole contents of the document must be searchable. There will be replication of static indexes. Security is to be applied on the wires, at the ingress and on sensitive data. Sensitive data will be kept separately, encrypted. Regards, Theepan On Thu, Sep 3, 2015 at 7:05 PM, Joe Armstrong wrote: > On Thu, Sep 3, 2015 at 1:21 AM, Kannan wrote: > > Team, > > > > We are evaluating to select a database that natively supports "full text > > search". Do you have any inputs? The following are key: > > > > * Erlang integration > > * Document orientation > > * Weights for different keys or keys at different depths (in a JSON > > document) > > * Performance -- In-memory should be fine. > > * Scalability > > > > Sorry but I have to ask: > > How much memory have to got? > (in-memory might be fine, but it's vague, are we talking GBytes, > Mbytes, TeraBytes?) > How many document/second do you want to index/search? > How many words per document? > How big in the corpus? > What do you want to retrieve (name of file(s) where words occur?) > Is the index write append only or must it be updatable > Is the index replicated? > Security? > What at the input documents (text, html, pdf, ...)? > What languages are the input documents in? > > Without stating your requirement it is impossible to give a good answer > there is an incredible spectrum of answers. > > /Joe > > > > > Thanks, > > Theepan > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.joseph.davis@REDACTED Fri Sep 4 00:39:22 2015 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Thu, 3 Sep 2015 17:39:22 -0500 Subject: [erlang-questions] Cost of doing +sbwt? In-Reply-To: References: Message-ID: Sooo.... Distributed systems are hard. Sometimes its useful to take a step back and look at everything going on around you so that you aren't missing any important details. Like say, perhaps, the recent inclusion of auditd on your infrastructure. I'm sure that collective groan I'm imagining indicates everyone knows where this one is going. Although, there is a pretty picture linked below that's actually a bit interesting. As it turns out security people showed up and said that we need to do something to comply with some nonsensical acronym and the judgement was that auditd was the easiest way to accomplish that. Well howdy do then. One of our operator/releng engineers found that perf was suggesting auditd was possibly a culprit for increased system CPU so we did the obvious test and turned it off. The sudden drop in system CPU usage made it instantly obvious we'd found the issue. He happened to do that on a VM configured with +sbwt very_long while testing another theory so I ended up reproducing his experiment as well as extending the analysis a bit after he went home today. The image below captures everything I did over the course of an hour that shows quite clearly that auditd is terrible and I hate it. http://i.imgur.com/v5Ggwc4.png A couple things about the picture. In my original post I mentioned that we had three versions for our upgrade. In the graph we've installed the three separate versions on different nodes (ie, db1 is version A, db2 is version B, db3 is version C, where A and B are on R14B01, and C is the same code as B but on 17.5). One of our engineers had that clever idea and its certainly made observing relative CPU changes rather easy. The entire timeline of these graphs are also while there's a sustained load on the cluster. Its nothing special, just enough work to get the CPUs doing things so we can change configurations and observe the relative differences. Also, all three machines are identical hardware and kernels and all that. Also important as that the 17.5 node is using the default value for +sbwt so there's nothing special with that for this data. The few things to note that I found interesting is that while auditd affected both R14B01 and 17.5, the effect on 17.5 was more pronounced. R14B01's sys CPU was increased roughly 2x while it was closer to 3x on 17.5. The second interesting observation was that it wasn't the syscall rules we had for auditd, but the file modification rules that were the cause of significant CPU usage. And the file rules had a much more pronounced effect on 17.5. You can see where the system CPU starts raising noticeably around 18:55 on db3. So I guess I now have to disappear into the bureaucracy to figure out how to maintain compliance while not leaving performance dead at the side of the road. Lastly a few other observations so far after testing more with auditd disabled: On our baseline config (mostly +A 32 and a few limits raised) 17.5 uses ~4% more system CPU than R14B01 +scl false reduces system CPU usage a couple percent +scl false +sbt db decreases slightly from the baseline +sbt db alone increases system CPU noticeably above the baseline +scl false +sfwi 500 doesn't really change CPU usage from just +scl false (but then you'd expect that when its under constant load) So basically the moral of the story is that Erlang is fine. We'll obviously redo the various VM changes on an unmixed cluster to look at performance but its obvious that auditd is our issue for the system CPU issues. Cheers, Paul Davis On Wed, Sep 2, 2015 at 7:23 AM, Jesper Louis Andersen wrote: > If memory serves, R14 can't experience scheduler collapse since it doesn't > do rebalancing of work the same way as R15 and onwards. So I think this is a > red herring. > > Have you established a baseline for the locking in R14? You are contending > on the runqueue lock quite a lot, which could account for all the spinning > you are seeing, but it is hard to say if this is a high or low number > without some baseline you can use to relate. Also, many of the futex() calls > are probably for this contention as well. There is a chance your scheduler > utilization isn't that high, but you are getting into the spinning all the > time. If utilization is fairly low, then the 50% CPU isn't of concern: just > load the system more :) > > Chances are you are hunting the wrong mark as well: You have 2 or more > pathologies, and they overlap in what you are seeing. Hence you get > distracted by the noise generated by the other problems. It may be you have > a CPU problem and on top of that, you have a latency/blocking problem in an > I/O layer as well. One could account for the latency spikes, whereas the > other would explain the high CPU. But if you don't know you are sitting with > two problems in the first place, then their cooperation in the system > confuses you. > > If you have elevated CPU, then a snapshot of the current thread stacks at > 97hz per second[1] should tell you where things are taking time. This in > itself could give you hints as to where you are spending all of your time in > the system, and also what you are spinning on, if anything. > > [1] Old trick: Never snapshot at 100hz or something which means you can get > into phase with other jobs. Pick some prime around your target. > > > On Tue, Sep 1, 2015 at 9:42 PM, Lukas Larsson > wrote: >> >> On Tue, Sep 1, 2015 at 9:14 PM, Paul Davis >> wrote: >>> >>> Also, does anyone have a quick pointer to where the busy wait loop is? >>> After I look at the scheduler time I was going to go find that code >>> and see if I couldn't come up with a better idea of what exactly might >>> be changing with that setting. >> >> >> This should be the code that does the waiting: >> https://github.com/erlang/otp/blob/master/erts/lib_src/pthread/ethr_event.c#L65-L161 >> >> The mutex implementation that calls it is in here: >> https://github.com/erlang/otp/blob/master/erts/lib_src/common/ethr_mutex.c >> >> The different spin options are set here: >> https://github.com/erlang/otp/blob/master/erts/emulator/beam/erl_process.c#L5325-L5364 >> >> There are also a couple of other places where it spind in erl_process.c, >> just search for spin and you'll find them :) > > > > > -- > J. From rdm@REDACTED Fri Sep 4 01:03:23 2015 From: rdm@REDACTED (Rich Morin) Date: Thu, 3 Sep 2015 16:03:23 -0700 Subject: [erlang-questions] Cost of doing +sbwt? In-Reply-To: References: Message-ID: Have you considered offering to re-implement auditd in Erlang? That might satisfy the security people but give acceptable performance. -r -- http://www.cfcl.com/rdm Rich Morin rdm@REDACTED http://www.cfcl.com/rdm/resume San Bruno, CA, USA +1 650-873-7841 Software system design, development, and documentation From paul.joseph.davis@REDACTED Fri Sep 4 01:13:46 2015 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Thu, 3 Sep 2015 18:13:46 -0500 Subject: [erlang-questions] Cost of doing +sbwt? In-Reply-To: References: Message-ID: On Thu, Sep 3, 2015 at 6:03 PM, Rich Morin wrote: > Have you considered offering to re-implement auditd in Erlang? That > might satisfy the security people but give acceptable performance. > > -r > Its my personal experience that security people are never satisfied. If it were up to them I'm sure all of our computers would be turned off and locked in a safe under the desk. > -- > http://www.cfcl.com/rdm Rich Morin rdm@REDACTED > http://www.cfcl.com/rdm/resume San Bruno, CA, USA +1 650-873-7841 > > Software system design, development, and documentation > > From neerajsharma.live@REDACTED Fri Sep 4 07:01:54 2015 From: neerajsharma.live@REDACTED (Neeraj Sharma) Date: Fri, 4 Sep 2015 10:31:54 +0530 Subject: [erlang-questions] Porint erlang on rumpkernel In-Reply-To: References: Message-ID: The pure-erlang patch though un-reviewed works for me and was pretty easy to integrate too. Yay! I have shelved my integration work on using the stable c-based epmd for later, since there are other interesting things to do right now ;) announcement on rumprun in case you care: https://www.freelists.org/post/rumpkernel-users/You-can-now-run-a-cluster-of-Erlangrumprun-unikernel -Neeraj On Thu, Sep 3, 2015 at 6:40 PM, Neeraj Sharma wrote: > I have invested some time already, but for the near term this will be very > useful. > > Thanks > > -Neeraj > On 3 Sep 2015 14:22, "Alexey Lebedeff" wrote: > >> Hi Neeraj, >> >> You could look into https://github.com/erlang/otp/pull/815 for >> pure-erlang empd. >> >> Best, >> Alexey >> >> >> 2015-09-02 22:46 GMT+03:00 Neeraj Sharma : >> >>> I have completed the first phase of erlang on rumprun port. >>> >>> I have intentionally reverted all my changes (except one for the ssl >>> build-script) in-line with the remove-till-you-cant-take-away philosophy >>> and additionally avoid maintenance overhead. Having said that the second >>> phase would see those changes though. >>> >>> At present there is no epmd, no smp, while it does have crypto and ssl. >>> I was in the dilemma to go via ose way or *nix, but then the comment >>> stating ose being unsupported made that choice a bit easier though not from >>> porting point of view. >>> >>> https://www.freelists.org/post/rumpkernel-users/Erlang-on-rumprun >>> >>> >>> -Neeraj >>> >>> On Wed, Aug 26, 2015 at 4:01 PM, Neeraj Sharma < >>> neerajsharma.live@REDACTED> wrote: >>> >>>> Thanks Lukas and Motiejus. >>>> >>>> On Wed, Aug 26, 2015 at 2:26 PM, Motiejus Jak?tys < >>>> desired.mta@REDACTED> wrote: >>>> >>>>> On Wed, Aug 26, 2015 at 8:25 AM, Neeraj Sharma >>>>> wrote: >>>>> > Hi, >>>>> > >>>>> > I am trying to port erlang on rumpkernel and running into some >>>>> issues. >>>>> > RumpKernel do not have fork/execve, so there are some constraints. >>>>> Please >>>>> > note that I am new to erlang internals, so please be a bit more >>>>> verbose. >>>>> >>>>> Hi, Neeraj, >>>>> >>>>> Erlang was recently ported to OSv, which also doesn't have fork(). >>>>> Might help: >>>>> >>>>> * https://www.youtube.com/watch?v=tz1TWwc9KhQ >>>>> * https://github.com/cloudius-systems/osv-apps/tree/master/erlang >>>>> >>>>> Good luck! >>>>> >>>>> Motiejus >>>>> >>>> >>>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Fri Sep 4 10:09:31 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 4 Sep 2015 10:09:31 +0200 Subject: [erlang-questions] Cost of doing +sbwt? In-Reply-To: References: Message-ID: On Fri, Sep 4, 2015 at 12:39 AM, Paul Davis wrote: > Sooo.... > > Distributed systems are hard. > One have to celebrate the small victories. You know what is causing the trouble in the first place, and this is an important milestone. Surely, auditd will impose an overhead in the running system, but this impact it is having seems excessively high for an audit system to me. The slightly higher CPU is to be expected, given that 15+ and onwards does spinning waiting for runq entries. One nice trick is to enable monitor events (erlang:system_monitor, see Basho's riak_sysmon as well). These will warn you about long schedules. In particular long schedules on system calls that are expected to return quickly :) Perhaps this will help in the future when someone changes assumptions underneath the setup. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuncer.ayaz@REDACTED Fri Sep 4 11:40:32 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Fri, 4 Sep 2015 11:40:32 +0200 Subject: [erlang-questions] rebar: how to set which C compiler to use In-Reply-To: <55E847EE.1090103@llaisdy.com> References: <55E847EE.1090103@llaisdy.com> Message-ID: On Thu, Sep 3, 2015 at 3:15 PM, Ivan Uemlianin wrote: > Dear All > > I have downloaded an erlang project which includes some C files, and > which uses rebar as a build tool. I'd like to explicitly set which > compiler rebar uses to compile the C files. Is this possible in the > rebar.config? If you don't want to use environment variables, you can configure CC in rebar.config's {port_env, []}. > Some details: > > - ./rebar compile returns a load of C compiler errors > - adding CFLAGS and LDFLAGS helps a bit > - Compiling C files individually I notice different C compilers have > very different results. The platform this is all on "cc" is gcc 4.1, > while "gcc" (which gets best results) is gcc 4.8. > - rebar seems to be using "cc" (judging by the errors). cc is used for portability, as not every platform has CC=gcc. Therefore, default CC=cc and CXX=c++. > - Setting a CC environment variable seems not to have any effect. Are you sure about that? $ env CC=ivancc rebar compile [...] sh: 1: exec: ivancc: not found In addition to the usual CC, CXX, *FLAGS, and Erlang-specific *FLAGS, there's also explicit support for cross-compilation: Cross-arch environment variables: REBAR_TARGET_ARCH to set the tool chain name to use REBAR_TARGET_ARCH_WORDSIZE optional (if CC fails to determine word size) fallback word size is 32 REBAR_TARGET_ARCH_VSN optional (if a special version of CC/CXX is requested) From ivan@REDACTED Fri Sep 4 11:57:07 2015 From: ivan@REDACTED (Ivan Uemlianin) Date: Fri, 4 Sep 2015 10:57:07 +0100 Subject: [erlang-questions] rebar: how to set which C compiler to use In-Reply-To: References: <55E847EE.1090103@llaisdy.com> Message-ID: <55E96AF3.9090104@llaisdy.com> Dear Tuncer Thanks for your help. You are right, $ env CC=gcc works, I was doing it wrong. Best wishes Ivan On 04/09/2015 10:40, Tuncer Ayaz wrote: >> - Setting a CC environment variable seems not to have any effect. > Are you sure about that? > > $ env CC=ivancc rebar compile > [...] > sh: 1: exec: ivancc: not found -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED @llaisdy llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From tuncer.ayaz@REDACTED Fri Sep 4 17:01:40 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Fri, 4 Sep 2015 17:01:40 +0200 Subject: [erlang-questions] rebar: how to set which C compiler to use In-Reply-To: <55E96AF3.9090104@llaisdy.com> References: <55E847EE.1090103@llaisdy.com> <55E96AF3.9090104@llaisdy.com> Message-ID: On Fri, Sep 4, 2015 at 11:57 AM, Ivan Uemlianin wrote: > Dear Tuncer > > Thanks for your help. You are right, $ env CC=gcc works, I was doing > it wrong. I'm glad to hear that. Thanks for asking the question, Ivan, as it reminded me that I forgot to merge existing code comments with what 'rebar help compile' prints for the port compiler plugin: https://github.com/rebar/rebar/pull/540 From rtrlists@REDACTED Fri Sep 4 22:10:58 2015 From: rtrlists@REDACTED (Robert Raschke) Date: Fri, 4 Sep 2015 22:10:58 +0200 Subject: [erlang-questions] Erlang DDS implementation or binding In-Reply-To: <55E7F301.1000106@xs4all.nl> References: <55E7F301.1000106@xs4all.nl> Message-ID: Hi Frans, what is DDS? Cheers, Robby On Sep 3, 2015 9:13 AM, "Frans Schneider" wrote: > Hi list, > > I am looking for an DDS Erlang implementation or binding to a DDS library. > Google came up with nothing useful. > > Thanks, > > Frans > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From reynaldomic@REDACTED Fri Sep 4 22:21:41 2015 From: reynaldomic@REDACTED (Reynaldo Baquerizo) Date: Fri, 4 Sep 2015 15:21:41 -0500 Subject: [erlang-questions] Erlang DDS implementation or binding In-Reply-To: References: <55E7F301.1000106@xs4all.nl> Message-ID: Data Distribution Service There is a recent talk on the topic http://www.erlang-factory.com/euc2015/radoslaw-gruchalski -- On Fri, Sep 4, 2015 at 3:10 PM, Robert Raschke wrote: > Hi Frans, > > what is DDS? > > Cheers, > Robby > On Sep 3, 2015 9:13 AM, "Frans Schneider" wrote: > >> Hi list, >> >> I am looking for an DDS Erlang implementation or binding to a DDS >> library. Google came up with nothing useful. >> >> Thanks, >> >> Frans >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.joseph.davis@REDACTED Fri Sep 4 22:27:12 2015 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Fri, 4 Sep 2015 15:27:12 -0500 Subject: [erlang-questions] Cost of doing +sbwt? In-Reply-To: References: Message-ID: On Fri, Sep 4, 2015 at 3:09 AM, Jesper Louis Andersen wrote: > > On Fri, Sep 4, 2015 at 12:39 AM, Paul Davis > wrote: >> >> Sooo.... >> >> Distributed systems are hard. > > > One have to celebrate the small victories. You know what is causing the > trouble in the first place, and this is an important milestone. Surely, > auditd will impose an overhead in the running system, but this impact it is > having seems excessively high for an audit system to me. > Yeah, its interesting to see that it seems to affect 17.5 so much more than R14B01 but beyond that I don't plan on digging into the kernel to satisfy any more curiosity on that front. What's most amusing is that we spent so much time testing 17.5 and then auditd got shoved out between testing and deployment. For some value of amusing I guess. > The slightly higher CPU is to be expected, given that 15+ and onwards does > spinning waiting for runq entries. One nice trick is to enable monitor > events (erlang:system_monitor, see Basho's riak_sysmon as well). These will > warn you about long schedules. In particular long schedules on system calls > that are expected to return quickly :) Perhaps this will help in the future > when someone changes assumptions underneath the setup. > > > -- > J. I agree. The slightly higher CPU of a couple percent or so is well within reason. Also good call on the long schedule monitors. I'm gonna add a ticket now for that so I don't forget to add it now that 17.5 is a thing for us. Thanks for all your help and suggestions. I knew it was definitely a stretch with "17.5 makes our clusters sad" and thankfully we had a heads up engineer catch the auditd difference. Distributed systems are hard. Cheers, Paul Davis From achowdhury918@REDACTED Sat Sep 5 01:53:14 2015 From: achowdhury918@REDACTED (Akash Chowdhury) Date: Fri, 4 Sep 2015 19:53:14 -0400 Subject: [erlang-questions] =?utf-8?q?Setting_maximum_amount_of_data_alloc?= =?utf-8?q?ated_by_an_Erlang_node=2E=E2=80=8F?= Message-ID: Hi, I would like to set a number for a) simultaneously alive maximum Erlang processes in an Erlang node and b) for maximum amount of data allocated by an Erlang node I know that for process, I have to use the *+P * command-line flag. But what I have to do for setting maximum amount of data allocated by an Erlang node. I looked at the following documentation: http://www.erlang.org/doc/efficiency_guide/advanced.html It says setting maximum amount of data allocated by an Erlang node is possible, but it doesn?t say how. Any help will be highly appreciated. Thanks. - Akash -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Sat Sep 5 02:15:01 2015 From: vasdeveloper@REDACTED (Kannan) Date: Sat, 5 Sep 2015 05:45:01 +0530 Subject: [erlang-questions] =?utf-8?q?Setting_maximum_amount_of_data_alloc?= =?utf-8?q?ated_by_an_Erlang_node=2E=E2=80=8F?= In-Reply-To: References: Message-ID: Please look at OS level configurations. If you are using Linux or Unix system, check ULIMIT command. Theepan On Sat, Sep 5, 2015 at 5:23 AM, Akash Chowdhury wrote: > Hi, > > I would like to set a number for > > a) simultaneously alive maximum Erlang processes in an Erlang node > and > > b) for maximum amount of data allocated by an Erlang node > > I know that for process, I have to use the *+P > * command-line > flag. But what I have to do for setting maximum amount of data allocated by > an Erlang node. > > I looked at the following documentation: > > http://www.erlang.org/doc/efficiency_guide/advanced.html > > It says setting maximum amount of data allocated by an Erlang node is > possible, but it doesn?t say how. > > Any help will be highly appreciated. > > Thanks. > > - > > Akash > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Sat Sep 5 07:22:06 2015 From: vasdeveloper@REDACTED (Kannan) Date: Sat, 5 Sep 2015 10:52:06 +0530 Subject: [erlang-questions] =?utf-8?q?Setting_maximum_amount_of_data_alloc?= =?utf-8?q?ated_by_an_Erlang_node=2E=E2=80=8F?= In-Reply-To: References: Message-ID: What is the use case? Why are you looking at limiting it in the first place? May be, you are trying to cure the symptom than the root cause. Theepan On Sat, Sep 5, 2015 at 7:14 AM, Zulfiqer Sekender wrote: > Thanks for the reply. Can't we do anything from Erlang level? > Thanks. > > ------------------------------ > Date: Sat, 5 Sep 2015 05:45:01 +0530 > From: vasdeveloper@REDACTED > To: achowdhury918@REDACTED > CC: erlang-questions@REDACTED > Subject: Re: [erlang-questions] Setting maximum amount of data allocated > by an Erlang node.? > > > Please look at OS level configurations. If you are using Linux or Unix > system, check ULIMIT command. > > Theepan > > On Sat, Sep 5, 2015 at 5:23 AM, Akash Chowdhury > wrote: > > Hi, > > I would like to set a number for > > a) simultaneously alive maximum Erlang processes in an Erlang node > and > > b) for maximum amount of data allocated by an Erlang node > > I know that for process, I have to use the *+P > * command-line > flag. But what I have to do for setting maximum amount of data allocated by > an Erlang node. > > I looked at the following documentation: > > http://www.erlang.org/doc/efficiency_guide/advanced.html > > It says setting maximum amount of data allocated by an Erlang node is > possible, but it doesn?t say how. > > Any help will be highly appreciated. > > Thanks. > > - > > Akash > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ erlang-questions mailing > list erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From msekender@REDACTED Sat Sep 5 03:44:33 2015 From: msekender@REDACTED (Zulfiqer Sekender) Date: Sat, 5 Sep 2015 01:44:33 +0000 Subject: [erlang-questions] =?cp1256?q?Setting_maximum_amount_of_data_allo?= =?cp1256?q?cated_by_an_Erlang_node=2E=FE?= In-Reply-To: References: , Message-ID: Thanks for the reply. Can't we do anything from Erlang level? Thanks. Date: Sat, 5 Sep 2015 05:45:01 +0530 From: vasdeveloper@REDACTED To: achowdhury918@REDACTED CC: erlang-questions@REDACTED Subject: Re: [erlang-questions] Setting maximum amount of data allocated by an Erlang node.? Please look at OS level configurations. If you are using Linux or Unix system, check ULIMIT command. Theepan On Sat, Sep 5, 2015 at 5:23 AM, Akash Chowdhury wrote: Hi,I would like to set a number fora) simultaneously alive maximum Erlang processes in an Erlang node andb) for maximum amount of data allocated by an Erlang nodeI know that for process, I have to use the +P command-line flag. But what I have to do for setting maximum amount of data allocated by an Erlang node.I looked at the following documentation:http://www.erlang.org/doc/efficiency_guide/advanced.htmlIt says setting maximum amount of data allocated by an Erlang node is possible, but it doesn?t say how.Any help will be highly appreciated.Thanks.-Akash _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From schneider@REDACTED Sat Sep 5 09:47:37 2015 From: schneider@REDACTED (Frans Schneider) Date: Sat, 05 Sep 2015 09:47:37 +0200 Subject: [erlang-questions] Erlang DDS implementation or binding In-Reply-To: References: <55E7F301.1000106@xs4all.nl> Message-ID: <55EA9E19.6020303@xs4all.nl> Quoting OMG's DDS site [1]: "The Data Distribution Service (DDS?) is a middleware protocol and API standard for data-centric connectivity from the Object Management Group? (OMG?)." I am looking for the core functionality DCPS and RTPS [2, 3]. [1] http://portals.omg.org/dds/ [2] http://www.omg.org/spec/DDS/ [3] http://www.omg.org/spec/DDSI-RTPS/ On 09/04/2015 10:10 PM, Robert Raschke wrote: > > Hi Frans, > > what is DDS? > > Cheers, > Robby > > On Sep 3, 2015 9:13 AM, "Frans Schneider" > wrote: > > Hi list, > > I am looking for an DDS Erlang implementation or binding to a DDS > library. Google came up with nothing useful. > > Thanks, > > Frans > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Sat Sep 5 10:11:20 2015 From: ulf@REDACTED (Ulf Wiger) Date: Sat, 5 Sep 2015 10:11:20 +0200 Subject: [erlang-questions] =?utf-8?q?Setting_maximum_amount_of_data_alloc?= =?utf-8?q?ated_by_an_Erlang_node=2E=E2=80=8F?= In-Reply-To: References: Message-ID: > On 05 Sep 2015, at 03:44, Zulfiqer Sekender wrote: > > Thanks for the reply. Can't we do anything from Erlang level? A few things, but also note that hard limits on memory usage can lead to very un-erlang-like behavior. - Adding the option `+P ` to the `erl` command line will adjust the maximum number of simultaneous processes, but the actual limit chosen may differ from the one you specify. Please see the documentation. - If you want to let your application adjust preemptively as you are approaching certain limits, this could be done using something like Jobs [1], which supports a sampling approach to react to passing high watermarks. Samplers can rely e.g. on the Memsup functionality, the memory usage BIFs, monitoring the size of ets tables, system_info(process_count), etc. - whatever works as good proxies for resource usage in your system. BR, Ulf W [1] https://github.com/uwiger/jobs Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Sat Sep 5 10:20:07 2015 From: mjtruog@REDACTED (Michael Truog) Date: Sat, 05 Sep 2015 01:20:07 -0700 Subject: [erlang-questions] =?utf-8?q?Setting_maximum_amount_of_data_alloc?= =?utf-8?q?ated_by_an_Erlang_node=2E=E2=80=8F?= In-Reply-To: References: , Message-ID: <55EAA5B7.2000802@gmail.com> On 09/04/2015 06:44 PM, Zulfiqer Sekender wrote: > Thanks for the reply. Can't we do anything from Erlang level? > Thanks. Its a bit complex to track the exact amount of memory inside the Erlang VM in Erlang source code to do something useful with the information due to the various memory pools and the garbage collection. However, there is some code at https://github.com/okeuday/erlang_term/ which returns the upper limit for some Erlang term's memory consumption. The result can then be used in Erlang source code for determining how to avoid excessive memory consumption, to avoid a crash. CloudI uses this to limit the incoming queue of a CloudI service with the queue_size service configuration option (http://cloudi.org/api.html#2_services_add_config_opts). > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > Date: Sat, 5 Sep 2015 05:45:01 +0530 > From: vasdeveloper@REDACTED > To: achowdhury918@REDACTED > CC: erlang-questions@REDACTED > Subject: Re: [erlang-questions] Setting maximum amount of data allocated by an Erlang node.? > > Please look at OS level configurations. If you are using Linux or Unix system, check ULIMIT command. > > Theepan > > On Sat, Sep 5, 2015 at 5:23 AM, Akash Chowdhury > wrote: > > Hi, > > I would like to set a number for > > a)simultaneously alive maximum Erlang processes in an Erlang node and > > b)for maximum amount of data allocated by an Erlang node > > I know that for process, I have to use the*+P *command-line flag. But what I have to do for setting maximum amount of data allocated by an Erlang node. > > I looked at the following documentation: > > http://www.erlang.org/doc/efficiency_guide/advanced.html > > It says setting maximum amount of data allocated by an Erlang node is possible, but it doesn?t say how. > > Any help will be highly appreciated. > > Thanks. > > - > > Akash > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Sat Sep 5 10:35:06 2015 From: ulf@REDACTED (Ulf Wiger) Date: Sat, 5 Sep 2015 10:35:06 +0200 Subject: [erlang-questions] Erlang DDS implementation or binding In-Reply-To: <55E7F301.1000106@xs4all.nl> References: <55E7F301.1000106@xs4all.nl> Message-ID: <477EFC18-0C06-4A8F-A70F-6DC7301365FB@feuerlabs.com> Not that I?ve come close to trying it, but some googling indicates that you should be able to talk DDS via DBus. If so, perhaps you can try https://github.com/tonyrog/dbus ? BR, Ulf W > On 03 Sep 2015, at 09:13, Frans Schneider wrote: > > Hi list, > > I am looking for an DDS Erlang implementation or binding to a DDS library. Google came up with nothing useful. > > Thanks, > > Frans > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From achowdhury918@REDACTED Sat Sep 5 14:07:25 2015 From: achowdhury918@REDACTED (Akash Chowdhury) Date: Sat, 5 Sep 2015 08:07:25 -0400 Subject: [erlang-questions] =?utf-8?q?Setting_maximum_amount_of_data_alloc?= =?utf-8?q?ated_by_an_Erlang_node=2E=E2=80=8F?= In-Reply-To: <55EAA5B7.2000802@gmail.com> References: <55EAA5B7.2000802@gmail.com> Message-ID: The use case is that the application is using huge memory. In some occurrences, one node grew until 100GB and the application as well as the box crashed. We would like to setup a limit so that application and box doesn't crash. Thanks. On Sat, Sep 5, 2015 at 4:20 AM, Michael Truog wrote: > On 09/04/2015 06:44 PM, Zulfiqer Sekender wrote: > > Thanks for the reply. Can't we do anything from Erlang level? > Thanks. > > Its a bit complex to track the exact amount of memory inside the Erlang VM > in Erlang source code to do > something useful with the information due to the various memory pools and > the garbage collection. > However, there is some code at https://github.com/okeuday/erlang_term/ > which returns the upper limit > for some Erlang term's memory consumption. The result can then be used in > Erlang source code for > determining how to avoid excessive memory consumption, to avoid a crash. > CloudI uses this to limit the > incoming queue of a CloudI service with the queue_size service > configuration option > (http://cloudi.org/api.html#2_services_add_config_opts). > > > > ------------------------------ > Date: Sat, 5 Sep 2015 05:45:01 +0530 > From: vasdeveloper@REDACTED > To: achowdhury918@REDACTED > CC: erlang-questions@REDACTED > Subject: Re: [erlang-questions] Setting maximum amount of data allocated > by an Erlang node.? > > Please look at OS level configurations. If you are using Linux or Unix > system, check ULIMIT command. > > Theepan > > On Sat, Sep 5, 2015 at 5:23 AM, Akash Chowdhury > wrote: > > Hi, > > I would like to set a number for > > a) simultaneously alive maximum Erlang processes in an Erlang node > and > > b) for maximum amount of data allocated by an Erlang node > > I know that for process, I have to use the *+P > * command-line > flag. But what I have to do for setting maximum amount of data allocated by > an Erlang node. > > I looked at the following documentation: > > http://www.erlang.org/doc/efficiency_guide/advanced.html > > It says setting maximum amount of data allocated by an Erlang node is > possible, but it doesn?t say how. > > Any help will be highly appreciated. > > Thanks. > > - > > Akash > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ erlang-questions mailing > list erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From schneider@REDACTED Sat Sep 5 16:15:36 2015 From: schneider@REDACTED (Schneider) Date: Sat, 5 Sep 2015 16:15:36 +0200 Subject: [erlang-questions] Erlang DDS implementation or binding In-Reply-To: <477EFC18-0C06-4A8F-A70F-6DC7301365FB@feuerlabs.com> References: <55E7F301.1000106@xs4all.nl> <477EFC18-0C06-4A8F-A70F-6DC7301365FB@feuerlabs.com> Message-ID: DBus and OMG's DDS are two totally different beasts. It is more a machine to machine thing on a pub sub basis [1]. And yes, it is a true beast. Frans [1] https://en.m.wikipedia.org/wiki/Data_Distribution_Service > Op 5 sep. 2015 om 10:35 heeft Ulf Wiger het volgende geschreven: > > Not that I?ve come close to trying it, but some googling indicates that you should be able to talk DDS via DBus. > > If so, perhaps you can try https://github.com/tonyrog/dbus ? > > BR, > Ulf W > > >> On 03 Sep 2015, at 09:13, Frans Schneider wrote: >> >> Hi list, >> >> I am looking for an DDS Erlang implementation or binding to a DDS library. Google came up with nothing useful. >> >> Thanks, >> >> Frans >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > From vasdeveloper@REDACTED Sat Sep 5 21:27:16 2015 From: vasdeveloper@REDACTED (Kannan) Date: Sun, 6 Sep 2015 00:57:16 +0530 Subject: [erlang-questions] =?utf-8?q?Setting_maximum_amount_of_data_alloc?= =?utf-8?q?ated_by_an_Erlang_node=2E=E2=80=8F?= In-Reply-To: References: <55EAA5B7.2000802@gmail.com> Message-ID: Akash, First step of curing this is finding out what is eating the memory. Data on the sockets, opened files, in-memory database, atom storage, in-memory data structure, number of Erlang processes spawned with each using considerable amount of memory... there are plenty more. If you figure out what is eating the memory, you can put a CAP on that specific thing, than just limiting the total memory available to the VM/OS process. I would suggest to find out the root cause and revisit the design. Theepan On Sat, Sep 5, 2015 at 5:37 PM, Akash Chowdhury wrote: > The use case is that the application is using huge memory. In some > occurrences, one node grew until 100GB and the application as well as the > box crashed. We would like to setup a limit so that application and box > doesn't crash. > Thanks. > > On Sat, Sep 5, 2015 at 4:20 AM, Michael Truog wrote: > >> On 09/04/2015 06:44 PM, Zulfiqer Sekender wrote: >> >> Thanks for the reply. Can't we do anything from Erlang level? >> Thanks. >> >> Its a bit complex to track the exact amount of memory inside the Erlang >> VM in Erlang source code to do >> something useful with the information due to the various memory pools and >> the garbage collection. >> However, there is some code at https://github.com/okeuday/erlang_term/ >> which returns the upper limit >> for some Erlang term's memory consumption. The result can then be used >> in Erlang source code for >> determining how to avoid excessive memory consumption, to avoid a crash. >> CloudI uses this to limit the >> incoming queue of a CloudI service with the queue_size service >> configuration option >> (http://cloudi.org/api.html#2_services_add_config_opts). >> >> >> >> ------------------------------ >> Date: Sat, 5 Sep 2015 05:45:01 +0530 >> From: vasdeveloper@REDACTED >> To: achowdhury918@REDACTED >> CC: erlang-questions@REDACTED >> Subject: Re: [erlang-questions] Setting maximum amount of data allocated >> by an Erlang node.? >> >> Please look at OS level configurations. If you are using Linux or Unix >> system, check ULIMIT command. >> >> Theepan >> >> On Sat, Sep 5, 2015 at 5:23 AM, Akash Chowdhury >> wrote: >> >> Hi, >> >> I would like to set a number for >> >> a) simultaneously alive maximum Erlang processes in an Erlang node >> and >> >> b) for maximum amount of data allocated by an Erlang node >> >> I know that for process, I have to use the *+P >> * command-line >> flag. But what I have to do for setting maximum amount of data allocated by >> an Erlang node. >> >> I looked at the following documentation: >> >> http://www.erlang.org/doc/efficiency_guide/advanced.html >> >> It says setting maximum amount of data allocated by an Erlang node is >> possible, but it doesn?t say how. >> >> Any help will be highly appreciated. >> >> Thanks. >> >> - >> >> Akash >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> _______________________________________________ erlang-questions mailing >> list erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> _______________________________________________ >> erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sat Sep 5 23:24:40 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 5 Sep 2015 23:24:40 +0200 Subject: [erlang-questions] =?utf-8?q?Setting_maximum_amount_of_data_alloc?= =?utf-8?q?ated_by_an_Erlang_node=2E=E2=80=8F?= In-Reply-To: References: Message-ID: On Sat, Sep 5, 2015 at 1:53 AM, Akash Chowdhury wrote: > b) for maximum amount of data allocated by an Erlang node One thing that has not been touched yet is that the Erlang runtime is a bit different in its allocation policy compared to many other programming languages with garbage collectors. In the other category, the runtime wants as much memory as possible, and you can make the system run faster by giving it more memory to play with. The thing here is that it improves productivity of the system because garbage collection happens more rarely than it noramlly would do. So you can make a trade-off: Lower the available memory the GC has available. This lowers productivity, but makes the program stay within bounds. The BEAM Erlang runtime system on the other hand works differently. It usually doesn't allocate data unless it needs it, and it tends to grow processes slowly as they need more and more memory. Hence, the BEAM has much smaller overhead in memory compared to a GC solution. This suggests that if a system uses 100 Gigabytes of memory, then this is almost all taken up by something in the program. Thus, it will not help setting a system limit, which will just crash the VM. You will have to make sure your system doesn't overallocate memory by looking at the amount of data it is storing and limiting itself. The first step should be, as Theepan writes, to investigate what takes up memory in the system, and then firgure out hwo to limit that memory usage. Alternatively, if you can't compress memory more, is to look at distribution. But this open s up a nother can of worms because distribted systems have their own set of complications you then have to handle. However, if you can't keep your data resident on one machine, you either have to look into storing data temporarily on disk. Or to employ more than a single machine containing the data in memory. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Sun Sep 6 02:59:20 2015 From: mjtruog@REDACTED (Michael Truog) Date: Sat, 05 Sep 2015 17:59:20 -0700 Subject: [erlang-questions] Erlang DDS implementation or binding In-Reply-To: References: <55E7F301.1000106@xs4all.nl> <477EFC18-0C06-4A8F-A70F-6DC7301365FB@feuerlabs.com> Message-ID: <55EB8FE8.1000905@gmail.com> On 09/05/2015 07:15 AM, Schneider wrote: > DBus and OMG's DDS are two totally different beasts. It is more a machine to machine thing on a pub sub basis [1]. And yes, it is a true beast. There is overlap between the functionality in a DDS implementation and CloudI (http://cloudi.org). The obvious difference is that DDS was created by a committee (OMG) standardization process to create a specification and implementations based on the specification revision. At a high-level, CloudI focuses on finer-grained concurrency (execution thread level) rather than the coarser-grained concurrency of separate servers with monolithic server applications. The normal argument would be: "I don't care, I will run everything in a container so I can still have some type of finer-grained concurrency without thinking about it due to virtualization with my monolithic server application (besides, I often have no choice due to it being legacy software maintenance) and I accept the inefficiency of this approach!" which would be for pursuing a DDS-type approach instead of CloudI. Another high-level difference is that CloudI is more focused on fault-tolerance of software's execution which is an aspect of QoS. CloudI can easily retry service requests for messaging QoS (Erlang/Elixir source code can currently use cloudi_queue and cloudi_service_queue for doing an automatic retry). CloudI service requests have a timeout that is decremented based on the time spent processing or queued, so the messaging within CloudI has a finite lifetime. CloudI service requests also have a priority which can change which service request will be processed first, on the CloudI service which queued the incoming service request. CloudI nodes are using distributed Erlang communication for a fully connected network, so LAN usage focused on fault-tolerance in an AP-type distributed system. I haven't looked at the deeper details of DDS to summarize all the differences, beyond having different protocols and API functions, but I wanted to mention CloudI due to the overlap I saw with the DDS description. There is a messaging comparison at http://cloudi.org/faq.html#1_Messaging which may help. > > Frans > > [1] https://en.m.wikipedia.org/wiki/Data_Distribution_Service > > >> Op 5 sep. 2015 om 10:35 heeft Ulf Wiger het volgende geschreven: >> >> Not that I?ve come close to trying it, but some googling indicates that you should be able to talk DDS via DBus. >> >> If so, perhaps you can try https://github.com/tonyrog/dbus ? >> >> BR, >> Ulf W >> >> >>> On 03 Sep 2015, at 09:13, Frans Schneider wrote: >>> >>> Hi list, >>> >>> I am looking for an DDS Erlang implementation or binding to a DDS library. Google came up with nothing useful. >>> >>> Thanks, >>> >>> Frans >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. >> http://feuerlabs.com >> >> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From vasdeveloper@REDACTED Sun Sep 6 20:45:39 2015 From: vasdeveloper@REDACTED (Theepan) Date: Mon, 7 Sep 2015 00:15:39 +0530 Subject: [erlang-questions] Accessing a single value from MAPS Message-ID: Team, What is the syntactic sugar to access single value from MAPS? I know there is this "get" method, but will be good to have something like DOT notation, like we do with records. Could not find it in the documents. It will be handy to access nested MAPS. Thanks, Theepan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sun Sep 6 21:02:58 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sun, 6 Sep 2015 21:02:58 +0200 Subject: [erlang-questions] Accessing a single value from MAPS In-Reply-To: References: Message-ID: On Sun, Sep 6, 2015 at 8:45 PM, Theepan wrote: > What is the syntactic sugar to access single value from MAPS? #{ key := Value } = Map There are currently only injection operators available for maps in the syntax and no availability of projection operators. The closest thing you get to a projection is maps:get/2 indeed. As for the nested case: nest_lookup([], V) -> V; nest_lookup([K|Ks], M) -> nest_lookup(Ks, maps:get(K, M)). And you are ready to do nested lookups: nest_lookup([a,b,c], Map). -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From minostro@REDACTED Sun Sep 6 21:02:00 2015 From: minostro@REDACTED (Milton Inostroza Aguilera) Date: Sun, 6 Sep 2015 14:02:00 -0500 Subject: [erlang-questions] Accessing a single value from MAPS In-Reply-To: References: Message-ID: <99891FB1-21C7-48A9-8B67-B8285919B1A5@minostro.com> Hi Theepan, > On Sep 6, 2015, at 1:45 PM, Theepan wrote: > > Team, > > What is the syntactic sugar to access single value from MAPS? I know there is this "get" method, but will be good to have something like DOT notation, like we do with records. Could not find it in the documents. A = #{my_key => 1}. #{my_key := Val} = A. Val. For information about maps you can read this [1] Hope this helps, [1] http://learnyousomeerlang.com/maps#about-this-chapter > > It will be handy to access nested MAPS. > > Thanks, > Theepan > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Sun Sep 6 23:45:23 2015 From: lukas@REDACTED (Lukas Larsson) Date: Sun, 6 Sep 2015 23:45:23 +0200 Subject: [erlang-questions] =?utf-8?q?Setting_maximum_amount_of_data_alloc?= =?utf-8?q?ated_by_an_Erlang_node=2E=E2=80=8F?= In-Reply-To: References: Message-ID: Hello Akash, On Sat, Sep 5, 2015 at 1:53 AM, Akash Chowdhury wrote: > b) for maximum amount of data allocated by an Erlang node > Yes it is possible since at least 17.0 to limit the amount of memory the VM uses on OSs with a good virtual memory management interface. You need to supply two memory allocator flags in combination: +MMscs +Musac false. You can read up on what they do in the erts_alloc manual. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Mon Sep 7 03:23:24 2015 From: vasdeveloper@REDACTED (Theepan) Date: Mon, 7 Sep 2015 06:53:24 +0530 Subject: [erlang-questions] =?utf-8?q?Setting_maximum_amount_of_data_alloc?= =?utf-8?q?ated_by_an_Erlang_node=2E=E2=80=8F?= In-Reply-To: References: Message-ID: Hi Lukas, I presume that erts_alloc is a control layer on top of malloc/calloc/realloc or their variants to handle different types of allocations. The question is, if a limit is set -- of course, with very clear knowledge of what is being done -- can we expect relevant exception/error from the LIB/STDLIB/BIF functions? Theepan On Mon, Sep 7, 2015 at 3:15 AM, Lukas Larsson wrote: > Hello Akash, > > On Sat, Sep 5, 2015 at 1:53 AM, Akash Chowdhury > wrote: > >> b) for maximum amount of data allocated by an Erlang node >> > Yes it is possible since at least 17.0 to limit the amount of memory the > VM uses on OSs with a good virtual memory management interface. You need to > supply two memory allocator flags in combination: +MMscs > +Musac false. You can read up on what they do in the erts_alloc manual. > > Lukas > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Mon Sep 7 03:34:23 2015 From: vasdeveloper@REDACTED (Theepan) Date: Mon, 7 Sep 2015 07:04:23 +0530 Subject: [erlang-questions] JSON for STDLIB Message-ID: Team, Is there any reason why a JSON library is not included into Erlang/OTP releases? Since it was initially planned for telco systems, it includes ASN.1 library. It has some other transfer-format libraries as well. Since Erlang is now widely used in internet domain, I believe that inclusion of JSON library under STDLIB is essential. There are many open source variants, but I like Jiffy and Jsx. They support MAPS too. I propose to the stakeholders, to include one of them into the Erlang/OTP package. This will ensure the continuity of the library as well. I don't think JSON will die very soon, although it is loosely defined. Some times, loose definition is its strength. Theepan -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Mon Sep 7 04:36:18 2015 From: zxq9@REDACTED (zxq9) Date: Mon, 07 Sep 2015 11:36:18 +0900 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: References: Message-ID: <3887112.QzT9K7F4Nz@changa> On 2015?9?7? ??? 07:04:23 Theepan wrote: > Team, ... > I propose to the stakeholders, to include one of them into the Erlang/OTP > package. This will ensure the continuity of the library as well. > > I don't think JSON will die very soon, although it is loosely defined. Some > times, loose definition is its strength. And so it begins... From zhuo.quan@REDACTED Mon Sep 7 04:35:00 2015 From: zhuo.quan@REDACTED (=?utf-8?B?5YWo5Y2T?=) Date: Mon, 7 Sep 2015 10:35:00 +0800 Subject: [erlang-questions] =?utf-8?b?562U5aSNOiAgSlNPTiBmb3IgU1RETElC?= Message-ID: Expect that too. ???: Theepan ????: 2015?9?7? 9:34 ???: Erlang Questions Mailing List ??: [erlang-questions] JSON for STDLIB Team, Is there any reason why a JSON library is not included into Erlang/OTP releases? Since it was initially planned for telco systems, it includes ASN.1 library. It has some other transfer-format libraries as well. Since Erlang is now widely used in internet domain, I believe that inclusion of JSON library under STDLIB is essential. There are many open source variants, but I like Jiffy and Jsx. They support MAPS too. I propose to the stakeholders, to include one of them into the Erlang/OTP package. This will ensure the continuity of the library as well. I don't think JSON will die very soon, although it is loosely defined. Some times, loose definition is its strength. Theepan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ahe.sanath@REDACTED Mon Sep 7 05:52:34 2015 From: ahe.sanath@REDACTED (Sanath Prasanna) Date: Mon, 7 Sep 2015 09:22:34 +0530 Subject: [erlang-questions] Unexpected behavior of HTTPC module - tls_connection process take more memory Message-ID: Hi all, I am running HTTP client using httpc Module to send both http and https requests. normally sending arround 300 request per second without any issue. however sometimes erlang node become very slow responsive. at that time server load average is very high and using etop can identify "tls_connection" process take more memory. when restart the erlang node its become normal. as per my investigation normal time memory, processors, loadAverage is not increasing. following is the HTTP request config httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync, false}]) below is the etop output at that time procs 1134 processes 1504844 code 9309 runq 0 atom 420 ets 29692 Pid Name or Initial Func Time Reds Memory MsgQ Current Function ---------------------------------------------------------------------------------------- <5490.26428.14>tls_connection:init/ '-' 733224580768 0 gen_fsm:loop/7 <5490.26429.14>tls_connection:init/ '-' 1328524580768 0 gen_fsm:loop/7 <5490.26430.14>tls_connection:init/ '-' 528924580768 0 gen_fsm:loop/7 <5490.26431.14>tls_connection:init/ '-' 1432224580768 0 gen_fsm:loop/7 <5490.26432.14>tls_connection:init/ '-' 024580768 0 gen_fsm:loop/7 <5490.26433.14>tls_connection:init/ '-' 024580768 0 gen_fsm:loop/7 <5490.26434.14>tls_connection:init/ '-' 024580768 0 gen_fsm:loop/7 <5490.26435.14>tls_connection:init/ '-' 024580768 0 gen_fsm:loop/7 <5490.26436.14>tls_connection:init/ '-' 024580768 0 gen_fsm:loop/7 can some one help me to solve this issue? Br, A.H.E. Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Mon Sep 7 07:50:45 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 7 Sep 2015 08:50:45 +0300 Subject: [erlang-questions] =?utf-8?b?562U5aSNOiBKU09OIGZvciBTVERMSUI=?= In-Reply-To: References: Message-ID: At first: there is no determined way of mapping json to erlang terms and back. So it is better to have this library out of standard library. -------------- next part -------------- An HTML attachment was scrubbed... URL: From garazdawi@REDACTED Mon Sep 7 08:00:46 2015 From: garazdawi@REDACTED (Lukas Larsson) Date: Mon, 7 Sep 2015 08:00:46 +0200 Subject: [erlang-questions] =?utf-8?q?Setting_maximum_amount_of_data_alloc?= =?utf-8?q?ated_by_an_Erlang_node=2E=E2=80=8F?= In-Reply-To: References: Message-ID: Hello, > On 7 sep 2015, at 03:23, Theepan wrote: > > Hi Lukas, > > I presume that erts_alloc is a control layer on top of malloc/calloc/realloc or their variants to handle different types of allocations. The question is, if a limit is set -- of course, with very clear knowledge of what is being done -- can we expect relevant exception/error from the LIB/STDLIB/BIF functions? From some, but far from all. When you run out of memory it is almost always a catastrophic failure resulting in an erlang vm crash. > > Theepan > >> On Mon, Sep 7, 2015 at 3:15 AM, Lukas Larsson wrote: >> Hello Akash, >> >>> On Sat, Sep 5, 2015 at 1:53 AM, Akash Chowdhury wrote: >>> b) for maximum amount of data allocated by an Erlang node >>> >> >> Yes it is possible since at least 17.0 to limit the amount of memory the VM uses on OSs with a good virtual memory management interface. You need to supply two memory allocator flags in combination: +MMscs +Musac false. You can read up on what they do in the erts_alloc manual. >> >> Lukas >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Mon Sep 7 08:31:34 2015 From: vasdeveloper@REDACTED (Theepan) Date: Mon, 7 Sep 2015 12:01:34 +0530 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: References: Message-ID: You mean floats? On Monday, September 7, 2015, Max Lapshin wrote: > At first: there is no determined way of mapping json to erlang terms and > back. > > So it is better to have this library out of standard library. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Mon Sep 7 08:44:22 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 7 Sep 2015 09:44:22 +0300 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: References: Message-ID: floats, empty objects, strings vs binaries. How to store objects in erlang: map or proplist. If proplist, then what to do with non-unique keys? If maps, then how to maintain key order. -------------- next part -------------- An HTML attachment was scrubbed... URL: From be.dmitry@REDACTED Mon Sep 7 09:15:43 2015 From: be.dmitry@REDACTED (Dmitry Belyaev) Date: Mon, 7 Sep 2015 17:15:43 +1000 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: References: Message-ID: Something still may be represented unambigously. https://github.com/talentdeficit/jsx#modulehandle_event2 shows an example of a parser similar to SAX which is something every json library does implement in some way. With this component made a part of OTP other libraries would consume event stream and build whatever representation they want. Best wishes, -- Dmitry Belyaev On Mon, Sep 7, 2015 at 4:44 PM, Max Lapshin wrote: > floats, empty objects, strings vs binaries. How to store objects in > erlang: map or proplist. If proplist, then what to do with non-unique keys? > If maps, then how to maintain key order. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Mon Sep 7 09:38:48 2015 From: n.oxyde@REDACTED (Anthony Ramine) Date: Mon, 7 Sep 2015 09:38:48 +0200 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: References: Message-ID: <665C0638-211E-412D-B135-742F08C69A59@gmail.com> Le 7 sept. 2015 ? 09:15, Dmitry Belyaev a ?crit : > With this component made a part of OTP other libraries would consume event stream and build whatever representation they want. And the point would be? From essen@REDACTED Mon Sep 7 09:39:23 2015 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Mon, 7 Sep 2015 09:39:23 +0200 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: References: Message-ID: <55ED3F2B.1040809@ninenines.eu> On 09/07/2015 03:34 AM, Theepan wrote: > Team, > > Is there any reason why a JSON library is not included into Erlang/OTP > releases? Since it was initially planned for telco systems, it includes > ASN.1 library. It has some other transfer-format libraries as well. ASN.1 isn't just a "transfer-format", it's something OTP itself *uses*. Without the ASN.1 application, there would be no public_key application, and therefore no SSL, to name one. JSON would be just sitting there. What's the point? Another argument against it is one you cited in your email: there are already a variety of open source JSON libraries. Each has different properties and are useful to different people. Why would you include one and basically doom the others? Which one would you bundle? Will that choice make sense to everyone? Finally, I'm not sure what's the point? If you are using Erlang.mk, to use jsx, you need to add this single line to your Makefile: DEPS = jsx If jsx was part of OTP, you would need to add this single line to your Makefile: OTP_DEPS = jsx The advantage of bundling it in OTP are not obvious, to say the least. In general I believe OTP should reduce the number of libraries it comes with, not increase it. It's easy to find a library fitting your needs nowadays. Erlang.mk for example has 470 packages and you can search them in one easy command: make search q=json Cheers, -- Lo?c Hoguin http://ninenines.eu Author of The Erlanger Playbook, A book about software development using Erlang From ahe.sanath@REDACTED Mon Sep 7 12:50:31 2015 From: ahe.sanath@REDACTED (Sanath Prasanna) Date: Mon, 7 Sep 2015 16:20:31 +0530 Subject: [erlang-questions] Unexpected behavior of HTTPC module - tls_connection process take more memory In-Reply-To: References: Message-ID: Hi All, Any suggestion on below problem? Br, Robert On Mon, Sep 7, 2015 at 9:22 AM, Sanath Prasanna wrote: > Hi all, > > I am running HTTP client using httpc Module to send both http and https > requests. normally sending arround 300 request per second without any > issue. however sometimes erlang node become very slow responsive. at that > time server load average is very high and using etop can identify > "tls_connection" process take more memory. when restart the erlang node its > become normal. as per my investigation normal time memory, processors, > loadAverage is not increasing. following is the HTTP request config > > > > httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync, false}]) > > > > below is the etop output at that time > > > > procs 1134 processes 1504844 > code 9309 > > runq 0 atom 420 > ets 29692 > > > > Pid Name or Initial Func Time Reds Memory MsgQ > Current Function > > > ---------------------------------------------------------------------------------------- > > <5490.26428.14>tls_connection:init/ '-' 733224580768 0 > gen_fsm:loop/7 > > <5490.26429.14>tls_connection:init/ '-' 1328524580768 0 > gen_fsm:loop/7 > > <5490.26430.14>tls_connection:init/ '-' 528924580768 0 > gen_fsm:loop/7 > > <5490.26431.14>tls_connection:init/ '-' 1432224580768 0 > gen_fsm:loop/7 > > <5490.26432.14>tls_connection:init/ '-' 024580768 0 > gen_fsm:loop/7 > > <5490.26433.14>tls_connection:init/ '-' 024580768 0 > gen_fsm:loop/7 > > <5490.26434.14>tls_connection:init/ '-' 024580768 0 > gen_fsm:loop/7 > > <5490.26435.14>tls_connection:init/ '-' 024580768 0 > gen_fsm:loop/7 > > <5490.26436.14>tls_connection:init/ '-' 024580768 0 > gen_fsm:loop/7 > > > can some one help me to solve this issue? > > Br, > > A.H.E. Robert > -------------- next part -------------- An HTML attachment was scrubbed... URL: From swvist@REDACTED Mon Sep 7 15:14:34 2015 From: swvist@REDACTED (Vipin Nair) Date: Mon, 7 Sep 2015 18:44:34 +0530 Subject: [erlang-questions] Closing a ssh channel from client side with killing the connection Message-ID: Hello, I am calling ssh_connection:exec/4 from a process implementing the ssh_channel behaviour. I want to ensure that all commands exit within some time limit. Say, someone runs a tail -f /var/log/syslog, I want to ensure that this is not run beyond X seconds. My idea was to close the channel after the timeout, but I am not sure how to do this. There may be multiple channel established over the same connection so I cannot kill the connection. How should I do this? A quick glance at the RFC[1] shows that either party can close the channel. So this perhaps could be added as a new public api in ssh_channel module. [1]: https://www.ietf.org/rfc/rfc4254.txt -- Regards, Vipin From eric.pailleau@REDACTED Mon Sep 7 14:23:18 2015 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Mon, 07 Sep 2015 14:23:18 +0200 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: <55ED3F2B.1040809@ninenines.eu> Message-ID: Hi, I agree but there is some lack however, zip/gzip exists in Otp, but not bzip nor lzma2. As it is ports, it should be done at core level while json can be external libraries. Those compressions formats are now very common in particular lzma2. Really missing. Regards Le?7 sept. 2015 09:39, Lo?c Hoguin a ?crit?: > > On 09/07/2015 03:34 AM, Theepan wrote: > > Team, > > > > Is there any reason why a JSON library is not included into Erlang/OTP > > releases? Since it was initially planned for telco systems, it includes > > ASN.1 library. It has some other transfer-format libraries as well. > > ASN.1 isn't just a "transfer-format", it's something OTP itself *uses*. > Without the ASN.1 application, there would be no public_key application, > and therefore no SSL, to name one. > > JSON would be just sitting there. What's the point? > > Another argument against it is one you cited in your email: there are > already a variety of open source JSON libraries. Each has different > properties and are useful to different people. Why would you include one > and basically doom the others? Which one would you bundle? Will that > choice make sense to everyone? > > Finally, I'm not sure what's the point? If you are using Erlang.mk, to > use jsx, you need to add this single line to your Makefile: > > DEPS = jsx > > If jsx was part of OTP, you would need to add this single line to your > Makefile: > > OTP_DEPS = jsx > > The advantage of bundling it in OTP are not obvious, to say the least. > > In general I believe OTP should reduce the number of libraries it comes > with, not increase it. It's easy to find a library fitting your needs > nowadays. Erlang.mk for example has 470 packages and you can search them > in one easy command: make search q=json > > Cheers, > > -- > Lo?c Hoguin > http://ninenines.eu > Author of The Erlanger Playbook, > A book about software development using Erlang > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From seancribbs@REDACTED Mon Sep 7 18:10:25 2015 From: seancribbs@REDACTED (Sean Cribbs) Date: Mon, 7 Sep 2015 11:10:25 -0500 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: References: <55ED3F2B.1040809@ninenines.eu> Message-ID: zip is used in reltool for making compressed code archives. gzip is used in a number of places, including term_to_binary with compression. I think the previous posters' points still stand. I think the proper path for JSON into the standard library is the way many other language communities have done it. Let competing implementations start as a third-party libraries, and then if (and only if) there is enough momentum and pressure around a single candidate take steps to include it. The main challenge I see for any of this happening with Erlang/OTP is that getting patches accepted is difficult enough already, and every one of the existing JSON libraries uses a slightly different style. It's probably best as third party library. On Mon, Sep 7, 2015 at 7:23 AM, ?ric Pailleau wrote: > Hi, > I agree but there is some lack however, zip/gzip exists in Otp, but not > bzip nor lzma2. > As it is ports, it should be done at core level while json can be external > libraries. > Those compressions formats are now very common in particular lzma2. Really > missing. > Regards > > Le 7 sept. 2015 09:39, Lo?c Hoguin a ?crit : > > > > On 09/07/2015 03:34 AM, Theepan wrote: > > > Team, > > > > > > Is there any reason why a JSON library is not included into Erlang/OTP > > > releases? Since it was initially planned for telco systems, it includes > > > ASN.1 library. It has some other transfer-format libraries as well. > > > > ASN.1 isn't just a "transfer-format", it's something OTP itself *uses*. > > Without the ASN.1 application, there would be no public_key application, > > and therefore no SSL, to name one. > > > > JSON would be just sitting there. What's the point? > > > > Another argument against it is one you cited in your email: there are > > already a variety of open source JSON libraries. Each has different > > properties and are useful to different people. Why would you include one > > and basically doom the others? Which one would you bundle? Will that > > choice make sense to everyone? > > > > Finally, I'm not sure what's the point? If you are using Erlang.mk, to > > use jsx, you need to add this single line to your Makefile: > > > > DEPS = jsx > > > > If jsx was part of OTP, you would need to add this single line to your > > Makefile: > > > > OTP_DEPS = jsx > > > > The advantage of bundling it in OTP are not obvious, to say the least. > > > > In general I believe OTP should reduce the number of libraries it comes > > with, not increase it. It's easy to find a library fitting your needs > > nowadays. Erlang.mk for example has 470 packages and you can search them > > in one easy command: make search q=json > > > > Cheers, > > > > -- > > Lo?c Hoguin > > http://ninenines.eu > > Author of The Erlanger Playbook, > > A book about software development using Erlang > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Mon Sep 7 18:17:59 2015 From: vasdeveloper@REDACTED (Theepan) Date: Mon, 7 Sep 2015 21:47:59 +0530 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: <55ED3F2B.1040809@ninenines.eu> References: <55ED3F2B.1040809@ninenines.eu> Message-ID: Hi, ASN.1 isn't just a "transfer-format", it's something OTP itself *uses*. Without the ASN.1 application, there would be no public_key application, and therefore no SSL, to name one. JSON would be just sitting there. What's the point? If I argue to your point -- not all the libraries in the STDLIB are used internally. Moreover, in a software development platform, libraries are not just meant to be used internally. Another argument against it is one you cited in your email: there are already a variety of open source JSON libraries. Each has different properties and are useful to different people. Why would you include one and basically doom the others? Which one would you bundle? Will that choice make sense to everyone? The library that caters to the language necessities, that imposes better coding practice, that performs better, that is hardware efficient should be included, may be with modifications. If the libraries are out in the wild, the developer will spend their valuable time in selecting the library , than concentrating on solving the business problem. Finally, I'm not sure what's the point? If you are using Erlang.mk, to use jsx, you need to add this single line to your Makefile: DEPS = jsx If jsx was part of OTP, you would need to add this single line to your Makefile: OTP_DEPS = jsx How you include it into the build is not the issue. It does not have to be through Erlang.mk. There are couple of other package management systems for Erlang. Also, after a direct install, you can simply refer it in the release file in one line. The issue is not of technical nature, but the bigger picture. Continuity/Faster sync with Erlang evolutions/Simplicity/Regulation/Completeness. etc The advantage of bundling it in OTP are not obvious, to say the least. In general I believe OTP should reduce the number of libraries it comes with, not increase it. It's easy to find a library fitting your needs nowadays. Erlang.mk for example has 470 packages and you can search them in one easy command: make search q=json As a software development, developers would not expect to have fragmentation of libraries. As I already mentioned, Erlang.mk is not the only Erlang package management out there, but it is not the issue here. JSON library is a good candidate to be including into the STDLIB, with of course bounds on loose definitions like JavaScript engines do. On Mon, Sep 7, 2015 at 1:09 PM, Lo?c Hoguin wrote: > On 09/07/2015 03:34 AM, Theepan wrote: > >> Team, >> >> Is there any reason why a JSON library is not included into Erlang/OTP >> releases? Since it was initially planned for telco systems, it includes >> ASN.1 library. It has some other transfer-format libraries as well. >> > > ASN.1 isn't just a "transfer-format", it's something OTP itself *uses*. > Without the ASN.1 application, there would be no public_key application, > and therefore no SSL, to name one. > > JSON would be just sitting there. What's the point? > > Another argument against it is one you cited in your email: there are > already a variety of open source JSON libraries. Each has different > properties and are useful to different people. Why would you include one > and basically doom the others? Which one would you bundle? Will that choice > make sense to everyone? > > Finally, I'm not sure what's the point? If you are using Erlang.mk, to use > jsx, you need to add this single line to your Makefile: > > DEPS = jsx > > If jsx was part of OTP, you would need to add this single line to your > Makefile: > > OTP_DEPS = jsx > > The advantage of bundling it in OTP are not obvious, to say the least. > > In general I believe OTP should reduce the number of libraries it comes > with, not increase it. It's easy to find a library fitting your needs > nowadays. Erlang.mk for example has 470 packages and you can search them in > one easy command: make search q=json > > Cheers, > > -- > Lo?c Hoguin > http://ninenines.eu > Author of The Erlanger Playbook, > A book about software development using Erlang > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Mon Sep 7 18:27:35 2015 From: vasdeveloper@REDACTED (Theepan) Date: Mon, 7 Sep 2015 21:57:35 +0530 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: References: Message-ID: Other then floats, I don't see issues in handling the others. Key order is not defined in JSON, and JavaScript engines are handling it in different ways. When you have the keys, order might not be important. If the order is important, define a parameter for each key that is sortable. If the keys are not unique, the first key will be selected - for example OR define one key and put the values into an array. Some expectations need to be set. On Mon, Sep 7, 2015 at 12:14 PM, Max Lapshin wrote: > floats, empty objects, strings vs binaries. How to store objects in > erlang: map or proplist. If proplist, then what to do with non-unique keys? > If maps, then how to maintain key order. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Tue Sep 8 01:50:23 2015 From: zxq9@REDACTED (zxq9) Date: Tue, 08 Sep 2015 08:50:23 +0900 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: References: Message-ID: <2643468.EpdLaSXBee@changa> On 2015?9?7? ??? 21:57:35 Theepan wrote: > Other then floats, I don't see issues in handling the others. > > Key order is not defined in JSON, and JavaScript engines are handling it in > different ways. When you have the keys, order might not be important. If > the order is important, define a parameter for each key that is sortable. > > If the keys are not unique, the first key will be selected - for example OR > define one key and put the values into an array. > > Some expectations need to be set. That is the problem. Javascript/ECMA standards are a big fat mess. Erlang standards are, for the most part, not. We have ways of handling JSON, but each method differs somewhat. There is no clear optimum for JSON against which a goal could be set, no definite "this is the right way to do JSON that is technically sound, reasonably easy to use, and efficient". Defining such a goal would be the first step -- because once Erlang includes something like that as a core library we are *all* stuck with it (and supporting that particular API) forever and ever and ever, amen. Goal definition in the face of sloppy standards is already a non-trivial task. In the presence of decent (but very different) third party libs that already exist it is hard to see what the cost in terms of effort would actually buy us, especially considering that... There is not a great clamor for this. If there is, it certainly hasn't hit the mailing list yet. The problem is a lot deeper than "floats" (and anyway, js/JSON lacks integers; everything is already a float -- which is another ball of poo we don't need to deal with inside a core library). It also happens to lack atoms. And ordered dicts. And... anyway, there really isn't a clear mapping. In the face of this ambiguity it is a good thing that we have several third party libs that cater to different interpretations/mapping *and* exhibit different performance characteristics instead of a central canonical library that has to make enough compromises to get included that it just doesn't quite suit anyone's actual use case, and is therefore left lonely in the core libs, unused by most (the way various XML libs in several language core libs have been, for example) but still a derelict maintenance burden. I think the argument for including bzip2 and lzma2 is much stronger, if for no other reason than it is an unambiguous goal with a very clear and obvious implementation path. But even that is a tough sell. -Craig From pierrefenoll@REDACTED Tue Sep 8 04:06:19 2015 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Mon, 7 Sep 2015 19:06:19 -0700 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: <2643468.EpdLaSXBee@changa> References: <2643468.EpdLaSXBee@changa> Message-ID: But if you know about the issues of translating JS JSON into Erlang JSON, you still need to send JSON data through your API. I guess that we then have to agree on which trade offs we have to pick from all the different JSON libraries. But for new comers or people that are fine with these trade offs, that can be the easiest and most useful choice. And I think that the default choice would be the one need. (Ok now we need to pick the most probable default choice then) > On 07 Sep 2015, at 16:50, zxq9 wrote: > >> On 2015?9?7? ??? 21:57:35 Theepan wrote: >> Other then floats, I don't see issues in handling the others. >> >> Key order is not defined in JSON, and JavaScript engines are handling it in >> different ways. When you have the keys, order might not be important. If >> the order is important, define a parameter for each key that is sortable. >> >> If the keys are not unique, the first key will be selected - for example OR >> define one key and put the values into an array. >> >> Some expectations need to be set. > > That is the problem. Javascript/ECMA standards are a big fat mess. Erlang standards are, for the most part, not. We have ways of handling JSON, but each method differs somewhat. There is no clear optimum for JSON against which a goal could be set, no definite "this is the right way to do JSON that is technically sound, reasonably easy to use, and efficient". > > Defining such a goal would be the first step -- because once Erlang includes something like that as a core library we are *all* stuck with it (and supporting that particular API) forever and ever and ever, amen. Goal definition in the face of sloppy standards is already a non-trivial task. In the presence of decent (but very different) third party libs that already exist it is hard to see what the cost in terms of effort would actually buy us, especially considering that... > > There is not a great clamor for this. If there is, it certainly hasn't hit the mailing list yet. > > The problem is a lot deeper than "floats" (and anyway, js/JSON lacks integers; everything is already a float -- which is another ball of poo we don't need to deal with inside a core library). It also happens to lack atoms. And ordered dicts. And... anyway, there really isn't a clear mapping. > > In the face of this ambiguity it is a good thing that we have several third party libs that cater to different interpretations/mapping *and* exhibit different performance characteristics instead of a central canonical library that has to make enough compromises to get included that it just doesn't quite suit anyone's actual use case, and is therefore left lonely in the core libs, unused by most (the way various XML libs in several language core libs have been, for example) but still a derelict maintenance burden. > > I think the argument for including bzip2 and lzma2 is much stronger, if for no other reason than it is an unambiguous goal with a very clear and obvious implementation path. But even that is a tough sell. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ok@REDACTED Tue Sep 8 05:37:35 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 8 Sep 2015 15:37:35 +1200 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: <2643468.EpdLaSXBee@changa> References: <2643468.EpdLaSXBee@changa> Message-ID: <6E7C7EAF-46E3-401B-86E8-2F91E549BD8B@cs.otago.ac.nz> How quickly they forget. There is an EEP for JSON in Erlang, but even back then, there were already several libraries with incompatible answers to various questions. Since then JSON has changed a bit (for example, according to one "official" JSON "standard", having a number or string at top level is OK), and Erlang has changed more (we now have maps). Arguably, standardising on a JSON representation too early would have *harmed* Erlang by locking us into something that doesn't fit as well as maps do. I don't think JSON parsing has to be in stdlib. I do think it would be nice to have *one* place to look for Erlang packages, even if it's just a version-stamped-link farm. I don't care if there are 20 JSON libraries as long as there is one place to look for them. (Oh, and writing a JSON parser in Erlang really is *not* hard. The whole *point* of JSON is that it's supposed to be easy to process. Which is why there are so many libraries...) From antoine.koener@REDACTED Tue Sep 8 08:49:27 2015 From: antoine.koener@REDACTED (Antoine Koener) Date: Tue, 8 Sep 2015 08:49:27 +0200 Subject: [erlang-questions] Unexpected behavior of HTTPC module - tls_connection process take more memory In-Reply-To: References: Message-ID: <69026273-BE09-4B65-9759-5E39632F737E@gmail.com> Hi, This problem reminds me of something related to SSL connection cache. I'm sure that on this mailing list this have been discussed many times before. If you search this mailing list archives you may find more information. > Le 7 sept. 2015 ? 12:50, Sanath Prasanna a ?crit : > > Hi All, > Any suggestion on below problem? > Br, > Robert > > On Mon, Sep 7, 2015 at 9:22 AM, Sanath Prasanna > wrote: > Hi all, > > I am running HTTP client using httpc Module to send both http and https requests. normally sending arround 300 request per second without any issue. however sometimes erlang node become very slow responsive. at that time server load average is very high and using etop can identify "tls_connection" process take more memory. when restart the erlang node its become normal. as per my investigation normal time memory, processors, loadAverage is not increasing. following is the HTTP request config > > > httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync, false}]) > > > below is the etop output at that time > > > procs 1134 processes 1504844 code 9309 > > runq 0 atom 420 ets 29692 > > > Pid Name or Initial Func Time Reds Memory MsgQ Current Function > > ---------------------------------------------------------------------------------------- > > <5490.26428.14>tls_connection:init/ '-' 733224580768 0 gen_fsm:loop/7 > > <5490.26429.14>tls_connection:init/ '-' 1328524580768 0 gen_fsm:loop/7 > > <5490.26430.14>tls_connection:init/ '-' 528924580768 0 gen_fsm:loop/7 > > <5490.26431.14>tls_connection:init/ '-' 1432224580768 0 gen_fsm:loop/7 > > <5490.26432.14>tls_connection:init/ '-' 024580768 0 gen_fsm:loop/7 > > <5490.26433.14>tls_connection:init/ '-' 024580768 0 gen_fsm:loop/7 > > <5490.26434.14>tls_connection:init/ '-' 024580768 0 gen_fsm:loop/7 > > <5490.26435.14>tls_connection:init/ '-' 024580768 0 gen_fsm:loop/7 > > <5490.26436.14>tls_connection:init/ '-' 024580768 0 gen_fsm:loop/7 > > > > can some one help me to solve this issue? > > Br, > > A.H.E. Robert > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Tue Sep 8 10:01:57 2015 From: ingela.andin@REDACTED (Ingela Andin) Date: Tue, 8 Sep 2015 10:01:57 +0200 Subject: [erlang-questions] Unexpected behavior of HTTPC module - tls_connection process take more memory In-Reply-To: References: Message-ID: Hi! It could be an bug that in the ssl application that I just fixed. The default session cache was violating the API, and this in turn made the mechanism for not registering a lot of equivalent sessions in the client fail. Here is the patch: diff --git a/lib/ssl/src/ssl_session.erl b/lib/ssl/src/ssl_session.erl index 1770faf..0d6cc93 100644 --- a/lib/ssl/src/ssl_session.erl +++ b/lib/ssl/src/ssl_session.erl @@ -100,14 +100,14 @@ select_session([], _, _) -> no_session; select_session(Sessions, #ssl_options{ciphers = Ciphers}, OwnCert) -> IsNotResumable = - fun([_Id, Session]) -> + fun(Session) -> not (resumable(Session#session.is_resumable) andalso lists:member(Session#session.cipher_suite, Ciphers) andalso (OwnCert == Session#session.own_certificate)) end, case lists:dropwhile(IsNotResumable, Sessions) of [] -> no_session; - [[Id, _]|_] -> Id + [Session | _] -> Session#session.session_id end. is_resumable(_, _, #ssl_options{reuse_sessions = false}, _, _, _, _) -> diff --git a/lib/ssl/src/ssl_session_cache.erl b/lib/ssl/src/ssl_session_cach e.erl index 11ed310..cfc48cd 100644 --- a/lib/ssl/src/ssl_session_cache.erl +++ b/lib/ssl/src/ssl_session_cache.erl @@ -83,7 +83,7 @@ foldl(Fun, Acc0, Cache) -> %%-------------------------------------------------------------------- select_session(Cache, PartialKey) -> ets:select(Cache, - [{{{PartialKey,'$1'}, '$2'},[],['$$']}]). + [{{{PartialKey,'_'}, '$1'},[],['$1']}]). %%-------------------------------------------------------------------- %%% Internal functions Regards Ingela Erlang/OTP team - Ericsson AB 2015-09-07 5:52 GMT+02:00 Sanath Prasanna : > Hi all, > > I am running HTTP client using httpc Module to send both http and https > requests. normally sending arround 300 request per second without any > issue. however sometimes erlang node become very slow responsive. at that > time server load average is very high and using etop can identify > "tls_connection" process take more memory. when restart the erlang node its > become normal. as per my investigation normal time memory, processors, > loadAverage is not increasing. following is the HTTP request config > > > > httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync, false}]) > > > > below is the etop output at that time > > > > procs 1134 processes 1504844 > code 9309 > > runq 0 atom 420 > ets 29692 > > > > Pid Name or Initial Func Time Reds Memory MsgQ > Current Function > > > ---------------------------------------------------------------------------------------- > > <5490.26428.14>tls_connection:init/ '-' 733224580768 0 > gen_fsm:loop/7 > > <5490.26429.14>tls_connection:init/ '-' 1328524580768 0 > gen_fsm:loop/7 > > <5490.26430.14>tls_connection:init/ '-' 528924580768 0 > gen_fsm:loop/7 > > <5490.26431.14>tls_connection:init/ '-' 1432224580768 0 > gen_fsm:loop/7 > > <5490.26432.14>tls_connection:init/ '-' 024580768 0 > gen_fsm:loop/7 > > <5490.26433.14>tls_connection:init/ '-' 024580768 0 > gen_fsm:loop/7 > > <5490.26434.14>tls_connection:init/ '-' 024580768 0 > gen_fsm:loop/7 > > <5490.26435.14>tls_connection:init/ '-' 024580768 0 > gen_fsm:loop/7 > > <5490.26436.14>tls_connection:init/ '-' 024580768 0 > gen_fsm:loop/7 > > > can some one help me to solve this issue? > > Br, > > A.H.E. Robert > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eranga.erl@REDACTED Tue Sep 8 10:06:22 2015 From: eranga.erl@REDACTED (Eranga Udesh) Date: Tue, 8 Sep 2015 13:36:22 +0530 Subject: [erlang-questions] Unexpected behavior of HTTPC module - tls_connection process take more memory In-Reply-To: <69026273-BE09-4B65-9759-5E39632F737E@gmail.com> References: <69026273-BE09-4B65-9759-5E39632F737E@gmail.com> Message-ID: Hi Antoine, It doesn't seem to be related to the SSL Session Cache issue. As seen their ETS size is only 29692. Regards, - Eranga On Tue, Sep 8, 2015 at 12:19 PM, Antoine Koener wrote: > Hi, > > This problem reminds me of something related to SSL connection cache. > I'm sure that on this mailing list this have been discussed many times > before. > If you search this mailing list archives you may find more information. > > Le 7 sept. 2015 ? 12:50, Sanath Prasanna a ?crit : > > Hi All, > Any suggestion on below problem? > Br, > Robert > > On Mon, Sep 7, 2015 at 9:22 AM, Sanath Prasanna > wrote: > >> Hi all, >> >> I am running HTTP client using httpc Module to send both http and https >> requests. normally sending arround 300 request per second without any >> issue. however sometimes erlang node become very slow responsive. at that >> time server load average is very high and using etop can identify >> "tls_connection" process take more memory. when restart the erlang node its >> become normal. as per my investigation normal time memory, processors, >> loadAverage is not increasing. following is the HTTP request config >> >> >> httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync, false}]) >> >> >> below is the etop output at that time >> >> >> procs 1134 processes 1504844 >> code 9309 >> >> runq 0 atom 420 >> ets 29692 >> >> >> Pid Name or Initial Func Time Reds Memory MsgQ >> Current Function >> >> >> ---------------------------------------------------------------------------------------- >> >> <5490.26428.14>tls_connection:init/ '-' 733224580768 0 >> gen_fsm:loop/7 >> >> <5490.26429.14>tls_connection:init/ '-' 1328524580768 0 >> gen_fsm:loop/7 >> >> <5490.26430.14>tls_connection:init/ '-' 528924580768 0 >> gen_fsm:loop/7 >> >> <5490.26431.14>tls_connection:init/ '-' 1432224580768 0 >> gen_fsm:loop/7 >> >> <5490.26432.14>tls_connection:init/ '-' 024580768 0 >> gen_fsm:loop/7 >> >> <5490.26433.14>tls_connection:init/ '-' 024580768 0 >> gen_fsm:loop/7 >> >> <5490.26434.14>tls_connection:init/ '-' 024580768 0 >> gen_fsm:loop/7 >> >> <5490.26435.14>tls_connection:init/ '-' 024580768 0 >> gen_fsm:loop/7 >> >> <5490.26436.14>tls_connection:init/ '-' 024580768 0 >> gen_fsm:loop/7 >> >> >> can some one help me to solve this issue? >> >> Br, >> >> A.H.E. Robert >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ahe.sanath@REDACTED Tue Sep 8 10:22:36 2015 From: ahe.sanath@REDACTED (Sanath Prasanna) Date: Tue, 8 Sep 2015 13:52:36 +0530 Subject: [erlang-questions] Unexpected behavior of HTTPC module - tls_connection process take more memory In-Reply-To: References: Message-ID: Hi Ingela. Tx a lot for your help & patch related to that.I'll inform you the result after applying & testing patch. Br, Robert On Tue, Sep 8, 2015 at 1:31 PM, Ingela Andin wrote: > Hi! > > It could be an bug that in the ssl application that I just fixed. The > default session cache > was violating the API, and this in turn made the mechanism for not > registering a lot of equivalent > sessions in the client fail. > > Here is the patch: > > > diff --git a/lib/ssl/src/ssl_session.erl b/lib/ssl/src/ssl_session.erl > index 1770faf..0d6cc93 100644 > --- a/lib/ssl/src/ssl_session.erl > +++ b/lib/ssl/src/ssl_session.erl > @@ -100,14 +100,14 @@ select_session([], _, _) -> > no_session; > select_session(Sessions, #ssl_options{ciphers = Ciphers}, OwnCert) -> > IsNotResumable = > - fun([_Id, Session]) -> > + fun(Session) -> > not (resumable(Session#session.is_resumable) andalso > lists:member(Session#session.cipher_suite, Ciphers) > andalso (OwnCert == Session#session.own_certificate)) > end, > case lists:dropwhile(IsNotResumable, Sessions) of > [] -> no_session; > - [[Id, _]|_] -> Id > + [Session | _] -> Session#session.session_id > end. > > is_resumable(_, _, #ssl_options{reuse_sessions = false}, _, _, _, _) -> > diff --git a/lib/ssl/src/ssl_session_cache.erl > b/lib/ssl/src/ssl_session_cach > e.erl > index 11ed310..cfc48cd 100644 > --- a/lib/ssl/src/ssl_session_cache.erl > +++ b/lib/ssl/src/ssl_session_cache.erl > @@ -83,7 +83,7 @@ foldl(Fun, Acc0, Cache) -> > %%-------------------------------------------------------------------- > select_session(Cache, PartialKey) -> > ets:select(Cache, > - [{{{PartialKey,'$1'}, '$2'},[],['$$']}]). > + [{{{PartialKey,'_'}, '$1'},[],['$1']}]). > > %%-------------------------------------------------------------------- > %%% Internal functions > > > Regards Ingela Erlang/OTP team - Ericsson AB > > > 2015-09-07 5:52 GMT+02:00 Sanath Prasanna : > >> Hi all, >> >> I am running HTTP client using httpc Module to send both http and https >> requests. normally sending arround 300 request per second without any >> issue. however sometimes erlang node become very slow responsive. at that >> time server load average is very high and using etop can identify >> "tls_connection" process take more memory. when restart the erlang node its >> become normal. as per my investigation normal time memory, processors, >> loadAverage is not increasing. following is the HTTP request config >> >> >> >> httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync, false}]) >> >> >> >> below is the etop output at that time >> >> >> >> procs 1134 processes 1504844 >> code 9309 >> >> runq 0 atom 420 >> ets 29692 >> >> >> >> Pid Name or Initial Func Time Reds Memory MsgQ >> Current Function >> >> >> ---------------------------------------------------------------------------------------- >> >> <5490.26428.14>tls_connection:init/ '-' 733224580768 0 >> gen_fsm:loop/7 >> >> <5490.26429.14>tls_connection:init/ '-' 1328524580768 0 >> gen_fsm:loop/7 >> >> <5490.26430.14>tls_connection:init/ '-' 528924580768 0 >> gen_fsm:loop/7 >> >> <5490.26431.14>tls_connection:init/ '-' 1432224580768 0 >> gen_fsm:loop/7 >> >> <5490.26432.14>tls_connection:init/ '-' 024580768 0 >> gen_fsm:loop/7 >> >> <5490.26433.14>tls_connection:init/ '-' 024580768 0 >> gen_fsm:loop/7 >> >> <5490.26434.14>tls_connection:init/ '-' 024580768 0 >> gen_fsm:loop/7 >> >> <5490.26435.14>tls_connection:init/ '-' 024580768 0 >> gen_fsm:loop/7 >> >> <5490.26436.14>tls_connection:init/ '-' 024580768 0 >> gen_fsm:loop/7 >> >> >> can some one help me to solve this issue? >> >> Br, >> >> A.H.E. Robert >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From spawn.think@REDACTED Tue Sep 8 10:33:38 2015 From: spawn.think@REDACTED (Ahmed Omar) Date: Tue, 8 Sep 2015 10:33:38 +0200 Subject: [erlang-questions] segfault erts-5.10.4 (R16B03-1) Message-ID: Hi, We have been experiencing a segfault on our servers running a custom version of Ejabberd. We managed to get a core file from the last crash This is what we see running gdb on it: ###### Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Core was generated by `/var/lib/ejabberd/erts-5.10.4/bin/beam.smp -K true -A 128 -P 2500000 -Q 500000'. Program terminated with signal 11, Segmentation fault. #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, block=0x128) at beam/erl_goodfit_alloc.c:439 439 beam/erl_goodfit_alloc.c: No such file or directory. in beam/erl_goodfit_alloc.c ###### If we run bt full in gdb we get: ###### (gdb) bt full #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, block=0x128) at beam/erl_goodfit_alloc.c:439 gfallctr = 0x15e32c0 blk = 0x128 sz = 0 i = #1 0x00000000015e32c0 in ?? () No symbol table info available. #2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, size=Unhandled dwarf expression opcode 0xf3 ) at beam/erl_alloc_util.c:2370 crr = 0x128 new_p = old_blk_sz = 287 blk = 0x117 new_blk = cand_blk = cand_blk_sz = blk_sz = 3748409 nxt_blk = 0x236 nxt_blk_sz = 22950592 is_last_blk = 296 get_blk_sz = 140602277246336 #3 0x0000000000000000 in ?? () No symbol table info available. ####### Is there a way to get more information? maybe which driver made the realloc call? Best Regards, - Ahmed Omar http://about.me/spawn.think/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From garazdawi@REDACTED Tue Sep 8 10:51:41 2015 From: garazdawi@REDACTED (Lukas Larsson) Date: Tue, 8 Sep 2015 10:51:41 +0200 Subject: [erlang-questions] segfault erts-5.10.4 (R16B03-1) In-Reply-To: References: Message-ID: Hello, On Tue, Sep 8, 2015 at 10:33 AM, Ahmed Omar wrote: > Hi, > We have been experiencing a segfault on our servers running a custom > version of Ejabberd. We managed to get a core file from the last crash > This is what we see running gdb on it: > ###### > Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols > found)...done. > Loaded symbols for /lib64/ld-linux-x86-64.so.2 > Core was generated by `/var/lib/ejabberd/erts-5.10.4/bin/beam.smp -K true > -A 128 -P 2500000 -Q 500000'. > Program terminated with signal 11, Segmentation fault. > #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, block=0x128) > at beam/erl_goodfit_alloc.c:439 > 439 beam/erl_goodfit_alloc.c: No such file or directory. > in beam/erl_goodfit_alloc.c > ###### > > If we run bt full in gdb we get: > ###### > (gdb) bt full > #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, block=0x128) > at beam/erl_goodfit_alloc.c:439 > gfallctr = 0x15e32c0 > blk = 0x128 > sz = 0 > i = > #1 0x00000000015e32c0 in ?? () > No symbol table info available. > #2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, > size=Unhandled dwarf expression opcode 0xf3 > ) at beam/erl_alloc_util.c:2370 > crr = 0x128 > new_p = > old_blk_sz = 287 > blk = 0x117 > new_blk = > cand_blk = > cand_blk_sz = > blk_sz = 3748409 > nxt_blk = 0x236 > nxt_blk_sz = 22950592 > is_last_blk = 296 > get_blk_sz = 140602277246336 > #3 0x0000000000000000 in ?? () > No symbol table info available. > ####### > > Is there a way to get more information? maybe which driver made the > realloc call? > Something is wrong/missing from this stacktrace. The gdb that you are using does not seem to understand the dwarf2 extension (at least that's what I guess after googling "Unhandled dwarf expression opcode 0xf3"), and can only find two of the frames. Try to install a later version of gdb and then do a bt full. > > Best Regards, > - Ahmed Omar > http://about.me/spawn.think/ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ameretat.reith@REDACTED Tue Sep 8 10:04:58 2015 From: ameretat.reith@REDACTED (Ameretat Reith) Date: Tue, 8 Sep 2015 12:34:58 +0430 Subject: [erlang-questions] Cannot add supervisor to application's supervisor in runtime Message-ID: <20150908123458.66553359@gmail.com> Hi, I'm trying to add new supervisor `ch_sup` and It's worker `ch` to an application supervisor `app_sup` with this appup: ``` {"0.1.3", [{"0.1.2", [{add_module, ch}, {add_module, ch_sup}, {update, app_sup, supervisor}, {apply, {supervisor, restart_child, [app_sup, ch_sup]}} ] }], [{"0.1.2", [{apply, {supervisor, terminate_child, [app_sup, ch_sup]}}, {apply, {supervisor, delete_child, [app_sup, ch_sup]}}, {update, app_sup, supervisor}, {delete_module, ch_sup}, {delete_module, ch} ] }] }. ``` And this relup generated by systools:make_relup: ``` {"0.0.3", [{"0.0.2",[], [{load_object_code,{app,"0.1.3", [ch,ch_sup,app_sup]}}, point_of_no_return, {load,{ch,brutal_purge,brutal_purge}}, {load,{ch_sup,brutal_purge,brutal_purge}}, {suspend,[app_sup]}, {load,{app_sup,brutal_purge,brutal_purge}}, {code_change,up,[{app_sup,[]}]}, {resume,[app_sup]}, {apply,{supervisor,restart_child,[app_sup,ch_sup]}}]}], [{"0.0.2",[], [{load_object_code,{app,"0.1.2",[app_sup]}}, point_of_no_return, {apply,{supervisor,terminate_child,[app_sup,ch_sup]}}, {apply,{supervisor,delete_child,[app_sup,ch_sup]}}, {suspend,[app_sup]}, {load,{app_sup,brutal_purge,brutal_purge}}, {code_change,down,[{app_sup,[]}]}, {resume,[app_sup]}, {remove,{ch_sup,brutal_purge,brutal_purge}}, {purge,[ch_sup]}, {remove,{ch,brutal_purge,brutal_purge}}, {purge,[ch]}]}]}. ``` New release installs But app_sup's child spec wont change, neither ch_sup start until I terminate app_sup and start It again with `supervisor:restart_child(real_app_sup, app_sup)`. app is an included application which is put in real_app's supervision tree. I don't think app as an included application, should be regarded special here, since I use It just as a supervisor. But seems I'm wrong.. So, What is the problem? Can I achieve changing app_sup's main supervisor without restarting It's process? Thanks From jesper.louis.andersen@REDACTED Tue Sep 8 11:42:00 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 8 Sep 2015 11:42:00 +0200 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: References: Message-ID: On Mon, Sep 7, 2015 at 3:34 AM, Theepan wrote: > Since Erlang is now widely used in internet domain, I believe that > inclusion of JSON library under STDLIB is essential. There are many open > source variants, but I like Jiffy and Jsx. They support MAPS too. > Roughly, two variants of programming language standard libraries exist, where most languages sit somewhere on the continuum of these extreme points: 1. Put as little as possible in the standard library. Usually only put the things necessary to write the compiler itself, and the libraries which are impossible to write without close support from the compiler/runtime. 2. Use the Stdlib to "bless" certain implementations of commonly used libraries as the default. The Stdlib is considerably larger, but is guaranteed to have seen lots of testing. It is not a priori clear one solution beats the other. There are advantages and disadvantages to both models. Personally I lean toward option 1, because I "come from" languages which used this model: Standard ML and OCaml. The danger of such a model however is that you end up with many stdlib implementations, and in statically typed languages you often end up in situations where packages built for one stdlib are not compatible with the other stdlib. This makes people yearn for option 2. But it is not without its consequences either. Work on the stdlib is now centered around a few people, so it will invariably move slower. The other problem is that updates to the stdlib is now locked to the update rate of the compiler. This is rarely desirable. And when new releases come out, everyone has to scramble in order to upgrade everything in the stdlib they rely on. In a more compartamentalized world, you can go update packages separately. JSON is particularly nasty because different characteristics are mandated by different users. Some want fast JSON parsing. Some want correct JSON parsing. Some want mapping rules into their Erlang world. For this reason, there are always many disparate JSON libraries in any language (C has at least 10, Java has at least 20). You *can* pick one and put into the stdlib, but people still have to build their own to their wants. In my opinion, there are far too many libraries in the standard Erlang/OTP distribution. This means more work on the Ericsson Erlang/OTP team and we can't mint a new fix package for a separate area without rolling a new point release for Erlang/OTP. Moving some things into separate repositories would help a lot. But it also requires some work on package management so the transition is as seamless as possible. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From spawn.think@REDACTED Tue Sep 8 11:45:58 2015 From: spawn.think@REDACTED (Ahmed Omar) Date: Tue, 8 Sep 2015 11:45:58 +0200 Subject: [erlang-questions] segfault erts-5.10.4 (R16B03-1) In-Reply-To: References: Message-ID: Hi Lukas, Thanks for your reply. I tried with the latest version of gdb (7.10) : ### (gdb) bt full #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, block=0x128) at beam/erl_goodfit_alloc.c:439 gfallctr = 0x15e32c0 blk = 0x128 sz = 0 i = #1 0x00000000015e32c0 in ?? () No symbol table info available. #2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, size=, busy_pcrr_pp=0x8, alcu_flgs=0) at beam/erl_alloc_util.c:2370 crr = 0x128 new_p = old_blk_sz = 287 blk = 0x117 new_blk = cand_blk = cand_blk_sz = blk_sz = 3748409 nxt_blk = 0x236 nxt_blk_sz = 22950592 is_last_blk = 296 get_blk_sz = 140602277246336 #3 0x0000000000000000 in ?? () No symbol table info available. ### Best Regards, - Ahmed Omar http://about.me/spawn.think/ 2015-09-08 10:51 GMT+02:00 Lukas Larsson : > Hello, > > On Tue, Sep 8, 2015 at 10:33 AM, Ahmed Omar wrote: > >> Hi, >> We have been experiencing a segfault on our servers running a custom >> version of Ejabberd. We managed to get a core file from the last crash >> This is what we see running gdb on it: >> ###### >> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/ld-linux-x86-64.so.2 >> Core was generated by `/var/lib/ejabberd/erts-5.10.4/bin/beam.smp -K true >> -A 128 -P 2500000 -Q 500000'. >> Program terminated with signal 11, Segmentation fault. >> #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, block=0x128) >> at beam/erl_goodfit_alloc.c:439 >> 439 beam/erl_goodfit_alloc.c: No such file or directory. >> in beam/erl_goodfit_alloc.c >> ###### >> >> If we run bt full in gdb we get: >> ###### >> (gdb) bt full >> #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, block=0x128) >> at beam/erl_goodfit_alloc.c:439 >> gfallctr = 0x15e32c0 >> blk = 0x128 >> sz = 0 >> i = >> #1 0x00000000015e32c0 in ?? () >> No symbol table info available. >> #2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, >> size=Unhandled dwarf expression opcode 0xf3 >> ) at beam/erl_alloc_util.c:2370 >> crr = 0x128 >> new_p = >> old_blk_sz = 287 >> blk = 0x117 >> new_blk = >> cand_blk = >> cand_blk_sz = >> blk_sz = 3748409 >> nxt_blk = 0x236 >> nxt_blk_sz = 22950592 >> is_last_blk = 296 >> get_blk_sz = 140602277246336 >> #3 0x0000000000000000 in ?? () >> No symbol table info available. >> ####### >> >> Is there a way to get more information? maybe which driver made the >> realloc call? >> > > Something is wrong/missing from this stacktrace. The gdb that you are > using does not seem to understand the dwarf2 extension (at least that's > what I guess after googling "Unhandled dwarf expression opcode 0xf3"), and > can only find two of the frames. Try to install a later version of gdb and > then do a bt full. > > >> >> Best Regards, >> - Ahmed Omar >> http://about.me/spawn.think/ >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From garazdawi@REDACTED Tue Sep 8 12:11:38 2015 From: garazdawi@REDACTED (Lukas Larsson) Date: Tue, 8 Sep 2015 12:11:38 +0200 Subject: [erlang-questions] segfault erts-5.10.4 (R16B03-1) In-Reply-To: References: Message-ID: Hello, Unfortunately that did not help, it just made some more arguments available. I was hoping that it would give a full stack. If you do "p allctr->name_prefix" you can get to know which allocator it is that is miss behaving. I'm guessing that it will be "driver_", which is any allocations done in nifs+linked-in drivers. Without a full stacktrace it will be hard to figure out what is wrong. One thing that you could do it to use the etp gdb macros distributed with the Erlang/OTP source code. If you in gdb do "source $ERL_TOP/erts/etc/unix/etp-commands" (replacing $ERL_TOP with the path to the source of R16B03-1 Erlang/OTP from github. It is only this file that is needed, so if you need to copy this onto a server somewhere you only need this file) you will get access to a lot of helpful gdb macros. If you then do "etp-ports" you will get printed to the shell all ports that are alive at the moment of the crash. Look for any ports with a state that looks different. That will most likely be the port that is just executing. e.g. for me this is a currently running port: Pix: 2576 Port: #Port<0.322> Name: tty_sl -c -e State: connected soft-eof Scheduler flags: GARBAGE Connected: <0.25.0> Pointer: (Port *) 0x7ffff54809d8 to get the name of the currently running driver do "p ((Port *)0x7ffff54809d8)->drv_ptr->name". If no port is executing, it might be a nif, then you can do "etp-processes" to get a list of all processes in the system. Again look for any state that looks different. e.g. Pix: 200 Pid: <0.25.0> State: trapping-exit | running | active | prq-prio-normal | usr-prio-normal | act-prio-normal Registered name: user_drv I: #Cp Heap size: 610 Old-heap size: 987 Mbuf size: 0 Msgq len: 0 (inner=0, outer=0) Parent: <0.24.0> Pointer: (Process *) 0x7ffff51c4df0 This is a currently running process (State: running). You can get a stackdump of the process by doing: etp-stackdump ((Process*)0x7ffff51c4df0) Note that etp-processes and etp-ports will take quite some time to run until they finish. They need to iterate over all possibly processes/ports, and gdb is not the fastest scripting language in the world. For some help with the etp gdb commands you can issue "etp-help". Happy hunting! Lukas On Tue, Sep 8, 2015 at 11:45 AM, Ahmed Omar wrote: > Hi Lukas, > Thanks for your reply. I tried with the latest version of gdb (7.10) : > > ### > (gdb) bt full > #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, block=0x128) > at beam/erl_goodfit_alloc.c:439 > gfallctr = 0x15e32c0 > blk = 0x128 > sz = 0 > i = > #1 0x00000000015e32c0 in ?? () > No symbol table info available. > #2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, > size=, busy_pcrr_pp=0x8, alcu_flgs=0) at > beam/erl_alloc_util.c:2370 > crr = 0x128 > new_p = > old_blk_sz = 287 > blk = 0x117 > new_blk = > cand_blk = > cand_blk_sz = > blk_sz = 3748409 > nxt_blk = 0x236 > nxt_blk_sz = 22950592 > is_last_blk = 296 > get_blk_sz = 140602277246336 > #3 0x0000000000000000 in ?? () > No symbol table info available. > ### > > Best Regards, > - Ahmed Omar > http://about.me/spawn.think/ > > 2015-09-08 10:51 GMT+02:00 Lukas Larsson : > >> Hello, >> >> On Tue, Sep 8, 2015 at 10:33 AM, Ahmed Omar >> wrote: >> >>> Hi, >>> We have been experiencing a segfault on our servers running a custom >>> version of Ejabberd. We managed to get a core file from the last crash >>> This is what we see running gdb on it: >>> ###### >>> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols >>> found)...done. >>> Loaded symbols for /lib64/ld-linux-x86-64.so.2 >>> Core was generated by `/var/lib/ejabberd/erts-5.10.4/bin/beam.smp -K >>> true -A 128 -P 2500000 -Q 500000'. >>> Program terminated with signal 11, Segmentation fault. >>> #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, >>> block=0x128) at beam/erl_goodfit_alloc.c:439 >>> 439 beam/erl_goodfit_alloc.c: No such file or directory. >>> in beam/erl_goodfit_alloc.c >>> ###### >>> >>> If we run bt full in gdb we get: >>> ###### >>> (gdb) bt full >>> #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, >>> block=0x128) at beam/erl_goodfit_alloc.c:439 >>> gfallctr = 0x15e32c0 >>> blk = 0x128 >>> sz = 0 >>> i = >>> #1 0x00000000015e32c0 in ?? () >>> No symbol table info available. >>> #2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, >>> size=Unhandled dwarf expression opcode 0xf3 >>> ) at beam/erl_alloc_util.c:2370 >>> crr = 0x128 >>> new_p = >>> old_blk_sz = 287 >>> blk = 0x117 >>> new_blk = >>> cand_blk = >>> cand_blk_sz = >>> blk_sz = 3748409 >>> nxt_blk = 0x236 >>> nxt_blk_sz = 22950592 >>> is_last_blk = 296 >>> get_blk_sz = 140602277246336 >>> #3 0x0000000000000000 in ?? () >>> No symbol table info available. >>> ####### >>> >>> Is there a way to get more information? maybe which driver made the >>> realloc call? >>> >> >> Something is wrong/missing from this stacktrace. The gdb that you are >> using does not seem to understand the dwarf2 extension (at least that's >> what I guess after googling "Unhandled dwarf expression opcode 0xf3"), and >> can only find two of the frames. Try to install a later version of gdb and >> then do a bt full. >> >> >>> >>> Best Regards, >>> - Ahmed Omar >>> http://about.me/spawn.think/ >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From askjuise@REDACTED Tue Sep 8 12:33:29 2015 From: askjuise@REDACTED (Alexander Petrovsky) Date: Tue, 8 Sep 2015 13:33:29 +0300 Subject: [erlang-questions] Dialyzer and erlsom Message-ID: Hi! I've built PLTs with the follow commands via Makefile: DIALYZER_APPS = asn1 compiler crypto erts inets kernel public_key sasl ssl stdlib syntax_tools tools ... dialyzer --build_plt --output_plt erlang.plt --apps $(DIALYZER_APPS) dialyzer --build_plt --output_plt my_app.plt ebin/ deps/*/ebin/ then I want to check my code with dialyzer but got the follow errors: $ make dialyzer dialyzer --src src --plts erlang.plt my_app.plt -Wunmatched_returns -Werror_handling -Wrace_conditions -Wunderspecs | fgrep -v -f ./dialyzer.ignore-warnings Checking whether the PLT erlang.plt is up-to-date... yes Checking whether the PLT my_app.plt is up-to-date... yes Proceeding with analysis... dialyzer: Analysis failed with error: Could not scan the following file(s): /home/juise/my_app/src/my_app_parser.erl:10: can't find include lib "erlsom/src/erlsom_sax.hrl" /home/juise/my_app/src/my_app_parser.erl:99: record attribute undefined /home/juise/my_app/src/my_app_parser.erl:100: variable 'Name' is unbound /home/juise/my_app/src/my_app_parser.erl:100: variable 'Name' is unbound /home/juise/my_app/src/my_app_parser.erl:100: variable 'Value' is unbound In my_app_parser I have: -include_lib("erlsom/src/erlsom_sax.hrl"). How to say dialyzer to find erlsom_sax.hrl in deps/erlsom/src/ ? How can I fix it? -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: From aronisstav@REDACTED Tue Sep 8 12:43:53 2015 From: aronisstav@REDACTED (Stavros Aronis) Date: Tue, 8 Sep 2015 12:43:53 +0200 Subject: [erlang-questions] Dialyzer and erlsom In-Reply-To: References: Message-ID: Hi Alexander, when it comes to included files (or parse transforms), Dialyzer accepts the same flags as the compiler. In your case adding "-I deps/erlsom/src/" should be enough. It is however *much* simpler to just make sure that you compile your code with "+debug_info" and run Dialyzer directly on the .beam file(s). Best, Stavros Aronis On Tue, Sep 8, 2015 at 12:33 PM, Alexander Petrovsky wrote: > Hi! > > I've built PLTs with the follow commands via Makefile: > > DIALYZER_APPS = asn1 compiler crypto erts inets kernel public_key sasl ssl > stdlib syntax_tools tools > > ... > > dialyzer --build_plt --output_plt erlang.plt --apps $(DIALYZER_APPS) > dialyzer --build_plt --output_plt my_app.plt ebin/ deps/*/ebin/ > > then I want to check my code with dialyzer but got the follow errors: > > $ make dialyzer > dialyzer --src src --plts erlang.plt my_app.plt -Wunmatched_returns > -Werror_handling -Wrace_conditions -Wunderspecs | fgrep -v -f > ./dialyzer.ignore-warnings > Checking whether the PLT erlang.plt is up-to-date... yes > Checking whether the PLT my_app.plt is up-to-date... yes > Proceeding with analysis... > dialyzer: Analysis failed with error: > Could not scan the following file(s): > /home/juise/my_app/src/my_app_parser.erl:10: can't find include lib > "erlsom/src/erlsom_sax.hrl" > /home/juise/my_app/src/my_app_parser.erl:99: record attribute undefined > /home/juise/my_app/src/my_app_parser.erl:100: variable 'Name' is unbound > /home/juise/my_app/src/my_app_parser.erl:100: variable 'Name' is unbound > /home/juise/my_app/src/my_app_parser.erl:100: variable 'Value' is unbound > > In my_app_parser I have: > > -include_lib("erlsom/src/erlsom_sax.hrl"). > > How to say dialyzer to find erlsom_sax.hrl in deps/erlsom/src/ ? How can I > fix it? > > -- > ?????????? ????????? / Alexander Petrovsky, > > Skype: askjuise > Phone: +7 914 8 820 815 > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Tue Sep 8 13:30:11 2015 From: kostis@REDACTED (Kostis Sagonas) Date: Tue, 08 Sep 2015 13:30:11 +0200 Subject: [erlang-questions] Dialyzer and erlsom In-Reply-To: References: Message-ID: <55EEC6C3.7070803@cs.ntua.gr> On 09/08/2015 12:33 PM, Alexander Petrovsky wrote: > Hi! > > I've built PLTs with the follow commands via Makefile: > > DIALYZER_APPS = asn1 compiler crypto erts inets kernel public_key sasl > ssl stdlib syntax_tools tools > > ... > > dialyzer --build_plt --output_plt erlang.plt --apps $(DIALYZER_APPS) > dialyzer --build_plt --output_plt my_app.plt ebin/ deps/*/ebin/ > > then I want to check my code with dialyzer but got the follow errors: > > $ make dialyzer > dialyzer --src src --plts erlang.plt my_app.plt -Wunmatched_returns > -Werror_handling -Wrace_conditions -Wunderspecs | fgrep -v -f > ./dialyzer.ignore-warnings > Checking whether the PLT erlang.plt is up-to-date... yes > Checking whether the PLT my_app.plt is up-to-date... yes > Proceeding with analysis... > dialyzer: Analysis failed with error: > ... > > In my_app_parser I have: > > -include_lib("erlsom/src/erlsom_sax.hrl"). > > How to say dialyzer to find erlsom_sax.hrl in deps/erlsom/src/ ? How can > I fix it? Stavros has already replied to your question (alas in a top-post :-( ) mentioning the -I option that dialyzer accepts and the fact that you'll be better off analyzing .beam files instead of .erl ones. I wanted to point out something else, slightly unrelated to your question, because it's a common error I see in the use of dialyzer. You start by building an erlang.plt file for all Erlang/OTP apps you are using. That's great. But then you are building a PLT not only for the dependencies of your application, which is fine, but also for the files (those under ebin/) that you want to then analyze. Why do you do that? There is no real harm in doing so, but this means that if you subsequently modify any of the files under your src/ and re-compile, the information in the cached my_app.plt file will need to be rebuilt. The idea of PLT files is that you cache there type information of files that are considered as the "trusted base" of dialyzer's analysis, which consists of files that rarely change (like those in the Erlang/OTP system you employ). So, I would argue against building/including in the PLT any information about the files of the application you want to dialyze. Kostis From john.foldager@REDACTED Tue Sep 8 13:58:04 2015 From: john.foldager@REDACTED (John Foldager) Date: Tue, 8 Sep 2015 13:58:04 +0200 Subject: [erlang-questions] TLS cipher suite with Galois Counter Mode (GCM) Message-ID: We're using RabbitMQ and now have a request for supporting the following cipher suites: TLS_RSA_WITH_AES_128_GCM_SHA256 (0X009C) TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0XC02F) TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0X009E) TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0XC02B) However, if we use the following commands from the Erlang CLI we don't see these cipher suites: io:format("~p", [ssl:cipher_suites(openssl)]). io:format("~p", [ssl:cipher_suites(erlang)]). So how can we make these cipher suites available to Erlang.... and then RabbitMQ that runs on top of Erlang? -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuncer.ayaz@REDACTED Tue Sep 8 14:05:48 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Tue, 8 Sep 2015 14:05:48 +0200 Subject: [erlang-questions] TLS cipher suite with Galois Counter Mode (GCM) In-Reply-To: References: Message-ID: On Tue, Sep 8, 2015 at 1:58 PM, John Foldager wrote: > We're using RabbitMQ and now have a request for supporting the > following cipher suites: > > TLS_RSA_WITH_AES_128_GCM_SHA256 (0X009C) > TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0XC02F) > TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0X009E) > TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0XC02B) > > However, if we use the following commands from the Erlang CLI we > don't see these cipher suites: > > io:format("~p", [ssl:cipher_suites(openssl)]). > io:format("~p", [ssl:cipher_suites(erlang)]). > > So how can we make these cipher suites available to Erlang.... and > then RabbitMQ that runs on top of Erlang? To rule out the obvious and assuming there's no difference in the OpenSSL install between the build and deploy host, does openssl list it as available? $ openssl ciphers -v|grep AES128-GCM-SHA256 From askjuise@REDACTED Tue Sep 8 14:25:38 2015 From: askjuise@REDACTED (Alexander Petrovsky) Date: Tue, 8 Sep 2015 15:25:38 +0300 Subject: [erlang-questions] Dialyzer and erlsom In-Reply-To: References: Message-ID: Stavros, thanks a lot! I've tried to run with "-I deps/erlsom/src/", but it's doesn't help, also I've tried like smth this: "-I include/* deps/*/src/* deps/*/include/*" but it's doesn't work too. As you suggested I run Dialyzer directly on the .beam files and it's work perfectly! Thanks! 2015-09-08 13:43 GMT+03:00 Stavros Aronis : > Hi Alexander, > > when it comes to included files (or parse transforms), Dialyzer accepts > the same flags as the compiler. In your case adding "-I deps/erlsom/src/" > should be enough. > > It is however *much* simpler to just make sure that you compile your code > with "+debug_info" and run Dialyzer directly on the .beam file(s). > > Best, > > Stavros Aronis > > On Tue, Sep 8, 2015 at 12:33 PM, Alexander Petrovsky > wrote: > >> Hi! >> >> I've built PLTs with the follow commands via Makefile: >> >> DIALYZER_APPS = asn1 compiler crypto erts inets kernel public_key sasl >> ssl stdlib syntax_tools tools >> >> ... >> >> dialyzer --build_plt --output_plt erlang.plt --apps $(DIALYZER_APPS) >> dialyzer --build_plt --output_plt my_app.plt ebin/ deps/*/ebin/ >> >> then I want to check my code with dialyzer but got the follow errors: >> >> $ make dialyzer >> dialyzer --src src --plts erlang.plt my_app.plt -Wunmatched_returns >> -Werror_handling -Wrace_conditions -Wunderspecs | fgrep -v -f >> ./dialyzer.ignore-warnings >> Checking whether the PLT erlang.plt is up-to-date... yes >> Checking whether the PLT my_app.plt is up-to-date... yes >> Proceeding with analysis... >> dialyzer: Analysis failed with error: >> Could not scan the following file(s): >> /home/juise/my_app/src/my_app_parser.erl:10: can't find include lib >> "erlsom/src/erlsom_sax.hrl" >> /home/juise/my_app/src/my_app_parser.erl:99: record attribute undefined >> /home/juise/my_app/src/my_app_parser.erl:100: variable 'Name' is unbound >> /home/juise/my_app/src/my_app_parser.erl:100: variable 'Name' is unbound >> /home/juise/my_app/src/my_app_parser.erl:100: variable 'Value' is unbound >> >> In my_app_parser I have: >> >> -include_lib("erlsom/src/erlsom_sax.hrl"). >> >> How to say dialyzer to find erlsom_sax.hrl in deps/erlsom/src/ ? How can >> I fix it? >> >> -- >> ?????????? ????????? / Alexander Petrovsky, >> >> Skype: askjuise >> Phone: +7 914 8 820 815 >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: From askjuise@REDACTED Tue Sep 8 14:28:56 2015 From: askjuise@REDACTED (Alexander Petrovsky) Date: Tue, 8 Sep 2015 15:28:56 +0300 Subject: [erlang-questions] Dialyzer and erlsom In-Reply-To: <55EEC6C3.7070803@cs.ntua.gr> References: <55EEC6C3.7070803@cs.ntua.gr> Message-ID: Kostis, thanks for your advice. Do I understand correctly, you suggest just run dialyzer without my_app.plt? I'll think about it, looks like it make sense! 2015-09-08 14:30 GMT+03:00 Kostis Sagonas : > On 09/08/2015 12:33 PM, Alexander Petrovsky wrote: > >> Hi! >> >> I've built PLTs with the follow commands via Makefile: >> >> DIALYZER_APPS = asn1 compiler crypto erts inets kernel public_key sasl >> ssl stdlib syntax_tools tools >> >> ... >> >> dialyzer --build_plt --output_plt erlang.plt --apps $(DIALYZER_APPS) >> dialyzer --build_plt --output_plt my_app.plt ebin/ deps/*/ebin/ >> >> then I want to check my code with dialyzer but got the follow errors: >> >> $ make dialyzer >> dialyzer --src src --plts erlang.plt my_app.plt -Wunmatched_returns >> -Werror_handling -Wrace_conditions -Wunderspecs | fgrep -v -f >> ./dialyzer.ignore-warnings >> Checking whether the PLT erlang.plt is up-to-date... yes >> Checking whether the PLT my_app.plt is up-to-date... yes >> Proceeding with analysis... >> dialyzer: Analysis failed with error: >> ... >> >> In my_app_parser I have: >> >> -include_lib("erlsom/src/erlsom_sax.hrl"). >> >> How to say dialyzer to find erlsom_sax.hrl in deps/erlsom/src/ ? How can >> I fix it? >> > > Stavros has already replied to your question (alas in a top-post :-( ) > mentioning the -I option that dialyzer accepts and the fact that you'll be > better off analyzing .beam files instead of .erl ones. > > I wanted to point out something else, slightly unrelated to your question, > because it's a common error I see in the use of dialyzer. > > You start by building an erlang.plt file for all Erlang/OTP apps you are > using. That's great. But then you are building a PLT not only for the > dependencies of your application, which is fine, but also for the files > (those under ebin/) that you want to then analyze. Why do you do that? > There is no real harm in doing so, but this means that if you subsequently > modify any of the files under your src/ and re-compile, the information in > the cached my_app.plt file will need to be rebuilt. > > The idea of PLT files is that you cache there type information of files > that are considered as the "trusted base" of dialyzer's analysis, which > consists of files that rarely change (like those in the Erlang/OTP system > you employ). > > So, I would argue against building/including in the PLT any information > about the files of the application you want to dialyze. > > Kostis > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: From z@REDACTED Tue Sep 8 14:38:01 2015 From: z@REDACTED (Danil Zagoskin) Date: Tue, 8 Sep 2015 15:38:01 +0300 Subject: [erlang-questions] TLS cipher suite with Galois Counter Mode (GCM) In-Reply-To: References: Message-ID: Hi, John! What OTP version do you use? OTP17 and prior do not support GCM, you need OTP18. Old possible ciphers: https://github.com/erlang/otp/blob/maint-17/lib/ssl/src/ssl_cipher.erl#L46 New possible ciphers: https://github.com/erlang/otp/blob/maint-18/lib/ssl/src/ssl_cipher.erl#L48 (note the aes_128_gcm and aes_256_gcm ciphers). On Tue, Sep 8, 2015 at 2:58 PM, John Foldager wrote: > We're using RabbitMQ and now have a request for supporting the following > cipher suites: > TLS_RSA_WITH_AES_128_GCM_SHA256 (0X009C) > TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0XC02F) > TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0X009E) > TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0XC02B) > > However, if we use the following commands from the Erlang CLI we don't see > these cipher suites: > > io:format("~p", [ssl:cipher_suites(openssl)]). > io:format("~p", [ssl:cipher_suites(erlang)]). > > So how can we make these cipher suites available to Erlang.... and then > RabbitMQ that runs on top of Erlang? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From spawn.think@REDACTED Tue Sep 8 14:54:11 2015 From: spawn.think@REDACTED (Ahmed Omar) Date: Tue, 8 Sep 2015 14:54:11 +0200 Subject: [erlang-questions] segfault erts-5.10.4 (R16B03-1) In-Reply-To: References: Message-ID: Hi Lukas, Thanks a lot for the pointers, I'll see if I can find anything. Best Regards, - Ahmed Omar http://about.me/spawn.think/ 2015-09-08 12:11 GMT+02:00 Lukas Larsson : > Hello, > > Unfortunately that did not help, it just made some more arguments > available. I was hoping that it would give a full stack. If you do "p > allctr->name_prefix" you can get to know which allocator it is that is miss > behaving. I'm guessing that it will be "driver_", which is any allocations > done in nifs+linked-in drivers. Without a full stacktrace it will be hard > to figure out what is wrong. > > One thing that you could do it to use the etp gdb macros distributed with > the Erlang/OTP source code. If you in gdb do "source > $ERL_TOP/erts/etc/unix/etp-commands" (replacing $ERL_TOP with the path to > the source of R16B03-1 Erlang/OTP from github. It is only this file that is > needed, so if you need to copy this onto a server somewhere you only need > this file) you will get access to a lot of helpful gdb macros. > > If you then do "etp-ports" you will get printed to the shell all ports > that are alive at the moment of the crash. Look for any ports with a state > that looks different. That will most likely be the port that is just > executing. e.g. for me this is a currently running port: > > Pix: 2576 > Port: #Port<0.322> > Name: tty_sl -c -e > State: connected soft-eof > Scheduler flags: GARBAGE > Connected: <0.25.0> > Pointer: (Port *) 0x7ffff54809d8 > > to get the name of the currently running driver do "p ((Port > *)0x7ffff54809d8)->drv_ptr->name". > > If no port is executing, it might be a nif, then you can do > "etp-processes" to get a list of all processes in the system. Again look > for any state that looks different. e.g. > > Pix: 200 > Pid: <0.25.0> > State: trapping-exit | running | active | prq-prio-normal | > usr-prio-normal | act-prio-normal > Registered name: user_drv > I: #Cp > Heap size: 610 > Old-heap size: 987 > Mbuf size: 0 > Msgq len: 0 (inner=0, outer=0) > Parent: <0.24.0> > Pointer: (Process *) 0x7ffff51c4df0 > > This is a currently running process (State: running). You can get a > stackdump of the process by doing: etp-stackdump ((Process*)0x7ffff51c4df0) > > Note that etp-processes and etp-ports will take quite some time to run > until they finish. They need to iterate over all possibly processes/ports, > and gdb is not the fastest scripting language in the world. > > For some help with the etp gdb commands you can issue "etp-help". > > Happy hunting! > Lukas > > On Tue, Sep 8, 2015 at 11:45 AM, Ahmed Omar wrote: > >> Hi Lukas, >> Thanks for your reply. I tried with the latest version of gdb (7.10) : >> >> ### >> (gdb) bt full >> #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, block=0x128) >> at beam/erl_goodfit_alloc.c:439 >> gfallctr = 0x15e32c0 >> blk = 0x128 >> sz = 0 >> i = >> #1 0x00000000015e32c0 in ?? () >> No symbol table info available. >> #2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, >> size=, busy_pcrr_pp=0x8, alcu_flgs=0) at >> beam/erl_alloc_util.c:2370 >> crr = 0x128 >> new_p = >> old_blk_sz = 287 >> blk = 0x117 >> new_blk = >> cand_blk = >> cand_blk_sz = >> blk_sz = 3748409 >> nxt_blk = 0x236 >> nxt_blk_sz = 22950592 >> is_last_blk = 296 >> get_blk_sz = 140602277246336 >> #3 0x0000000000000000 in ?? () >> No symbol table info available. >> ### >> >> Best Regards, >> - Ahmed Omar >> http://about.me/spawn.think/ >> >> 2015-09-08 10:51 GMT+02:00 Lukas Larsson : >> >>> Hello, >>> >>> On Tue, Sep 8, 2015 at 10:33 AM, Ahmed Omar >>> wrote: >>> >>>> Hi, >>>> We have been experiencing a segfault on our servers running a custom >>>> version of Ejabberd. We managed to get a core file from the last crash >>>> This is what we see running gdb on it: >>>> ###### >>>> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging >>>> symbols found)...done. >>>> Loaded symbols for /lib64/ld-linux-x86-64.so.2 >>>> Core was generated by `/var/lib/ejabberd/erts-5.10.4/bin/beam.smp -K >>>> true -A 128 -P 2500000 -Q 500000'. >>>> Program terminated with signal 11, Segmentation fault. >>>> #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, >>>> block=0x128) at beam/erl_goodfit_alloc.c:439 >>>> 439 beam/erl_goodfit_alloc.c: No such file or directory. >>>> in beam/erl_goodfit_alloc.c >>>> ###### >>>> >>>> If we run bt full in gdb we get: >>>> ###### >>>> (gdb) bt full >>>> #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, >>>> block=0x128) at beam/erl_goodfit_alloc.c:439 >>>> gfallctr = 0x15e32c0 >>>> blk = 0x128 >>>> sz = 0 >>>> i = >>>> #1 0x00000000015e32c0 in ?? () >>>> No symbol table info available. >>>> #2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, >>>> size=Unhandled dwarf expression opcode 0xf3 >>>> ) at beam/erl_alloc_util.c:2370 >>>> crr = 0x128 >>>> new_p = >>>> old_blk_sz = 287 >>>> blk = 0x117 >>>> new_blk = >>>> cand_blk = >>>> cand_blk_sz = >>>> blk_sz = 3748409 >>>> nxt_blk = 0x236 >>>> nxt_blk_sz = 22950592 >>>> is_last_blk = 296 >>>> get_blk_sz = 140602277246336 >>>> #3 0x0000000000000000 in ?? () >>>> No symbol table info available. >>>> ####### >>>> >>>> Is there a way to get more information? maybe which driver made the >>>> realloc call? >>>> >>> >>> Something is wrong/missing from this stacktrace. The gdb that you are >>> using does not seem to understand the dwarf2 extension (at least that's >>> what I guess after googling "Unhandled dwarf expression opcode 0xf3"), and >>> can only find two of the frames. Try to install a later version of gdb and >>> then do a bt full. >>> >>> >>>> >>>> Best Regards, >>>> - Ahmed Omar >>>> http://about.me/spawn.think/ >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mremond@REDACTED Tue Sep 8 15:09:09 2015 From: mremond@REDACTED (=?utf-8?b?TWlja2HDq2wgUsOpbW9uZA==?=) Date: Tue, 08 Sep 2015 15:09:09 +0200 Subject: [erlang-questions] [ANN] Nine Nines, from Cowboy fame, joins Advanced Erlang Initiative Message-ID: <3928D494-7192-40A1-8595-9DBF998C11C2@process-one.net> Hello, We are happy to announce that Nine Nines is joining Advanced Erlang Initiative. Nine Nines is the company behind Cowboy web server, the reference Erlang HTTP server. Cowboy is used by many other projects like the Elixir Phoenix framework or ChicagoBoss, making it a very popular product, widely acclaimed in the Erlang community. When Advanced Erlang Initiative was launched a couple of months ago, it was clear that the ecosystem of Erlang products was vibrant. It is very good news to see new forces joining the promotion effort around products that are critical for the Erlang ecosystem. With Nine Nines joining the effort, Lo?c Hoguin is now one of the Erlang experts involved in making the initiative successful. As such, a new day-long workshop for Cowboy is announced and will take place in March 2016. The new workshop is growing the list of worldwide expert talks that will take place in the coming months, around Quickcheck, ejabberd, and now Cowboy. You can learn more about the workshop schedule from Advanced Erlang Initiative page: http://advanced-erlang.com See you in our workshops ! -- Advanced Erlang Initiative From kostis@REDACTED Tue Sep 8 15:16:59 2015 From: kostis@REDACTED (Kostis Sagonas) Date: Tue, 08 Sep 2015 15:16:59 +0200 Subject: [erlang-questions] Dialyzer and erlsom In-Reply-To: References: <55EEC6C3.7070803@cs.ntua.gr> Message-ID: <55EEDFCB.8050504@cs.ntua.gr> On 09/08/2015 02:28 PM, Alexander Petrovsky wrote: > Kostis, thanks for your advice. Do I understand correctly, you suggest > just run dialyzer without my_app.plt? No. I am suggesting that instead of building a my_app.plt file with the information of *both* ebin/ and deps/*/ebin in it, you create and use a my_app_deps.plt file containing *only* the information of the dependencies of your application (i.e. only with deps/*/ebin). Kostis From erlangsiri@REDACTED Tue Sep 8 15:42:38 2015 From: erlangsiri@REDACTED (Siri Hansen) Date: Tue, 8 Sep 2015 15:42:38 +0200 Subject: [erlang-questions] Cannot add supervisor to application's supervisor in runtime In-Reply-To: <20150908123458.66553359@gmail.com> References: <20150908123458.66553359@gmail.com> Message-ID: Can you see if your app_sup:init function is called during the upgrade? It should be called from supervisor:code_change/3 as a result of this instruction: {code_change,up,[{app_sup,[]}]}. What does it return? Unfortunately the real time error reporting here is not very good. Do the code paths look correct after the upgrade? Regards /siri@REDACTED 2015-09-08 10:04 GMT+02:00 Ameretat Reith : > Hi, > > I'm trying to add new supervisor `ch_sup` and It's worker `ch` to > an application supervisor `app_sup` with this appup: > > ``` > {"0.1.3", > [{"0.1.2", > [{add_module, ch}, > {add_module, ch_sup}, > {update, app_sup, supervisor}, > {apply, {supervisor, restart_child, [app_sup, ch_sup]}} > ] > }], > [{"0.1.2", > [{apply, {supervisor, terminate_child, [app_sup, ch_sup]}}, > {apply, {supervisor, delete_child, [app_sup, ch_sup]}}, > {update, app_sup, supervisor}, > {delete_module, ch_sup}, > {delete_module, ch} > ] > }] > }. > ``` > > And this relup generated by systools:make_relup: > > ``` > {"0.0.3", > [{"0.0.2",[], > [{load_object_code,{app,"0.1.3", > [ch,ch_sup,app_sup]}}, > point_of_no_return, > {load,{ch,brutal_purge,brutal_purge}}, > {load,{ch_sup,brutal_purge,brutal_purge}}, > {suspend,[app_sup]}, > {load,{app_sup,brutal_purge,brutal_purge}}, > {code_change,up,[{app_sup,[]}]}, > {resume,[app_sup]}, > {apply,{supervisor,restart_child,[app_sup,ch_sup]}}]}], > [{"0.0.2",[], > [{load_object_code,{app,"0.1.2",[app_sup]}}, > point_of_no_return, > {apply,{supervisor,terminate_child,[app_sup,ch_sup]}}, > {apply,{supervisor,delete_child,[app_sup,ch_sup]}}, > {suspend,[app_sup]}, > {load,{app_sup,brutal_purge,brutal_purge}}, > {code_change,down,[{app_sup,[]}]}, > {resume,[app_sup]}, > {remove,{ch_sup,brutal_purge,brutal_purge}}, > {purge,[ch_sup]}, > {remove,{ch,brutal_purge,brutal_purge}}, > {purge,[ch]}]}]}. > ``` > > New release installs But app_sup's child spec wont change, neither > ch_sup start until I terminate app_sup and start It again with > `supervisor:restart_child(real_app_sup, app_sup)`. app is an > included application which is put in real_app's supervision tree. I > don't think app as an included application, should be regarded > special here, since I use It just as a supervisor. But seems I'm wrong.. > > So, What is the problem? Can I achieve changing app_sup's main > supervisor without restarting It's process? > > Thanks > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From spawn.think@REDACTED Tue Sep 8 16:20:17 2015 From: spawn.think@REDACTED (Ahmed Omar) Date: Tue, 8 Sep 2015 16:20:17 +0200 Subject: [erlang-questions] segfault erts-5.10.4 (R16B03-1) In-Reply-To: References: Message-ID: Hi Lukas, Is it possible to use the etp-commands on a core file or gdb has to be attached to a live node? Best Regards, - Ahmed Omar http://about.me/spawn.think/ 2015-09-08 14:54 GMT+02:00 Ahmed Omar : > Hi Lukas, > > Thanks a lot for the pointers, I'll see if I can find anything. > > Best Regards, > - Ahmed Omar > http://about.me/spawn.think/ > > 2015-09-08 12:11 GMT+02:00 Lukas Larsson : > >> Hello, >> >> Unfortunately that did not help, it just made some more arguments >> available. I was hoping that it would give a full stack. If you do "p >> allctr->name_prefix" you can get to know which allocator it is that is miss >> behaving. I'm guessing that it will be "driver_", which is any allocations >> done in nifs+linked-in drivers. Without a full stacktrace it will be hard >> to figure out what is wrong. >> >> One thing that you could do it to use the etp gdb macros distributed with >> the Erlang/OTP source code. If you in gdb do "source >> $ERL_TOP/erts/etc/unix/etp-commands" (replacing $ERL_TOP with the path to >> the source of R16B03-1 Erlang/OTP from github. It is only this file that is >> needed, so if you need to copy this onto a server somewhere you only need >> this file) you will get access to a lot of helpful gdb macros. >> >> If you then do "etp-ports" you will get printed to the shell all ports >> that are alive at the moment of the crash. Look for any ports with a state >> that looks different. That will most likely be the port that is just >> executing. e.g. for me this is a currently running port: >> >> Pix: 2576 >> Port: #Port<0.322> >> Name: tty_sl -c -e >> State: connected soft-eof >> Scheduler flags: GARBAGE >> Connected: <0.25.0> >> Pointer: (Port *) 0x7ffff54809d8 >> >> to get the name of the currently running driver do "p ((Port >> *)0x7ffff54809d8)->drv_ptr->name". >> >> If no port is executing, it might be a nif, then you can do >> "etp-processes" to get a list of all processes in the system. Again look >> for any state that looks different. e.g. >> >> Pix: 200 >> Pid: <0.25.0> >> State: trapping-exit | running | active | prq-prio-normal | >> usr-prio-normal | act-prio-normal >> Registered name: user_drv >> I: #Cp >> Heap size: 610 >> Old-heap size: 987 >> Mbuf size: 0 >> Msgq len: 0 (inner=0, outer=0) >> Parent: <0.24.0> >> Pointer: (Process *) 0x7ffff51c4df0 >> >> This is a currently running process (State: running). You can get a >> stackdump of the process by doing: etp-stackdump ((Process*)0x7ffff51c4df0) >> >> Note that etp-processes and etp-ports will take quite some time to run >> until they finish. They need to iterate over all possibly processes/ports, >> and gdb is not the fastest scripting language in the world. >> >> For some help with the etp gdb commands you can issue "etp-help". >> >> Happy hunting! >> Lukas >> >> On Tue, Sep 8, 2015 at 11:45 AM, Ahmed Omar >> wrote: >> >>> Hi Lukas, >>> Thanks for your reply. I tried with the latest version of gdb (7.10) : >>> >>> ### >>> (gdb) bt full >>> #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, >>> block=0x128) at beam/erl_goodfit_alloc.c:439 >>> gfallctr = 0x15e32c0 >>> blk = 0x128 >>> sz = 0 >>> i = >>> #1 0x00000000015e32c0 in ?? () >>> No symbol table info available. >>> #2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, >>> size=, busy_pcrr_pp=0x8, alcu_flgs=0) at >>> beam/erl_alloc_util.c:2370 >>> crr = 0x128 >>> new_p = >>> old_blk_sz = 287 >>> blk = 0x117 >>> new_blk = >>> cand_blk = >>> cand_blk_sz = >>> blk_sz = 3748409 >>> nxt_blk = 0x236 >>> nxt_blk_sz = 22950592 >>> is_last_blk = 296 >>> get_blk_sz = 140602277246336 >>> #3 0x0000000000000000 in ?? () >>> No symbol table info available. >>> ### >>> >>> Best Regards, >>> - Ahmed Omar >>> http://about.me/spawn.think/ >>> >>> 2015-09-08 10:51 GMT+02:00 Lukas Larsson : >>> >>>> Hello, >>>> >>>> On Tue, Sep 8, 2015 at 10:33 AM, Ahmed Omar >>>> wrote: >>>> >>>>> Hi, >>>>> We have been experiencing a segfault on our servers running a custom >>>>> version of Ejabberd. We managed to get a core file from the last crash >>>>> This is what we see running gdb on it: >>>>> ###### >>>>> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging >>>>> symbols found)...done. >>>>> Loaded symbols for /lib64/ld-linux-x86-64.so.2 >>>>> Core was generated by `/var/lib/ejabberd/erts-5.10.4/bin/beam.smp -K >>>>> true -A 128 -P 2500000 -Q 500000'. >>>>> Program terminated with signal 11, Segmentation fault. >>>>> #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, >>>>> block=0x128) at beam/erl_goodfit_alloc.c:439 >>>>> 439 beam/erl_goodfit_alloc.c: No such file or directory. >>>>> in beam/erl_goodfit_alloc.c >>>>> ###### >>>>> >>>>> If we run bt full in gdb we get: >>>>> ###### >>>>> (gdb) bt full >>>>> #0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, >>>>> block=0x128) at beam/erl_goodfit_alloc.c:439 >>>>> gfallctr = 0x15e32c0 >>>>> blk = 0x128 >>>>> sz = 0 >>>>> i = >>>>> #1 0x00000000015e32c0 in ?? () >>>>> No symbol table info available. >>>>> #2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, >>>>> size=Unhandled dwarf expression opcode 0xf3 >>>>> ) at beam/erl_alloc_util.c:2370 >>>>> crr = 0x128 >>>>> new_p = >>>>> old_blk_sz = 287 >>>>> blk = 0x117 >>>>> new_blk = >>>>> cand_blk = >>>>> cand_blk_sz = >>>>> blk_sz = 3748409 >>>>> nxt_blk = 0x236 >>>>> nxt_blk_sz = 22950592 >>>>> is_last_blk = 296 >>>>> get_blk_sz = 140602277246336 >>>>> #3 0x0000000000000000 in ?? () >>>>> No symbol table info available. >>>>> ####### >>>>> >>>>> Is there a way to get more information? maybe which driver made the >>>>> realloc call? >>>>> >>>> >>>> Something is wrong/missing from this stacktrace. The gdb that you are >>>> using does not seem to understand the dwarf2 extension (at least that's >>>> what I guess after googling "Unhandled dwarf expression opcode 0xf3"), and >>>> can only find two of the frames. Try to install a later version of gdb and >>>> then do a bt full. >>>> >>>> >>>>> >>>>> Best Regards, >>>>> - Ahmed Omar >>>>> http://about.me/spawn.think/ >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From garazdawi@REDACTED Tue Sep 8 16:26:16 2015 From: garazdawi@REDACTED (Lukas Larsson) Date: Tue, 8 Sep 2015 16:26:16 +0200 Subject: [erlang-questions] segfault erts-5.10.4 (R16B03-1) In-Reply-To: References: Message-ID: On Tue, Sep 8, 2015 at 4:20 PM, Ahmed Omar wrote: > Is it possible to use the etp-commands on a core file or gdb has to be > attached to a live node? > It is possible to use on both core files and live sessions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From spawn.think@REDACTED Tue Sep 8 16:34:12 2015 From: spawn.think@REDACTED (Ahmed Omar) Date: Tue, 8 Sep 2015 16:34:12 +0200 Subject: [erlang-questions] segfault erts-5.10.4 (R16B03-1) In-Reply-To: References: Message-ID: Ok. I guess etp-ports require a live session then? (gdb) etp-ports No ports, since system isn't initialised! Is there a way to get port or process number from the information I got from bt ? #2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, size=, busy_pcrr_pp=0x8, alcu_flgs=0) Best Regards, - Ahmed Omar http://about.me/spawn.think/ 2015-09-08 16:26 GMT+02:00 Lukas Larsson : > On Tue, Sep 8, 2015 at 4:20 PM, Ahmed Omar wrote: > >> Is it possible to use the etp-commands on a core file or gdb has to be >> attached to a live node? >> > > It is possible to use on both core files and live sessions. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From garazdawi@REDACTED Tue Sep 8 16:45:58 2015 From: garazdawi@REDACTED (Lukas Larsson) Date: Tue, 8 Sep 2015 16:45:58 +0200 Subject: [erlang-questions] segfault erts-5.10.4 (R16B03-1) In-Reply-To: References: Message-ID: On Tue, Sep 8, 2015 at 4:34 PM, Ahmed Omar wrote: > Ok. I guess etp-ports require a live session then? > (gdb) etp-ports > No ports, since system isn't initialised! > > Is there a way to get port or process number from the information I got > from bt ? > #2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, > size=, busy_pcrr_pp=0x8, alcu_flgs=0) > > > All etp macros work on both core files a live nodes. The only time I've seen that error from etp-ports is when the VM crashed before the internal ports array was initialized. I'm starting to suspect that there is something wrong with your debugging session. The stack trace that you have printed is corrupt, and it cannot find a working port table. Make extra sure that you are using the same beam.smp to run your gdb session from as generated the core. Since the stack is corrupt it is not possible to know what called into the realloc from the stack trace. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Tue Sep 8 18:15:00 2015 From: bob@REDACTED (Bob Ippolito) Date: Tue, 8 Sep 2015 09:15:00 -0700 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: References: Message-ID: On Tue, Sep 8, 2015 at 2:42 AM, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > On Mon, Sep 7, 2015 at 3:34 AM, Theepan wrote: > >> Since Erlang is now widely used in internet domain, I believe that >> inclusion of JSON library under STDLIB is essential. There are many open >> source variants, but I like Jiffy and Jsx. They support MAPS too. >> > > Roughly, two variants of programming language standard libraries exist, > where most languages sit somewhere on the continuum of these extreme points: > > 1. Put as little as possible in the standard library. Usually only put the > things necessary to write the compiler itself, and the libraries which are > impossible to write without close support from the compiler/runtime. > > 2. Use the Stdlib to "bless" certain implementations of commonly used > libraries as the default. The Stdlib is considerably larger, but is > guaranteed to have seen lots of testing. > > It is not a priori clear one solution beats the other. There are > advantages and disadvantages to both models. Personally I lean toward > option 1, because I "come from" languages which used this model: Standard > ML and OCaml. The danger of such a model however is that you end up with > many stdlib implementations, and in statically typed languages you often > end up in situations where packages built for one stdlib are not compatible > with the other stdlib. > > This makes people yearn for option 2. But it is not without its > consequences either. Work on the stdlib is now centered around a few > people, so it will invariably move slower. The other problem is that > updates to the stdlib is now locked to the update rate of the compiler. > This is rarely desirable. And when new releases come out, everyone has to > scramble in order to upgrade everything in the stdlib they rely on. In a > more compartamentalized world, you can go update packages separately. > > JSON is particularly nasty because different characteristics are mandated > by different users. Some want fast JSON parsing. Some want correct JSON > parsing. Some want mapping rules into their Erlang world. For this reason, > there are always many disparate JSON libraries in any language (C has at > least 10, Java has at least 20). You *can* pick one and put into the > stdlib, but people still have to build their own to their wants. > > In my opinion, there are far too many libraries in the standard Erlang/OTP > distribution. This means more work on the Ericsson Erlang/OTP team and we > can't mint a new fix package for a separate area without rolling a new > point release for Erlang/OTP. Moving some things into separate repositories > would help a lot. But it also requires some work on package management so > the transition is as seamless as possible. > Yeah, what's really needed here is a working dependency management solution that the community standardizes on. Some brave person can build the high-performance library that does the low-level JSON stream parsing and nasty string/float encoding bits, and then a mountain of higher-level libraries can use that as a common dependency. -bob -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Tue Sep 8 18:18:59 2015 From: vasdeveloper@REDACTED (Theepan) Date: Tue, 8 Sep 2015 21:48:59 +0530 Subject: [erlang-questions] JSON for STDLIB In-Reply-To: References: Message-ID: Yes, That makes sense.. Theepan On Tue, Sep 8, 2015 at 9:45 PM, Bob Ippolito wrote: > > > On Tue, Sep 8, 2015 at 2:42 AM, Jesper Louis Andersen < > jesper.louis.andersen@REDACTED> wrote: > >> On Mon, Sep 7, 2015 at 3:34 AM, Theepan wrote: >> >>> Since Erlang is now widely used in internet domain, I believe that >>> inclusion of JSON library under STDLIB is essential. There are many open >>> source variants, but I like Jiffy and Jsx. They support MAPS too. >>> >> >> Roughly, two variants of programming language standard libraries exist, >> where most languages sit somewhere on the continuum of these extreme points: >> >> 1. Put as little as possible in the standard library. Usually only put >> the things necessary to write the compiler itself, and the libraries which >> are impossible to write without close support from the compiler/runtime. >> >> 2. Use the Stdlib to "bless" certain implementations of commonly used >> libraries as the default. The Stdlib is considerably larger, but is >> guaranteed to have seen lots of testing. >> >> It is not a priori clear one solution beats the other. There are >> advantages and disadvantages to both models. Personally I lean toward >> option 1, because I "come from" languages which used this model: Standard >> ML and OCaml. The danger of such a model however is that you end up with >> many stdlib implementations, and in statically typed languages you often >> end up in situations where packages built for one stdlib are not compatible >> with the other stdlib. >> >> This makes people yearn for option 2. But it is not without its >> consequences either. Work on the stdlib is now centered around a few >> people, so it will invariably move slower. The other problem is that >> updates to the stdlib is now locked to the update rate of the compiler. >> This is rarely desirable. And when new releases come out, everyone has to >> scramble in order to upgrade everything in the stdlib they rely on. In a >> more compartamentalized world, you can go update packages separately. >> >> JSON is particularly nasty because different characteristics are mandated >> by different users. Some want fast JSON parsing. Some want correct JSON >> parsing. Some want mapping rules into their Erlang world. For this reason, >> there are always many disparate JSON libraries in any language (C has at >> least 10, Java has at least 20). You *can* pick one and put into the >> stdlib, but people still have to build their own to their wants. >> >> In my opinion, there are far too many libraries in the standard >> Erlang/OTP distribution. This means more work on the Ericsson Erlang/OTP >> team and we can't mint a new fix package for a separate area without >> rolling a new point release for Erlang/OTP. Moving some things into >> separate repositories would help a lot. But it also requires some work on >> package management so the transition is as seamless as possible. >> > > Yeah, what's really needed here is a working dependency management > solution that the community standardizes on. Some brave person can build > the high-performance library that does the low-level JSON stream parsing > and nasty string/float encoding bits, and then a mountain of higher-level > libraries can use that as a common dependency. > > -bob > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Sep 9 04:32:58 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 9 Sep 2015 14:32:58 +1200 Subject: [erlang-questions] =?utf-8?q?Setting_maximum_amount_of_data_alloc?= =?utf-8?q?ated_by_an_Erlang_node=2E=E2=80=8F?= In-Reply-To: <55EAA5B7.2000802@gmail.com> References: , <55EAA5B7.2000802@gmail.com> Message-ID: On 5/09/2015, at 8:20 pm, Michael Truog wrote: > Its a bit complex to track the exact amount of memory inside the Erlang VM in Erlang source code to do > something useful with the information due to the various memory pools and the garbage collection. Every year in the concurrent programming paper I tell students "In a concurrent world there is no global NOW". I point out that this means, for example, that any answer to a question like "how much memory is in use NOW" is either meaningless or out of date when you look at it. More precisely, the only way an Erlang process could get a definitive answer to "how much memory is in use NOW" is if every process were frozen, garbage collected, and then that one process resumed, not letting the others do anything that might allocate (or free up) memory until it has finished deciding what to about the result. If you needed concurrency in the first place, this would most likely be a Bad Thing to Do. So whatever the use case is, it will have to be content with answers that are at best approximate and out of date. And this would be so with *any* programming language, including C++ and Java. (For what it's worth, I'm familiar with the idea of a "time-like slice" in General Relativity. So "no global NOW" is a very slight exaggeration.) From ok@REDACTED Wed Sep 9 04:37:43 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 9 Sep 2015 14:37:43 +1200 Subject: [erlang-questions] =?utf-8?q?Setting_maximum_amount_of_data_alloc?= =?utf-8?q?ated_by_an_Erlang_node=2E=E2=80=8F?= In-Reply-To: References: <55EAA5B7.2000802@gmail.com> Message-ID: On 6/09/2015, at 12:07 am, Akash Chowdhury wrote: > The use case is that the application is using huge memory. In some occurrences, one node grew until 100GB and the application as well as the box crashed. We would like to setup a limit so that application and box doesn't crash. Enquiring about total memory or limiting total memory does not seem like a very helpful thing to do here. You need to know *where* the memory is actually going. If you limit the memory available to that OS process, it will just make the application crash earlier (but may save the box); it won't actually fix the performance bug. In fact this is a much more hopeful case, because Erlang has good performance monitoring tools, and there is probably a single process that needs to control its *own* memory usage rather than bothering about what other processes are doing, and that *is* doable in a concurrent world. From john.foldager@REDACTED Wed Sep 9 09:37:48 2015 From: john.foldager@REDACTED (John Foldager) Date: Wed, 9 Sep 2015 09:37:48 +0200 Subject: [erlang-questions] TLS cipher suite with Galois Counter Mode (GCM) In-Reply-To: References: Message-ID: Output from that command gives: ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESGCM(128) Mac=AEAD DHE-DSS-AES128-GCM-SHA256 TLSv1.2 Kx=DH Au=DSS Enc=AESGCM(128) Mac=AEAD DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AESGCM(128) Mac=AEAD ECDH-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(128) Mac=AEAD ECDH-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(128) Mac=AEAD AES128-GCM-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AESGCM(128) Mac=AEAD As Danil mentioned we need to upgrade Erlang to OTP 18, but then we must first verify if RabbitMQ is supported on that Erlang version. Thanks! On Tue, Sep 8, 2015 at 2:05 PM, Tuncer Ayaz wrote: > On Tue, Sep 8, 2015 at 1:58 PM, John Foldager wrote: >> We're using RabbitMQ and now have a request for supporting the >> following cipher suites: >> >> TLS_RSA_WITH_AES_128_GCM_SHA256 (0X009C) >> TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0XC02F) >> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0X009E) >> TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0XC02B) >> >> However, if we use the following commands from the Erlang CLI we >> don't see these cipher suites: >> >> io:format("~p", [ssl:cipher_suites(openssl)]). >> io:format("~p", [ssl:cipher_suites(erlang)]). >> >> So how can we make these cipher suites available to Erlang.... and >> then RabbitMQ that runs on top of Erlang? > > To rule out the obvious and assuming there's no difference in the > OpenSSL install between the build and deploy host, does openssl > list it as available? > > $ openssl ciphers -v|grep AES128-GCM-SHA256 From john.foldager@REDACTED Wed Sep 9 09:39:52 2015 From: john.foldager@REDACTED (John Foldager) Date: Wed, 9 Sep 2015 09:39:52 +0200 Subject: [erlang-questions] TLS cipher suite with Galois Counter Mode (GCM) In-Reply-To: References: Message-ID: Thanks Danil. We're currently using Erlang 16 b3, so we'll check if we can upgrade to Erlang OTP 18 instead. We just need to verify with RabbitMQ if it is supported or not. Thanks On Tue, Sep 8, 2015 at 2:38 PM, Danil Zagoskin wrote: > Hi, John! > > What OTP version do you use? > OTP17 and prior do not support GCM, you need OTP18. > > Old possible ciphers: > https://github.com/erlang/otp/blob/maint-17/lib/ssl/src/ssl_cipher.erl#L46 > New possible ciphers: > https://github.com/erlang/otp/blob/maint-18/lib/ssl/src/ssl_cipher.erl#L48 > (note the aes_128_gcm and aes_256_gcm ciphers). > > On Tue, Sep 8, 2015 at 2:58 PM, John Foldager > wrote: >> >> We're using RabbitMQ and now have a request for supporting the following >> cipher suites: >> >> TLS_RSA_WITH_AES_128_GCM_SHA256 (0X009C) >> TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0XC02F) >> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0X009E) >> TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0XC02B) >> >> However, if we use the following commands from the Erlang CLI we don't see >> these cipher suites: >> >> io:format("~p", [ssl:cipher_suites(openssl)]). >> io:format("~p", [ssl:cipher_suites(erlang)]). >> >> So how can we make these cipher suites available to Erlang.... and then >> RabbitMQ that runs on top of Erlang? >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > Danil Zagoskin | z@REDACTED From jean-sebastien.pedron@REDACTED Wed Sep 9 10:54:14 2015 From: jean-sebastien.pedron@REDACTED (=?UTF-8?Q?Jean-S=c3=a9bastien_P=c3=a9dron?=) Date: Wed, 9 Sep 2015 10:54:14 +0200 Subject: [erlang-questions] TLS cipher suite with Galois Counter Mode (GCM) In-Reply-To: References: Message-ID: <55EFF3B6.1040107@dumbbell.fr> On 09.09.2015 09:37, John Foldager wrote: > As Danil mentioned we need to upgrade Erlang to OTP 18, but then we > must first verify if RabbitMQ is supported on that Erlang version. Hi John! We (the RabbitMQ team) did not receive any failure report about RabbitMQ on Erlang 18. Furthermore, I use this version when working on RabbitMQ and we have CI jobs running with it as well. So I expect RabbitMQ to be fine with Erlang 18. -- Jean-S?bastien P?dron -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 949 bytes Desc: OpenPGP digital signature URL: From roger@REDACTED Wed Sep 9 11:08:31 2015 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 9 Sep 2015 10:08:31 +0100 Subject: [erlang-questions] Deploying forked Erlang releases? Message-ID: We currently deploy Erlang to our production servers by using the Erlang Solutions .deb files. However, it looks like we need to deploy a custom build of Erlang 17.5, at least until https://github.com/erlang/otp/pull/824 is merged upstream. Questions: 1. What build options do the ESL packages use? 2. Are the ESL (or other) scripts for building the .deb file available? 3. Failing that, how are other people deploying Erlang? Do you just unpack a tarball into /? From john.foldager@REDACTED Wed Sep 9 11:08:18 2015 From: john.foldager@REDACTED (John Foldager) Date: Wed, 9 Sep 2015 11:08:18 +0200 Subject: [erlang-questions] TLS cipher suite with Galois Counter Mode (GCM) In-Reply-To: <55EFF3B6.1040107@dumbbell.fr> References: <55EFF3B6.1040107@dumbbell.fr> Message-ID: Hi Jean-S?bastien, thanks. So no official support for Erlang OTP 18 yet, but the RabbitMQ team will be helping out if we get into trouble with the combination of Erlang OTP 18 and latest RabbitMQ 3.5.4 (newest as of today) ? On Wed, Sep 9, 2015 at 10:54 AM, Jean-S?bastien P?dron wrote: > On 09.09.2015 09:37, John Foldager wrote: >> As Danil mentioned we need to upgrade Erlang to OTP 18, but then we >> must first verify if RabbitMQ is supported on that Erlang version. > > Hi John! > > We (the RabbitMQ team) did not receive any failure report about RabbitMQ > on Erlang 18. Furthermore, I use this version when working on RabbitMQ > and we have CI jobs running with it as well. > > So I expect RabbitMQ to be fine with Erlang 18. > > -- > Jean-S?bastien P?dron > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From aschultz@REDACTED Wed Sep 9 11:23:37 2015 From: aschultz@REDACTED (Andreas Schultz) Date: Wed, 9 Sep 2015 11:23:37 +0200 (CEST) Subject: [erlang-questions] TLS cipher suite with Galois Counter Mode (GCM) In-Reply-To: References: Message-ID: <1468286756.3609392.1441790617301.JavaMail.zimbra@tpip.net> ----- Original Message ----- > From: "John Foldager" > To: erlang-questions@REDACTED > Sent: Wednesday, September 9, 2015 9:39:52 AM > Subject: Re: [erlang-questions] TLS cipher suite with Galois Counter Mode (GCM) > Thanks Danil. > > We're currently using Erlang 16 b3, so we'll check if we can upgrade > to Erlang OTP 18 instead. We just need to verify with RabbitMQ if it > is supported or not. Hi John, Original pull request for GCM support was https://github.com/erlang/otp/pull/372. Those change *might* be applicable to R17 as well (if you need to stay with R17). R16 is most certainly to old for those changes. Andreas > Thanks > > On Tue, Sep 8, 2015 at 2:38 PM, Danil Zagoskin wrote: >> Hi, John! >> >> What OTP version do you use? >> OTP17 and prior do not support GCM, you need OTP18. >> >> Old possible ciphers: >> https://github.com/erlang/otp/blob/maint-17/lib/ssl/src/ssl_cipher.erl#L46 >> New possible ciphers: >> https://github.com/erlang/otp/blob/maint-18/lib/ssl/src/ssl_cipher.erl#L48 >> (note the aes_128_gcm and aes_256_gcm ciphers). >> >> On Tue, Sep 8, 2015 at 2:58 PM, John Foldager >> wrote: >>> >>> We're using RabbitMQ and now have a request for supporting the following >>> cipher suites: >>> >>> TLS_RSA_WITH_AES_128_GCM_SHA256 (0X009C) >>> TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0XC02F) >>> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0X009E) >>> TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0XC02B) >>> >>> However, if we use the following commands from the Erlang CLI we don't see >>> these cipher suites: >>> >>> io:format("~p", [ssl:cipher_suites(openssl)]). >>> io:format("~p", [ssl:cipher_suites(erlang)]). >>> >>> So how can we make these cipher suites available to Erlang.... and then >>> RabbitMQ that runs on top of Erlang? >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> >> -- >> Danil Zagoskin | z@REDACTED > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From john.foldager@REDACTED Wed Sep 9 11:38:46 2015 From: john.foldager@REDACTED (John Foldager) Date: Wed, 9 Sep 2015 11:38:46 +0200 Subject: [erlang-questions] TLS cipher suite with Galois Counter Mode (GCM) In-Reply-To: <1468286756.3609392.1441790617301.JavaMail.zimbra@tpip.net> References: <1468286756.3609392.1441790617301.JavaMail.zimbra@tpip.net> Message-ID: We'll probably upgrade directly from Erlang 16.x to 18.x, but we need to know the following: - Will RabbitMQ team be supportive if issues arise with the combination of Erlang 18.x and RabbitMQ 3.5.4+ ? - Will RabbitMQ be able to handle the GCM ciphers? My guess is yes because I would believe that RabbitMQ is only using the Erlang SSL/TLS layer and not restricting anything in it if it is supported. On Wed, Sep 9, 2015 at 11:23 AM, Andreas Schultz wrote: > ----- Original Message ----- >> From: "John Foldager" >> To: erlang-questions@REDACTED >> Sent: Wednesday, September 9, 2015 9:39:52 AM >> Subject: Re: [erlang-questions] TLS cipher suite with Galois Counter Mode (GCM) > >> Thanks Danil. >> >> We're currently using Erlang 16 b3, so we'll check if we can upgrade >> to Erlang OTP 18 instead. We just need to verify with RabbitMQ if it >> is supported or not. > > Hi John, > > Original pull request for GCM support was https://github.com/erlang/otp/pull/372. > Those change *might* be applicable to R17 as well (if you need to stay with R17). > > R16 is most certainly to old for those changes. > > Andreas > > >> Thanks >> >> On Tue, Sep 8, 2015 at 2:38 PM, Danil Zagoskin wrote: >>> Hi, John! >>> >>> What OTP version do you use? >>> OTP17 and prior do not support GCM, you need OTP18. >>> >>> Old possible ciphers: >>> https://github.com/erlang/otp/blob/maint-17/lib/ssl/src/ssl_cipher.erl#L46 >>> New possible ciphers: >>> https://github.com/erlang/otp/blob/maint-18/lib/ssl/src/ssl_cipher.erl#L48 >>> (note the aes_128_gcm and aes_256_gcm ciphers). >>> >>> On Tue, Sep 8, 2015 at 2:58 PM, John Foldager >>> wrote: >>>> >>>> We're using RabbitMQ and now have a request for supporting the following >>>> cipher suites: >>>> >>>> TLS_RSA_WITH_AES_128_GCM_SHA256 (0X009C) >>>> TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0XC02F) >>>> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0X009E) >>>> TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0XC02B) >>>> >>>> However, if we use the following commands from the Erlang CLI we don't see >>>> these cipher suites: >>>> >>>> io:format("~p", [ssl:cipher_suites(openssl)]). >>>> io:format("~p", [ssl:cipher_suites(erlang)]). >>>> >>>> So how can we make these cipher suites available to Erlang.... and then >>>> RabbitMQ that runs on top of Erlang? >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >>> >>> >>> -- >>> Danil Zagoskin | z@REDACTED >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From jesper.louis.andersen@REDACTED Wed Sep 9 12:39:58 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 9 Sep 2015 12:39:58 +0200 Subject: [erlang-questions] Deploying forked Erlang releases? In-Reply-To: References: Message-ID: On Wed, Sep 9, 2015 at 11:08 AM, Roger Lipscombe wrote: > 3. Failing that, how are other people deploying Erlang? Do you just > unpack a tarball into /? > Releases. A release packs an ERTS together with your code, so it is self-contained. You then don't have to deploy an Erlang package next to your code. Whatever ERTS the build-host uses is the one that is packaged into the release. This solves the problem of having a locally patched Erlang. For building Erlang releases, anything now goes. We use kerl, but we don't require special patches on top of Erlang, just some compiler flags (--disable-hipe, --enable-dirty-schedulers,--with-dynamic-trace=dtrace). However, this is easily adaptable to a patched world with the (experimental) kerl git option. We often simply deploy these as tarballs which we unpack somewhere (/opt/foo or /usr/local/foo depending on OS type). Then we install some start-script for them and we are good to go. I spent much time trying to get packaging right in $OS, but recently, I've had more success with the above low-level method. In a world where it is easy to "roll a new machine", you can just use that machine as a starting point. You don't have to worry too much about being able to remove software from the system again, which is one of the primary reasons for the existence of package managers. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From marc@REDACTED Wed Sep 9 13:20:21 2015 From: marc@REDACTED (Marc Worrell) Date: Wed, 9 Sep 2015 13:20:21 +0200 Subject: [erlang-questions] [ANN] Zotonic release 0.13.3 Message-ID: <5E1E83EC-EB68-4996-8E12-8441CDF1E82F@worrell.nl> Zotonic is the Erlang Content Management System and Framework. We have just released version 0.13.3. This is a maintenance release of Zotonic 0.13 Main changes are: ? refinements to the admin interface by Arthur Clemens ? included Bootstrap version has been upgraded to version 3.3.5 ? fixes for some issues around the newly introduced content groups ? CORS support for the API (Thanks to @ghosthamlet!) ? fixes some issues with the websocket connection ? fixes an issue where the wrong language could be selected You can find more about Zotonic on: http://zotonic.com/ The release can be downloaded from: https://github.com/zotonic/zotonic/releases/tag/release-0.13.3 And the complete Zotonic docs are here: http://zotonic.com/docs/latest/ Kind Regards, Marc Worrell Zotonic Core Team From spawn.think@REDACTED Wed Sep 9 14:19:16 2015 From: spawn.think@REDACTED (Ahmed Omar) Date: Wed, 9 Sep 2015 14:19:16 +0200 Subject: [erlang-questions] segfault erts-5.10.4 (R16B03-1) In-Reply-To: References: Message-ID: Hi Lukas, You were right, there was a mismatch between beam.smp on development and production (unfortunately, different build servers...) I ran the session now with the right beam.smp ### [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/var/lib/ejabberd/erts-5.10.4/bin/beam.smp -K true -A 128 -P 2500000 -Q 500000'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x000000000044d299 in bf_get_free_block () [Current thread is 1 (Thread 0x7fe077fbc700 (LWP 24141))] (gdb) bt full #0 0x000000000044d299 in bf_get_free_block () No symbol table info available. #1 0x0000000000442fa6 in mbc_alloc () No symbol table info available. #2 0x0000000000448803 in erts_alcu_alloc_thr_pref () No symbol table info available. #3 0x0000000000509508 in erts_alloc () No symbol table info available. #4 0x00000000005096de in erts_bs_append () No symbol table info available. #5 0x000000000053c1f8 in process_main () No symbol table info available. #6 0x000000000049da8b in sched_thread_func () No symbol table info available. #7 0x00000000005bd146 in thr_wrapper () No symbol table info available. #8 0x00000039a80079d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #9 0x00000039a78e88fd in clone () from /lib64/libc.so.6 No symbol table info available. ### I'll keep digging ... Best Regards, - Ahmed Omar http://about.me/spawn.think/ On Tue, Sep 8, 2015 at 4:45 PM, Lukas Larsson wrote: > > > On Tue, Sep 8, 2015 at 4:34 PM, Ahmed Omar wrote: > >> Ok. I guess etp-ports require a live session then? >> (gdb) etp-ports >> No ports, since system isn't initialised! >> > >> Is there a way to get port or process number from the information I got >> from bt ? >> #2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, >> size=, busy_pcrr_pp=0x8, alcu_flgs=0) >> >> >> > All etp macros work on both core files a live nodes. The only time I've > seen that error from etp-ports is when the VM crashed before the internal > ports array was initialized. > > I'm starting to suspect that there is something wrong with your debugging > session. The stack trace that you have printed is corrupt, and it cannot > find a working port table. Make extra sure that you are using the same > beam.smp to run your gdb session from as generated the core. > > Since the stack is corrupt it is not possible to know what called into the > realloc from the stack trace. > > Lukas > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From garazdawi@REDACTED Wed Sep 9 14:52:15 2015 From: garazdawi@REDACTED (Lukas Larsson) Date: Wed, 9 Sep 2015 14:52:15 +0200 Subject: [erlang-questions] segfault erts-5.10.4 (R16B03-1) In-Reply-To: References: Message-ID: On Wed, Sep 9, 2015 at 2:19 PM, Ahmed Omar wrote: > ### > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `/var/lib/ejabberd/erts-5.10.4/bin/beam.smp -K true > -A 128 -P 2500000 -Q 500000'. > Program terminated with signal SIGSEGV, Segmentation fault. > #0 0x000000000044d299 in bf_get_free_block () > [Current thread is 1 (Thread 0x7fe077fbc700 (LWP 24141))] > (gdb) bt full > #0 0x000000000044d299 in bf_get_free_block () > No symbol table info available. > #1 0x0000000000442fa6 in mbc_alloc () > No symbol table info available. > #2 0x0000000000448803 in erts_alcu_alloc_thr_pref () > No symbol table info available. > #3 0x0000000000509508 in erts_alloc () > No symbol table info available. > #4 0x00000000005096de in erts_bs_append () > No symbol table info available. > #5 0x000000000053c1f8 in process_main () > No symbol table info available. > #6 0x000000000049da8b in sched_thread_func () > No symbol table info available. > #7 0x00000000005bd146 in thr_wrapper () > No symbol table info available. > #8 0x00000039a80079d1 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #9 0x00000039a78e88fd in clone () from /lib64/libc.so.6 > No symbol table info available. > ### > > I'll keep digging ... > > It looks like you have a memory corruption issue. Normally when something goes wrong in the memory allocators, it is because a badly written nif/linked-in driver has written outside the buffer it requested by calling driver/enif_alloc. I would take a deep long look at any native code that you have running and see if you can find any such bugs. If you can reproduce the error, I would attempt to run the same scenario in valgrind as it is quite good at spotting these types of errors. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Wed Sep 9 16:10:14 2015 From: t@REDACTED (Tristan Sloughter) Date: Wed, 09 Sep 2015 09:10:14 -0500 Subject: [erlang-questions] Deploying forked Erlang releases? In-Reply-To: References: Message-ID: <1441807814.371889.378856505.200F5CD3@webmail.messagingengine.com> Right, no need to install Erlang itself on a server. Better to build your release into a tarball (including erts makes it a target system) which you can then unpack and run "anywhere" (since it contains erts it has to be the same platform). relx (which rebar3 uses and can be used standalone) generates a useful start script (set {extended_start_script, true} in the config of your project) with options for starting, stopping, getting a remote console, upgrading the release, and more. And unless you need special options you can have the tarball built even somewhere like travis. We have travis-ci build the tarball when a tag is pushed in github and upload it to s3. Then our deployment tools can fetch a version from s3, unpack it and start (or if it is doing an upgrade, it downloads the tarball and runs the start scripts 'upgrade' command). But since you need a custom build that isn't an option :), but if you have some other CI server the same can be done and you only have to install your custom Erlang build on it and dev machines instead of the servers. -- Tristan Sloughter t@REDACTED On Wed, Sep 9, 2015, at 05:39 AM, Jesper Louis Andersen wrote: > > On Wed, Sep 9, 2015 at 11:08 AM, Roger Lipscombe wrote: >> 3. Failing that, how are other people deploying Erlang? Do you just >> unpack a tarball into /? > Releases. > A release packs an ERTS together with your code, so it is self-contained. You then don't have to deploy an Erlang package next to your code. Whatever ERTS the build-host uses is the one that is packaged into the release. This solves the problem of having a locally patched Erlang. > For building Erlang releases, anything now goes. We use kerl, but we don't require special patches on top of Erlang, just some compiler flags (--disable-hipe, --enable-dirty-schedulers,--with-dynamic-trace=dtrace). However, this is easily adaptable to a patched world with the (experimental) kerl git option. > > We often simply deploy these as tarballs which we unpack somewhere (/opt/foo or /usr/local/foo depending on OS type). Then we install some start-script for them and we are good to go. > I spent much time trying to get packaging right in $OS, but recently, I've had more success with the above low-level method. In a world where it is easy to "roll a new machine", you can just use that machine as a starting point. You don't have to worry too much about being able to remove software from the system again, which is one of the primary reasons for the existence of package managers. > > > -- > J. > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From james@REDACTED Wed Sep 9 16:22:42 2015 From: james@REDACTED (James Fish) Date: Wed, 9 Sep 2015 15:22:42 +0100 Subject: [erlang-questions] Deploying forked Erlang releases? In-Reply-To: <1441807814.371889.378856505.200F5CD3@webmail.messagingengine.com> References: <1441807814.371889.378856505.200F5CD3@webmail.messagingengine.com> Message-ID: Not directly an answer to your question but if the raw options are applicable for all listen sockets you can use the kernel config `inet_default_listen_options` to set default (raw) options for the underlying tcp socket. On 9 September 2015 at 15:10, Tristan Sloughter wrote: > Right, no need to install Erlang itself on a server. Better to build your > release into a tarball (including erts makes it a target system) which you > can then unpack and run "anywhere" (since it contains erts it has to be the > same platform). > > relx (which rebar3 uses and can be used standalone) generates a useful > start script (set {extended_start_script, true} in the config of your > project) with options for starting, stopping, getting a remote console, > upgrading the release, and more. > > And unless you need special options you can have the tarball built even > somewhere like travis. We have travis-ci build the tarball when a tag is > pushed in github and upload it to s3. Then our deployment tools can fetch a > version from s3, unpack it and start (or if it is doing an upgrade, it > downloads the tarball and runs the start scripts 'upgrade' command). > > But since you need a custom build that isn't an option :), but if you have > some other CI server the same can be done and you only have to install your > custom Erlang build on it and dev machines instead of the servers. > > -- > Tristan Sloughter > t@REDACTED > > > > On Wed, Sep 9, 2015, at 05:39 AM, Jesper Louis Andersen wrote: > > > On Wed, Sep 9, 2015 at 11:08 AM, Roger Lipscombe > wrote: > > 3. Failing that, how are other people deploying Erlang? Do you just > unpack a tarball into /? > > Releases. > A release packs an ERTS together with your code, so it is self-contained. > You then don't have to deploy an Erlang package next to your code. Whatever > ERTS the build-host uses is the one that is packaged into the release. This > solves the problem of having a locally patched Erlang. > For building Erlang releases, anything now goes. We use kerl, but we don't > require special patches on top of Erlang, just some compiler flags > (--disable-hipe, --enable-dirty-schedulers,--with-dynamic-trace=dtrace). > However, this is easily adaptable to a patched world with the > (experimental) kerl git option. > > We often simply deploy these as tarballs which we unpack somewhere > (/opt/foo or /usr/local/foo depending on OS type). Then we install some > start-script for them and we are good to go. > I spent much time trying to get packaging right in $OS, but recently, I've > had more success with the above low-level method. In a world where it is > easy to "roll a new machine", you can just use that machine as a starting > point. You don't have to worry too much about being able to remove software > from the system again, which is one of the primary reasons for the > existence of package managers. > > > -- > J. > *_______________________________________________* > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From garcia.narbona@REDACTED Wed Sep 9 19:56:17 2015 From: garcia.narbona@REDACTED (=?UTF-8?Q?David_Garc=C3=ADa?=) Date: Wed, 9 Sep 2015 19:56:17 +0200 Subject: [erlang-questions] How combine WSDL along with XSD using erlsom + yaws_soap_lib Message-ID: Would it be possible to combine WSDL along with XSD file when using yaws_soap_lib? I have already tried importing XSD from WSDL, as well as copying the content of the XSD into the WSDL. Apparently there's a problem with erlsom dealing with sequence definitions in the XSD file: I get "*Group definition not found*" for the sequence elements, because it looks like erlsom uses a default prefix ("P:") for creating internal structure, but it does not use the prefix in the sequence erlsom internal structure, then it can not link it. I have already tried compiling and adding the XSD model to the WSDL model using erlsom API methods, but I didn't succeed to get it working. Any clue? Thanks in advance. -- David -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Thu Sep 10 02:16:28 2015 From: vasdeveloper@REDACTED (Theepan) Date: Thu, 10 Sep 2015 05:46:28 +0530 Subject: [erlang-questions] Deploying forked Erlang releases? In-Reply-To: References: <1441807814.371889.378856505.200F5CD3@webmail.messagingengine.com> Message-ID: Hi Roger, The above posts explain why an Erlang install is not required in the target server, here I explain how to create a release package. It is little messy, but if you get used to it, it will be very easy. You may want to create a script to do all the steps, but I am listing the steps, so that it may be helpful for some one in the future. ? In your build server, you will have to make install Erlang (under /usr/local/lib/) and your custom applications (under /usr/local/lib/erlang/lib/). I assume *X platforms. ? Then you create the release file (*.rel file) ? Create a configuration file (say sys.conf) for all the custom application you have included. You take the environment variables from your *.app file. It may contain some configurations for kernel and other standard apps as well, depending on your requirements. It has some conventions to follow, please look at the link provided below. ? Crate your release package using the code target_system.erl, which you can find in the link below. ? Unpack it on any compatible platform. ? Some times you will have to mark erl, and to_erl files as executable in some platforms, from the unpacked directories. ? Write your start/stop script. Here you will have to give the path where you unpacked, configuration file (sys.conf), and other parameters like node name, cookie (if you are running distributed systems), whether you want to run the node in the background, whether Ctrl+C can kill the node, whether you want to bind the schedulers to cores, mnesia directory (if you are using Mnesia) and there are plenty more options. Please check documentation for erl. ? You will have to carefully manage the Mnesia database. Create it into your code, or launching an empty node from your package using the a script and create it manually. Please follow the link below for details. It is pretty low level, but if you do it once, it will be very easy. http://www.erlang.org/doc/system_principles/create_target.html Hope this helps. Theepan On Wed, Sep 9, 2015 at 7:52 PM, James Fish wrote: > Not directly an answer to your question but if the raw options are > applicable for all listen sockets you can use the kernel config > `inet_default_listen_options` to set default (raw) options for the > underlying tcp socket. > > On 9 September 2015 at 15:10, Tristan Sloughter wrote: > >> Right, no need to install Erlang itself on a server. Better to build your >> release into a tarball (including erts makes it a target system) which you >> can then unpack and run "anywhere" (since it contains erts it has to be the >> same platform). >> >> relx (which rebar3 uses and can be used standalone) generates a useful >> start script (set {extended_start_script, true} in the config of your >> project) with options for starting, stopping, getting a remote console, >> upgrading the release, and more. >> >> And unless you need special options you can have the tarball built even >> somewhere like travis. We have travis-ci build the tarball when a tag is >> pushed in github and upload it to s3. Then our deployment tools can fetch a >> version from s3, unpack it and start (or if it is doing an upgrade, it >> downloads the tarball and runs the start scripts 'upgrade' command). >> >> But since you need a custom build that isn't an option :), but if you >> have some other CI server the same can be done and you only have to install >> your custom Erlang build on it and dev machines instead of the servers. >> >> -- >> Tristan Sloughter >> t@REDACTED >> >> >> >> On Wed, Sep 9, 2015, at 05:39 AM, Jesper Louis Andersen wrote: >> >> >> On Wed, Sep 9, 2015 at 11:08 AM, Roger Lipscombe >> wrote: >> >> 3. Failing that, how are other people deploying Erlang? Do you just >> unpack a tarball into /? >> >> Releases. >> A release packs an ERTS together with your code, so it is self-contained. >> You then don't have to deploy an Erlang package next to your code. Whatever >> ERTS the build-host uses is the one that is packaged into the release. This >> solves the problem of having a locally patched Erlang. >> For building Erlang releases, anything now goes. We use kerl, but we >> don't require special patches on top of Erlang, just some compiler flags >> (--disable-hipe, --enable-dirty-schedulers,--with-dynamic-trace=dtrace). >> However, this is easily adaptable to a patched world with the >> (experimental) kerl git option. >> >> We often simply deploy these as tarballs which we unpack somewhere >> (/opt/foo or /usr/local/foo depending on OS type). Then we install some >> start-script for them and we are good to go. >> I spent much time trying to get packaging right in $OS, but recently, >> I've had more success with the above low-level method. In a world where it >> is easy to "roll a new machine", you can just use that machine as a >> starting point. You don't have to worry too much about being able to remove >> software from the system again, which is one of the primary reasons for the >> existence of package managers. >> >> >> -- >> J. >> *_______________________________________________* >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From buaatianwanli@REDACTED Thu Sep 10 06:54:14 2015 From: buaatianwanli@REDACTED (Fredo Lee) Date: Thu, 10 Sep 2015 12:54:14 +0800 Subject: [erlang-questions] compile erlang source code error [obj/x86_64-unknown-linux-gnu/opt/smp/inet_drv.o] Message-ID: make[1]: Entering directory `/home/users/tianwanli01/otp_src_18.0/erts' make[2]: Entering directory `/home/users/tianwanli01/otp_src_18.0/erts/emulator' MAKE opt make[3]: Entering directory `/home/users/tianwanli01/otp_src_18.0/erts/emulator' GEN x86_64-unknown-linux-gnu/gen_git_version.mk CC obj/x86_64-unknown-linux-gnu/opt/smp/inet_drv.o drivers/common/inet_drv.c: In function `inet_init': drivers/common/inet_drv.c:4031: error: enumerator value for `compile_time_assert__' not integer constant drivers/common/inet_drv.c: In function `sctp_get_sendparams': drivers/common/inet_drv.c:6537: warning: assignment makes pointer from integer without a cast drivers/common/inet_drv.c: In function `sctp_set_opts': drivers/common/inet_drv.c:6651: warning: assignment makes pointer from integer without a cast drivers/common/inet_drv.c:6666: warning: assignment makes pointer from integer without a cast drivers/common/inet_drv.c:6849: warning: assignment makes pointer from integer without a cast drivers/common/inet_drv.c:6895: warning: assignment makes pointer from integer without a cast drivers/common/inet_drv.c: In function `sctp_fill_opts': drivers/common/inet_drv.c:7594: warning: assignment makes pointer from integer without a cast drivers/common/inet_drv.c:7620: warning: assignment makes pointer from integer without a cast drivers/common/inet_drv.c:7839: warning: assignment makes pointer from integer without a cast drivers/common/inet_drv.c:7895: warning: assignment makes pointer from integer without a cast drivers/common/inet_drv.c:7983: warning: assignment makes pointer from integer without a cast drivers/common/inet_drv.c:8055: warning: assignment makes pointer from integer without a cast drivers/common/inet_drv.c:8135: warning: assignment makes pointer from integer without a cast drivers/common/inet_drv.c: In function `inet_ctl': drivers/common/inet_drv.c:8622: warning: passing arg 2 of pointer to function makes pointer from integer without a cast drivers/common/inet_drv.c:8691: warning: passing arg 2 of pointer to function makes pointer from integer without a cast drivers/common/inet_drv.c: In function `packet_inet_ctl': drivers/common/inet_drv.c:11615: warning: passing arg 2 of pointer to function makes pointer from integer without a cast make[3]: *** [obj/x86_64-unknown-linux-gnu/opt/smp/inet_drv.o] Error 1 make[3]: Leaving directory `/home/users/tianwanli01/otp_src_18.0/erts/emulator' make[2]: *** [opt] Error 2 make[2]: Leaving directory `/home/users/tianwanli01/otp_src_18.0/erts/emulator' make[1]: *** [smp] Error 2 make[1]: Leaving directory `/home/users/tianwanli01/otp_src_18.0/erts' make: *** [emulator] Error 2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From w.a.de.jong@REDACTED Thu Sep 10 08:04:38 2015 From: w.a.de.jong@REDACTED (Willem de Jong) Date: Thu, 10 Sep 2015 08:04:38 +0200 Subject: [erlang-questions] How combine WSDL along with XSD using erlsom + yaws_soap_lib In-Reply-To: References: Message-ID: Hello David, Yes, it should be possible to use yaws_soap_lib with a WSDL and an XSD, either by importing the XSD or by embedding it in the WSDL. At first sight I would say that the definition of ClassA is valid and should be accepted by erlsom. Can you give a bit more information about what you are doing? Perhaps send me the WSDL and XSD and the commands that you have been trying. Regards, Willem On Wed, Sep 9, 2015 at 7:56 PM, David Garc?a wrote: > > Would it be possible to combine WSDL along with XSD file when using > yaws_soap_lib? > I have already tried importing XSD from WSDL, as well as copying the > content of the XSD into the WSDL. > Apparently there's a problem with erlsom dealing with sequence definitions > in the XSD file: > > > > > > > > > > > > > > > I get "*Group definition not found*" for the sequence elements, because > it looks like erlsom uses a default prefix ("P:") for creating internal > structure, but it does not use the prefix in the sequence erlsom internal > structure, then it can not link it. > > I have already tried compiling and adding the XSD model to the WSDL model > using erlsom API methods, but I didn't succeed to get it working. > > Any clue? > > Thanks in advance. > > -- > David > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Thu Sep 10 08:47:34 2015 From: zxq9@REDACTED (zxq9) Date: Thu, 10 Sep 2015 15:47:34 +0900 Subject: [erlang-questions] GCM block ciphers in the Windows binary distribution Message-ID: <2354853.leQ8EZiBBD@burrito> I just noticed that there are no GCM ciphers supported in the Windows binary distribution from Erlang Solutions (at least not R18). Is there any reason for this? On Windows: 1> io:format("~p~n", [ssl:cipher_suites()]). [{ecdhe_ecdsa,aes_256_cbc,sha384}, {ecdhe_rsa,aes_256_cbc,sha384}, {ecdh_ecdsa,aes_256_cbc,sha384}, {ecdh_rsa,aes_256_cbc,sha384}, {dhe_rsa,aes_256_cbc,sha256}, {dhe_dss,aes_256_cbc,sha256}, {rsa,aes_256_cbc,sha256}, {ecdhe_ecdsa,aes_128_cbc,sha256}, {ecdhe_rsa,aes_128_cbc,sha256}, {ecdh_ecdsa,aes_128_cbc,sha256}, {ecdh_rsa,aes_128_cbc,sha256}, {dhe_rsa,aes_128_cbc,sha256}, {dhe_dss,aes_128_cbc,sha256}, {rsa,aes_128_cbc,sha256}, {ecdhe_ecdsa,aes_256_cbc,sha}, {ecdhe_rsa,aes_256_cbc,sha}, {dhe_rsa,aes_256_cbc,sha}, {dhe_dss,aes_256_cbc,sha}, {ecdh_ecdsa,aes_256_cbc,sha}, {ecdh_rsa,aes_256_cbc,sha}, {rsa,aes_256_cbc,sha}, {ecdhe_ecdsa,'3des_ede_cbc',sha}, {ecdhe_rsa,'3des_ede_cbc',sha}, {dhe_rsa,'3des_ede_cbc',sha}, {dhe_dss,'3des_ede_cbc',sha}, {ecdh_ecdsa,'3des_ede_cbc',sha}, {ecdh_rsa,'3des_ede_cbc',sha}, {rsa,'3des_ede_cbc',sha}, {ecdhe_ecdsa,aes_128_cbc,sha}, {ecdhe_rsa,aes_128_cbc,sha}, {dhe_rsa,aes_128_cbc,sha}, {dhe_dss,aes_128_cbc,sha}, {ecdh_ecdsa,aes_128_cbc,sha}, {ecdh_rsa,aes_128_cbc,sha}, {rsa,aes_128_cbc,sha}, {dhe_rsa,des_cbc,sha}, {rsa,des_cbc,sha}] And checking OpenSSL's support... C:\Users\Craig Everett>openssl ciphers WARNING: can't open config file: /etc/ssl/openssl.cnf ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:SRP-DSS-AES-256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA:PSK-AES256-CBC-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:SRP-DSS-AES-128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:IDEA-CBC-SHA:PSK-AES128-CBC-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:PSK-RC4-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:SRP-DSS-3DES-EDE-CBC-SHA:SRP-RSA-3DES-EDE-CBC-SHA:SRP-3DES-EDE-CBC-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:DES-CBC3-SHA:PSK-3DES-EDE-CBC-SHA:EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA:DES-CBC-SHA Any reason these are not built in? "Its a PITA" is a perfectly acceptable answer... the Windows build process already looks convoluted. I was just surprised as it doesn't appear to be difficult to add this since the included OpenSSL already supports them. -Craig From aschultz@REDACTED Thu Sep 10 10:24:05 2015 From: aschultz@REDACTED (Andreas Schultz) Date: Thu, 10 Sep 2015 10:24:05 +0200 (CEST) Subject: [erlang-questions] GCM block ciphers in the Windows binary distribution In-Reply-To: <2354853.leQ8EZiBBD@burrito> References: <2354853.leQ8EZiBBD@burrito> Message-ID: <846452798.3619692.1441873445415.JavaMail.zimbra@tpip.net> ----- Original Message ----- > From: "zxq9" > To: erlang-questions@REDACTED > Sent: Thursday, September 10, 2015 8:47:34 AM > Subject: [erlang-questions] GCM block ciphers in the Windows binary distribution > I just noticed that there are no GCM ciphers supported in the Windows binary > distribution from Erlang Solutions (at least not R18). Is there any reason for > this? AES-GCM requires OpenSSL >= 1.0.1, the binary distribution was probably build against an older version. Andreas > > On Windows: > > 1> io:format("~p~n", [ssl:cipher_suites()]). > [{ecdhe_ecdsa,aes_256_cbc,sha384}, > {ecdhe_rsa,aes_256_cbc,sha384}, > {ecdh_ecdsa,aes_256_cbc,sha384}, > {ecdh_rsa,aes_256_cbc,sha384}, > {dhe_rsa,aes_256_cbc,sha256}, > {dhe_dss,aes_256_cbc,sha256}, > {rsa,aes_256_cbc,sha256}, > {ecdhe_ecdsa,aes_128_cbc,sha256}, > {ecdhe_rsa,aes_128_cbc,sha256}, > {ecdh_ecdsa,aes_128_cbc,sha256}, > {ecdh_rsa,aes_128_cbc,sha256}, > {dhe_rsa,aes_128_cbc,sha256}, > {dhe_dss,aes_128_cbc,sha256}, > {rsa,aes_128_cbc,sha256}, > {ecdhe_ecdsa,aes_256_cbc,sha}, > {ecdhe_rsa,aes_256_cbc,sha}, > {dhe_rsa,aes_256_cbc,sha}, > {dhe_dss,aes_256_cbc,sha}, > {ecdh_ecdsa,aes_256_cbc,sha}, > {ecdh_rsa,aes_256_cbc,sha}, > {rsa,aes_256_cbc,sha}, > {ecdhe_ecdsa,'3des_ede_cbc',sha}, > {ecdhe_rsa,'3des_ede_cbc',sha}, > {dhe_rsa,'3des_ede_cbc',sha}, > {dhe_dss,'3des_ede_cbc',sha}, > {ecdh_ecdsa,'3des_ede_cbc',sha}, > {ecdh_rsa,'3des_ede_cbc',sha}, > {rsa,'3des_ede_cbc',sha}, > {ecdhe_ecdsa,aes_128_cbc,sha}, > {ecdhe_rsa,aes_128_cbc,sha}, > {dhe_rsa,aes_128_cbc,sha}, > {dhe_dss,aes_128_cbc,sha}, > {ecdh_ecdsa,aes_128_cbc,sha}, > {ecdh_rsa,aes_128_cbc,sha}, > {rsa,aes_128_cbc,sha}, > {dhe_rsa,des_cbc,sha}, > {rsa,des_cbc,sha}] > > And checking OpenSSL's support... > > C:\Users\Craig Everett>openssl ciphers > WARNING: can't open config file: /etc/ssl/openssl.cnf > ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:SRP-DSS-AES-256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA:PSK-AES256-CBC-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:SRP-DSS-AES-128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DHE-RSA-SE > ED-SHA:DHE-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:IDEA-CBC-SHA:PSK-AES128-CBC-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:PSK-RC4-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:SRP-DSS-3DES-EDE-CBC-SHA:SRP-RSA-3DES-EDE-CBC-SHA:SRP-3DES-EDE-CBC-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:DES-CBC3-SHA:PSK-3DES-EDE-CBC-SHA:EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA:DES-CBC-SHA > > Any reason these are not built in? > > "Its a PITA" is a perfectly acceptable answer... the Windows build process > already looks convoluted. I was just surprised as it doesn't appear to be > difficult to add this since the included OpenSSL already supports them. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From akrupicka@REDACTED Thu Sep 10 11:01:13 2015 From: akrupicka@REDACTED (Adam Krupicka) Date: Thu, 10 Sep 2015 11:01:13 +0200 Subject: [erlang-questions] The inner workings of erlang:element/2 Message-ID: <1441874037-sup-3700@arch.local> Hello, I was recently profiling some code and noticed that erlang:element/2 doesn't show up in neither eprof nor fprof. How is this possible? Is the compiler being smart here and rewriting X = erlang:element(3, A) with {_, _, X} = A? If so, how can the compiler know the arity of the tuple - does it have some clever trick up it's sleeve that allows it to match only on part of the tuple, e.g. {_, _, X | _} = A? %% to borrow list syntax Thanks for reading, A. K. From bchesneau@REDACTED Thu Sep 10 11:02:56 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 10 Sep 2015 09:02:56 +0000 Subject: [erlang-questions] bad certificate if trying to verify StartSsl certificate In-Reply-To: References: Message-ID: I On Tue, Aug 11, 2015 at 9:54 AM Ingela Andin wrote: > Hi! > > 2015-07-16 11:16 GMT+02:00 Alex Hudich : > >> Hi! >> >> >> >> wget http://curl.haxx.se/ca/cacert.pem >> >> and then >> >> ssl:connect( "www.nicemine.ru", 443, >> [{verify,verify_peer},{server_name_indication,"www.nicemine.ru"},{depth,2},{cacertfile,"cacert.pem"}] >> ). >> >> gives me {error,{tls_alert,"bad certificate"}} >> >> >> >> > This site is not sending a correct certificate chain, I get all the > certificates that shall be in the chain but scrambled around and not in the > correct order, this is breaking the > SSL/TLS-protocol. OpenSSL will also get the error above when trying to > verify that chain, but later versions of OpenSSL and also other > implementations obviously tries to work around this by attempting to sort > them and run the validation again. > > You could do that too using the verify_fun if you really want to. We would > rather not make that a default feature as breaking security protocols is > usually a bad idea that could lead to vulnerabilities. > > > Regards Ingela Erlang/OTP Team - Ericsson AB > > I have the same issue on another host: rest-api.pay.nl: 15> ssl:connect( "rest-api.pay.nl", 443, [{verify,verify_peer},{server_name_indication,"rest-api.pay.nl"},{depth,2},{cacertfile, "priv/ca-bundle.crt"}] ). =ERROR REPORT==== 10-Sep-2015::11:01:31 === SSL: certify: ssl_handshake.erl:1476:Fatal error: bad certificate {error,{tls_alert,"bad certificate"}} the chain looks correct for me and curl handle it without issue. What do you mean by sorting certificates ? Any example? - benoit > > >> Why? Site can be opened ok in the browser. >> >> Erlang/OTP 17 [erts-6.3] >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Thu Sep 10 11:14:03 2015 From: roger@REDACTED (Roger Lipscombe) Date: Thu, 10 Sep 2015 10:14:03 +0100 Subject: [erlang-questions] Deploying forked Erlang releases? In-Reply-To: <1441807814.371889.378856505.200F5CD3@webmail.messagingengine.com> References: <1441807814.371889.378856505.200F5CD3@webmail.messagingengine.com> Message-ID: On 9 September 2015 at 15:10, Tristan Sloughter wrote: > relx (which rebar3 uses and can be used standalone) generates a useful start > script (set {extended_start_script, true} in the config of your project) > with options for starting, stopping, getting a remote console, upgrading the > release, and more. We're already using relx, so I just added the {include_erts, true} option. Next, to get the correct OTP build onto the build host, I added the following to the top of our Jenkins script: OTP_VERSION=$(cat .otp-version) KERL=./kerl # Is that installed somewhere? OTP_INSTALLATION=$($KERL list installations | grep "^$OTP_VERSION " | cut -d' ' -f2) if [ ! -s "$OTP_INSTALLATION/activate" ] ; then echo "Error: No OTP $OTP_VERSION installation; installing it using kerl..." if ! ( $KERL list builds | grep -q "^git,${OTP_VERSION}\$" ) ; then echo "Error: No OTP $OTP_VERSION build; building it using kerl..." # $OTP_VERSION is the tag in git and is used as the name of the build: $KERL build git https://github.com/electricimp/otp.git $OTP_VERSION $OTP_VERSION fi $KERL install $OTP_VERSION $HOME/erlang/$OTP_VERSION OTP_INSTALLATION=$HOME/erlang/$OTP_VERSION fi . $OTP_INSTALLATION/activate It slows down the build the first time that it's built on a new Jenkins slave, but you can short-cut that by having chef pre-install the relevant Erlang/OTP version beforehand if you want. We also use direnv for individual developer machines; the .envrc looks like this: OTP_VERSION=$(cat .otp-version) if has kerl; then OTP_INSTALLATION=$(kerl list installations | grep "^$OTP_VERSION " | cut -d' ' -f2) if [ -s "$OTP_INSTALLATION/activate" ] ; then echo "Using Erlang/OTP $OTP_VERSION (in $OTP_INSTALLATION) via kerl." . $OTP_INSTALLATION/activate export OTP_ROOT=$OTP_INSTALLATION export OTP_VERSION else echo "Erlang/OTP $OTP_VERSION not available via kerl; using default." fi else echo "kerl not available; using default Erlang." fi ...and we expect that developers have the correct Erlang/OTP installed using kerl, in an appropriate place. From essen@REDACTED Thu Sep 10 11:26:58 2015 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 10 Sep 2015 11:26:58 +0200 Subject: [erlang-questions] Deploying forked Erlang releases? In-Reply-To: References: <1441807814.371889.378856505.200F5CD3@webmail.messagingengine.com> Message-ID: <55F14CE2.7060409@ninenines.eu> On 09/10/2015 11:14 AM, Roger Lipscombe wrote: > We also use direnv for individual developer machines; the .envrc looks > like this: > > OTP_VERSION=$(cat .otp-version) > > if has kerl; then > OTP_INSTALLATION=$(kerl list installations | grep "^$OTP_VERSION " > | cut -d' ' -f2) > if [ -s "$OTP_INSTALLATION/activate" ] ; then > echo "Using Erlang/OTP $OTP_VERSION (in $OTP_INSTALLATION) via kerl." > . $OTP_INSTALLATION/activate > > export OTP_ROOT=$OTP_INSTALLATION > export OTP_VERSION > else > echo "Erlang/OTP $OTP_VERSION not available via kerl; using default." > fi > else > echo "kerl not available; using default Erlang." > fi > > ...and we expect that developers have the correct Erlang/OTP installed > using kerl, in an appropriate place. I have plans to allow Erlang.mk to manage the Erlang version a project needs. For example if you need 18.0.2 with dirty schedulers, you just put a little bit of configuration in your Makefile, and then Erlang.mk will build the missing Erlang and/or activate it if it's already there. This should make things a bit easier, particularly when you have to update Erlang. It's a natural extension of 'make -k ci' which already uses kerl to manage the Erlang versions you want to test your project against. Ticket is https://github.com/ninenines/erlang.mk/issues/328 if you want to subscribe or add more requirements. :-) -- Lo?c Hoguin http://ninenines.eu Author of The Erlanger Playbook, A book about software development using Erlang From sverker.eriksson@REDACTED Thu Sep 10 11:41:24 2015 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 10 Sep 2015 11:41:24 +0200 Subject: [erlang-questions] compile erlang source code error [obj/x86_64-unknown-linux-gnu/opt/smp/inet_drv.o] In-Reply-To: References: Message-ID: <55F15044.1090407@ericsson.com> On 09/10/2015 06:54 AM, Fredo Lee wrote: > make[1]: Entering directory `/home/users/tianwanli01/otp_src_18.0/erts' > make[2]: Entering directory > `/home/users/tianwanli01/otp_src_18.0/erts/emulator' > MAKE opt > make[3]: Entering directory > `/home/users/tianwanli01/otp_src_18.0/erts/emulator' > GEN x86_64-unknown-linux-gnu/gen_git_version.mk > > CC obj/x86_64-unknown-linux-gnu/opt/smp/inet_drv.o > drivers/common/inet_drv.c: In function `inet_init': > drivers/common/inet_drv.c:4031: error: enumerator value for > `compile_time_assert__' not integer constant > : The error is a bit cryptic, but looking at line 4031 we find this: /* Check the size of SCTP AssocID -- currently both this driver and the Erlang part require 32 bit: */ ERTS_CT_ASSERT(sizeof(sctp_assoc_t)==ASSOC_ID_LEN); This is a presumption about type 'sctp_assoc_t' being 32 bits long, which is for some reason not true in your build environment. A patch to fix this would need the add a more flexible treatment of type 'sctp_assoc_t' in both C and Erlang code. /Sverker, Erlang/OTP Ericsson -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Thu Sep 10 11:47:40 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 10 Sep 2015 09:47:40 +0000 Subject: [erlang-questions] bad certificate if trying to verify StartSsl certificate In-Reply-To: References: Message-ID: I have tested the SSL certificate on ssllab: https://www.ssllabs.com/ssltest/analyze.html?d=rest%2dapi.pay.nl&s=37.46.137.138&hideResults=on and the certificate seems OK. Not sure what is the issue then. For information, the CA bundle used come from the curl project itself: https://github.com/bagder/ca-bundle and I am using Erlang 18.0.3 . On Thu, Sep 10, 2015 at 11:02 AM Benoit Chesneau wrote: > I > On Tue, Aug 11, 2015 at 9:54 AM Ingela Andin > wrote: > >> Hi! >> >> 2015-07-16 11:16 GMT+02:00 Alex Hudich : >> >>> Hi! >>> >>> >>> >>> wget http://curl.haxx.se/ca/cacert.pem >>> >>> and then >>> >>> ssl:connect( "www.nicemine.ru", 443, >>> [{verify,verify_peer},{server_name_indication,"www.nicemine.ru"},{depth,2},{cacertfile,"cacert.pem"}] >>> ). >>> >>> gives me {error,{tls_alert,"bad certificate"}} >>> >>> >>> >>> >> This site is not sending a correct certificate chain, I get all the >> certificates that shall be in the chain but scrambled around and not in the >> correct order, this is breaking the >> SSL/TLS-protocol. OpenSSL will also get the error above when trying to >> verify that chain, but later versions of OpenSSL and also other >> implementations obviously tries to work around this by attempting to sort >> them and run the validation again. >> >> You could do that too using the verify_fun if you really want to. We >> would rather not make that a default feature as breaking security protocols >> is usually a bad idea that could lead to vulnerabilities. >> >> >> Regards Ingela Erlang/OTP Team - Ericsson AB >> >> > > I have the same issue on another host: rest-api.pay.nl: > > 15> ssl:connect( "rest-api.pay.nl", 443, > [{verify,verify_peer},{server_name_indication,"rest-api.pay.nl"},{depth,2},{cacertfile, > "priv/ca-bundle.crt"}] ). > > =ERROR REPORT==== 10-Sep-2015::11:01:31 === > SSL: certify: ssl_handshake.erl:1476:Fatal error: bad certificate > {error,{tls_alert,"bad certificate"}} > > > the chain looks correct for me and curl handle it without issue. What do > you mean by sorting certificates ? Any example? > > - benoit > > > > >> >> >>> Why? Site can be opened ok in the browser. >>> >>> Erlang/OTP 17 [erts-6.3] >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benmmurphy@REDACTED Thu Sep 10 12:44:30 2015 From: benmmurphy@REDACTED (Ben Murphy) Date: Thu, 10 Sep 2015 11:44:30 +0100 Subject: [erlang-questions] bad certificate if trying to verify StartSsl certificate In-Reply-To: References: Message-ID: I have a verify function that hacks around this problem. It adds the certs to a list during the 'verification' then it resorts and passes it off to the path validation. However, this function only supports validation from a single root cert because we are using it in production to connect to a server that has a chain signed by a non-public CA. You use it like: {verify_fun, fixed_root_lenient_verifier:create_verify_function(DerCaCert, 10)} Use this module at your own risk it may effectively disable your SSL security. I really think this resorting should be done in OTP or OTP should supply a cleaner hook for resorting. A hook that gives you the chain and the cacerts and lets you send back a new chain would be perfect :) -module(fixed_root_lenient_verifier). -record(state, {certs = [], root_cert, max_path_length}). -export([verify/3]). -export([create_verify_function/2]). create_verify_function(DerCertificate, MaxPathLength) -> DecodedCert = public_key:pkix_decode_cert(DerCertificate, otp), {fun fixed_root_lenient_verifier:verify/3, #state{root_cert = DecodedCert, max_path_length = MaxPathLength}}. append_cert(State, Cert) -> State#state{certs = [Cert | State#state.certs]}. default_verify(_Cert, Event, State) -> case Event of {bad_cert, Reason} -> {fail, Reason}; {extension, _} -> {unknown, State}; valid -> {valid, State}; valid_peer -> {valid, State} end. verify(Cert, Event, State) -> case Event of {bad_cert, _Reason} -> {valid, append_cert(State, Cert)}; {extension, _} -> {unknown, State}; valid -> {valid, append_cert(State, Cert)}; valid_peer -> case final_verification(State#state.certs, Cert, State#state.max_path_length, State#state.root_cert) of {ok, _} -> {valid, State}; {error, Reason} -> {fail, Reason} end end. final_verification(Certs, PeerCert, MaxPathLength, RootCertificate) -> Chain = fix_path(PeerCert, Certs), public_key:pkix_path_validation(RootCertificate, Chain, [{max_path_length, MaxPathLength}, {verify_fun, {fun default_verify/3, none}}]). fix_path(Peer, RestOfChain) -> make_chain(Peer, RestOfChain, []). make_chain(OtpCert, CertChain, ResultChain) -> case public_key:pkix_is_self_signed(OtpCert) of true -> [OtpCert | ResultChain ]; false -> case find_issuer(OtpCert, CertChain) of {ok, NewCert, NewCertChain} -> % we remove the cert that was found from the chain % to prevent infinite loops where a chain becomes % a loop make_chain(NewCert, NewCertChain, [OtpCert | ResultChain]); {error, issuer_not_found} -> % assume it is the 'trusted' certificate [OtpCert | ResultChain] end end. find_issuer(OtpCert, CertChain) -> {Not, Maybe} = lists:splitwith(fun(Candidate) -> not public_key:pkix_is_issuer(OtpCert, Candidate) end, CertChain), case Maybe of [Issuer | Rest] -> {ok, Issuer, Not ++ Rest}; [] -> {error, issuer_not_found} end. On Thu, Sep 10, 2015 at 10:02 AM, Benoit Chesneau wrote: > I > On Tue, Aug 11, 2015 at 9:54 AM Ingela Andin wrote: >> >> Hi! >> >> 2015-07-16 11:16 GMT+02:00 Alex Hudich : >>> >>> Hi! >>> >>> >>> >>> wget http://curl.haxx.se/ca/cacert.pem >>> >>> and then >>> >>> ssl:connect( "www.nicemine.ru", 443, >>> [{verify,verify_peer},{server_name_indication,"www.nicemine.ru"},{depth,2},{cacertfile,"cacert.pem"}] >>> ). >>> >>> gives me {error,{tls_alert,"bad certificate"}} >>> >>> >>> >> >> This site is not sending a correct certificate chain, I get all the >> certificates that shall be in the chain but scrambled around and not in the >> correct order, this is breaking the >> SSL/TLS-protocol. OpenSSL will also get the error above when trying to >> verify that chain, but later versions of OpenSSL and also other >> implementations obviously tries to work around this by attempting to sort >> them and run the validation again. >> >> You could do that too using the verify_fun if you really want to. We would >> rather not make that a default feature as breaking security protocols is >> usually a bad idea that could lead to vulnerabilities. >> >> >> Regards Ingela Erlang/OTP Team - Ericsson AB >> > > > I have the same issue on another host: rest-api.pay.nl: > > 15> ssl:connect( "rest-api.pay.nl", 443, > [{verify,verify_peer},{server_name_indication,"rest-api.pay.nl"},{depth,2},{cacertfile, > "priv/ca-bundle.crt"}] ). > > =ERROR REPORT==== 10-Sep-2015::11:01:31 === > SSL: certify: ssl_handshake.erl:1476:Fatal error: bad certificate > {error,{tls_alert,"bad certificate"}} > > > the chain looks correct for me and curl handle it without issue. What do you > mean by sorting certificates ? Any example? > > - benoit > > > >> >> >>> >>> Why? Site can be opened ok in the browser. >>> >>> Erlang/OTP 17 [erts-6.3] >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From bjorn@REDACTED Thu Sep 10 13:02:02 2015 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Thu, 10 Sep 2015 13:02:02 +0200 Subject: [erlang-questions] The inner workings of erlang:element/2 In-Reply-To: <1441874037-sup-3700@arch.local> References: <1441874037-sup-3700@arch.local> Message-ID: On Thu, Sep 10, 2015 at 11:01 AM, Adam Krupicka wrote: > Hello, > > I was recently profiling some code and noticed that erlang:element/2 > doesn't show up in neither eprof nor fprof. How is this possible? Is the > compiler being smart here and rewriting > > X = erlang:element(3, A) > > with > > {_, _, X} = A? > No. element/2 is implemented using a special instruction (or actually one of several different instructions depending on the context of the call). In general, any BIF that may be called in a guard are implemented using instructions that don't support tracing (i.e. don't show up in eprof/fprof). /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From kvratkin@REDACTED Thu Sep 10 10:40:17 2015 From: kvratkin@REDACTED (Kirill Ratkin) Date: Thu, 10 Sep 2015 11:40:17 +0300 Subject: [erlang-questions] Diameter service activation/deactivation timeout Message-ID: Hi, I have two functions listed below: service_start(Name) -> %% ... skipped some preparation steps for service activation Opts = [{'Origin-Host', atom_to_list(Name) ++ ".example.com"}, {'Origin-Realm', "example.com"}, {'Vendor-Id', ?VENDOR_ID}, {'Product-Name', "Just Diameter Server"}, {'Auth-Application-Id', Ids}] ++ Apps, TransportOpts = [{transport_module, T}, {transport_config, [{reuseaddr, true}, {ip, IP}, {port, P}]}], diameter:start_service(Name, Opts), diameter:add_transport(Name, {listen, TransportOpts}), diameter:subscribe(Name). service_stop(Name) -> diameter:unsubscribe(Name), diameter:remove_transport(Name, true), diameter:stop_service(Name). When I do service_start the Port is listen to incoming messages. Then I want to stop service. (there are no clients connected to service) I do 'service_stop' and diameter:services() says '[]'. So there aren't any active services. But 'netstat' shows port is in LISTEN state: tcp 0 0 127.0.0.1:3868 0.0.0.0:* LISTEN After about 30 seconds port is closed. It seems 30 secs is too long ... How can I configure this timeout? -------------- next part -------------- An HTML attachment was scrubbed... URL: From garcia.narbona@REDACTED Thu Sep 10 11:08:32 2015 From: garcia.narbona@REDACTED (=?UTF-8?Q?David_Garc=C3=ADa?=) Date: Thu, 10 Sep 2015 11:08:32 +0200 Subject: [erlang-questions] How combine WSDL along with XSD using erlsom + yaws_soap_lib In-Reply-To: References: Message-ID: Hi Wille, I attach you the WSDL (test.wsdl) I am using. I have tried three ways of using this WSDL: - first one this same one I am sending you - second one adding the import of the XSD file: - - - - - - third one, copying all the XSD content into the WSDL file I am using yaws_soap_lib.erl for doing so, along with erlsom library.I call the method: yaws_soap_lib:initModel("WHOLE_PATH_TO_test.wsdl"). I get the message: exited: {{shutdown, {failed_to_start_child,redsys_wsdl_connector, {{badmatch, {error,"Group definition not found Response/SEQ1"}}, [{yaws_soap_lib,addSchemas,4, [{file,"src/lib/yaws_soap_lib.erl"},{line,394}]}, {yaws_soap_lib,parseWsdls,4, [{file,"src/lib/yaws_soap_lib.erl"},{line,359}]}, {yaws_soap_lib,initModel2,5, [{file,"src/lib/yaws_soap_lib.erl"},{line,331}]}, {redsys_wsdl_connector,init,1, [{file,"src/redsys/redsys_wsdl_connector.erl"}, {line,29}]}, {gen_server,init_it,6, [{file,"gen_server.erl"},{line,304}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,239}]}]}}}, I also have been trying creating a config file to get headers for compiling WSDL. I also attach the config file I have prepared, but still can't getting working. I also attach a WSDL that is working for me with the same source code (the only difference is that XSD definition is embeded in the WSDL -- as in the third option I have tried with the non working example -- ). This XSD definition does not have sequences defined, then I don't have the problem I have with the non working WSDL. Regards, 2015-09-10 8:04 GMT+02:00 Willem de Jong : > Hello David, > > Yes, it should be possible to use yaws_soap_lib with a WSDL and an XSD, > either by importing the XSD or by embedding it in the WSDL. > > At first sight I would say that the definition of ClassA is valid and > should be accepted by erlsom. > > Can you give a bit more information about what you are doing? Perhaps send > me the WSDL and XSD and the commands that you have been trying. > > Regards, > Willem > > On Wed, Sep 9, 2015 at 7:56 PM, David Garc?a > wrote: > >> >> Would it be possible to combine WSDL along with XSD file when using >> yaws_soap_lib? >> I have already tried importing XSD from WSDL, as well as copying the >> content of the XSD into the WSDL. >> Apparently there's a problem with erlsom dealing with sequence >> definitions in the XSD file: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> I get "*Group definition not found*" for the sequence elements, because >> it looks like erlsom uses a default prefix ("P:") for creating internal >> structure, but it does not use the prefix in the sequence erlsom internal >> structure, then it can not link it. >> >> I have already tried compiling and adding the XSD model to the WSDL model >> using erlsom API methods, but I didn't succeed to get it working. >> >> Any clue? >> >> Thanks in advance. >> >> -- >> David >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- David -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: files.tar.gz Type: application/x-gzip Size: 2194 bytes Desc: not available URL: From akrupicka@REDACTED Thu Sep 10 14:14:36 2015 From: akrupicka@REDACTED (Adam Krupicka) Date: Thu, 10 Sep 2015 14:14:36 +0200 Subject: [erlang-questions] The inner workings of erlang:element/2 In-Reply-To: References: <1441874037-sup-3700@arch.local> Message-ID: <1441885521-sup-4787@arch.local> Excerpts from Bj?rn Gustavsson's message of 2015-09-10 13:02:02 +0200: > On Thu, Sep 10, 2015 at 11:01 AM, Adam Krupicka wrote: > > Hello, > > > > I was recently profiling some code and noticed that erlang:element/2 > > doesn't show up in neither eprof nor fprof. How is this possible? Is the > > compiler being smart here and rewriting > > > > X = erlang:element(3, A) > > > > with > > > > {_, _, X} = A? > > > > No. element/2 is implemented using a special instruction (or > actually one of several different instructions depending on > the context of the call). > > In general, any BIF that may be called in a guard are > implemented using instructions that don't support > tracing (i.e. don't show up in eprof/fprof). > > /Bjorn > I see. Thanks for the reply, makes things clearer. A. K. From tilman.holschuh@REDACTED Thu Sep 10 17:48:29 2015 From: tilman.holschuh@REDACTED (Tilman Holschuh) Date: Thu, 10 Sep 2015 08:48:29 -0700 Subject: [erlang-questions] Deploying forked Erlang releases? In-Reply-To: <1441807814.371889.378856505.200F5CD3@webmail.messagingengine.com> References: <1441807814.371889.378856505.200F5CD3@webmail.messagingengine.com> Message-ID: <09D440BD-D9E8-476A-B7EB-39F5F5755A6A@gmail.com> > But since you need a custom build that isn't an option :), but if you have some other CI server the same can be done and you only have to install your custom Erlang build on it and dev machines instead of the servers. relx also supports the ?-system_libs option which you can use to point your release to an Erlang installation somewhere on your file system. Cheers - Tilman From bchesneau@REDACTED Thu Sep 10 18:46:14 2015 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 10 Sep 2015 16:46:14 +0000 Subject: [erlang-questions] bad certificate if trying to verify StartSsl certificate In-Reply-To: References: Message-ID: On Thu, Sep 10, 2015 at 12:44 PM Ben Murphy wrote: > I have a verify function that hacks around this problem. It adds the > certs to a list during the 'verification' then it resorts and passes > it off to the path validation. However, this function only supports > validation from a single root cert because we are using it in > production to connect to a server that has a chain signed by a > non-public CA. You use it like: {verify_fun, > fixed_root_lenient_verifier:create_verify_function(DerCaCert, 10)} > > Use this module at your own risk it may effectively disable your SSL > security. I really think this resorting should be done in OTP or OTP > should supply a cleaner hook for resorting. A hook that gives you the > chain and the cacerts and lets you send back a new chain would be > perfect :) > ThanksBen! At least your code gave me some hints. I came to a simpler solution for now: https://github.com/benoitc/hackney/pull/241 Though I am not really sure it's correct. The only thing it does is to skip the self signed certificate and try to find a good one. Any feedback is appreciated. What would be the best way to reorder the certificate chain? - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz.biedrycki@REDACTED Thu Sep 10 21:36:30 2015 From: lukasz.biedrycki@REDACTED (=?UTF-8?Q?=C5=81ukasz_Biedrycki?=) Date: Thu, 10 Sep 2015 20:36:30 +0100 Subject: [erlang-questions] Diameter service activation/deactivation timeout In-Reply-To: References: Message-ID: Hi Kirill, I believe that the option you are looking for is ?? SO_LINGER ?. ?I? n Erlang it is {linger, {boolean(), integer() >= 0}} ??. In your case one of the transport_config option.? I recommend this article: http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html which explains in details how and when to use this option. I hope that helps. Btw what OS are you using? Cheers, Lukasz On Thu, Sep 10, 2015 at 9:40 AM, Kirill Ratkin wrote: > Hi, > > I have two functions listed below: > > service_start(Name) -> > %% ... skipped some preparation steps for service activation > Opts = [{'Origin-Host', atom_to_list(Name) ++ ".example.com"}, > {'Origin-Realm', "example.com"}, > {'Vendor-Id', ?VENDOR_ID}, > {'Product-Name', "Just Diameter Server"}, > {'Auth-Application-Id', Ids}] ++ Apps, > > TransportOpts = [{transport_module, T}, > {transport_config, [{reuseaddr, true}, > {ip, IP}, > {port, P}]}], > diameter:start_service(Name, Opts), > diameter:add_transport(Name, {listen, TransportOpts}), > diameter:subscribe(Name). > > > service_stop(Name) -> > diameter:unsubscribe(Name), > diameter:remove_transport(Name, true), > diameter:stop_service(Name). > > When I do service_start the Port is listen to incoming messages. > Then I want to stop service. (there are no clients connected to service) > I do 'service_stop' and diameter:services() says '[]'. So there aren't any > active services. > But 'netstat' shows port is in LISTEN state: > tcp 0 0 127.0.0.1:3868 0.0.0.0:* LISTEN > After about 30 seconds port is closed. > > It seems 30 secs is too long ... How can I configure this timeout? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Sep 11 01:17:13 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 11 Sep 2015 11:17:13 +1200 Subject: [erlang-questions] The inner workings of erlang:element/2 In-Reply-To: <1441885521-sup-4787@arch.local> References: <1441874037-sup-3700@arch.local> <1441885521-sup-4787@arch.local> Message-ID: On 11/09/2015, at 12:14 am, Adam Krupicka wrote: >>> I was recently profiling some code and noticed that erlang:element/2 >>> doesn't show up in neither eprof nor fprof. How is this possible? Is the >>> compiler being smart here and rewriting >>> >>> X = erlang:element(3, A) >>> >>> with >>> >>> {_, _, X} = A? % cat >elt.erl -module(elt). -export([elt/1]). elt(X) -> element(3, X). % erlc +"'S'" elt.erl % cat elt.S {function, elt, 1, 2}. {label,1}. {line,[{location,"elt.erl",4}]}. {func_info,{atom,elt},{atom,elt},1}. {label,2}. {line,[{location,"elt.erl",4}]}. {bif,element,{f,0},[{integer,3},{x,0}],{x,0}}. return. This does not contain a function call. From raimo+erlang-questions@REDACTED Fri Sep 11 10:40:49 2015 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Fri, 11 Sep 2015 10:40:49 +0200 Subject: [erlang-questions] The inner workings of erlang:element/2 In-Reply-To: References: <1441874037-sup-3700@arch.local> <1441885521-sup-4787@arch.local> Message-ID: <20150911084049.GA39332@erix.ericsson.se> On Fri, Sep 11, 2015 at 11:17:13AM +1200, Richard A. O'Keefe wrote: > > On 11/09/2015, at 12:14 am, Adam Krupicka wrote: > > >>> I was recently profiling some code and noticed that erlang:element/2 > >>> doesn't show up in neither eprof nor fprof. How is this possible? Is the > >>> compiler being smart here and rewriting > >>> > >>> X = erlang:element(3, A) > >>> > >>> with > >>> > >>> {_, _, X} = A? > > % cat >elt.erl > -module(elt). > -export([elt/1]). > > elt(X) -> element(3, X). > > % erlc +"'S'" elt.erl > % cat elt.S > > {function, elt, 1, 2}. > {label,1}. > {line,[{location,"elt.erl",4}]}. > {func_info,{atom,elt},{atom,elt},1}. > {label,2}. > {line,[{location,"elt.erl",4}]}. > {bif,element,{f,0},[{integer,3},{x,0}],{x,0}}. > return. > > > This does not contain a function call. True, but most {bif,...} calls are traceable. The loader translates this particular bif call to an virtual machine instruction that happens to not be traceable. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From akrupicka@REDACTED Fri Sep 11 11:21:52 2015 From: akrupicka@REDACTED (Adam Krupicka) Date: Fri, 11 Sep 2015 11:21:52 +0200 Subject: [erlang-questions] The inner workings of erlang:element/2 In-Reply-To: References: <1441874037-sup-3700@arch.local> <1441885521-sup-4787@arch.local> Message-ID: <1441962756-sup-1827@arch.local> > -module(elt). > -export([elt/1]). > > elt(X) -> element(3, X). > > % erlc +"'S'" elt.erl > % cat elt.S > > {function, elt, 1, 2}. > {label,1}. > {line,[{location,"elt.erl",4}]}. > {func_info,{atom,elt},{atom,elt},1}. > {label,2}. > {line,[{location,"elt.erl",4}]}. > {bif,element,{f,0},[{integer,3},{x,0}],{x,0}}. > return. > I added a function that calls setelement/3 and the difference in the generated assembler code is, indeed, apparent: {function, selt, 1, 4}. {label,3}. {line,[{location,"elt.erl",5}]}. {func_info,{atom,elt},{atom,selt},1}. {label,4}. {move,{x,0},{x,1}}. {move,{atom,val},{x,2}}. {move,{integer,3},{x,0}}. {line,[{location,"elt.erl",5}]}. {call_ext_only,3,{extfunc,erlang,setelement,3}}. A. K. From ingela.andin@REDACTED Fri Sep 11 12:09:41 2015 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 11 Sep 2015 12:09:41 +0200 Subject: [erlang-questions] Closing a ssh channel from client side with killing the connection In-Reply-To: References: Message-ID: Hi! Just terminate your channel process and the channel behavior will take care of closing the channel correctly. Regards Ingela Erlang/OTP team - Ericsson AB 2015-09-07 15:14 GMT+02:00 Vipin Nair : > Hello, > > I am calling ssh_connection:exec/4 from a process implementing > the ssh_channel behaviour. I want to ensure that all commands > exit within some time limit. Say, someone runs a tail -f > /var/log/syslog, I want to ensure that this is not run beyond X > seconds. > > My idea was to close the channel after the timeout, but I am not > sure how to do this. There may be multiple channel established > over the same connection so I cannot kill the connection. How > should I do this? > > A quick glance at the RFC[1] shows that either party can close > the channel. So this perhaps could be added as a new public api > in ssh_channel module. > > [1]: https://www.ietf.org/rfc/rfc4254.txt > > > -- > Regards, > Vipin > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From w.a.de.jong@REDACTED Fri Sep 11 12:14:10 2015 From: w.a.de.jong@REDACTED (Willem de Jong) Date: Fri, 11 Sep 2015 12:14:10 +0200 Subject: [erlang-questions] How combine WSDL along with XSD using erlsom + yaws_soap_lib In-Reply-To: References: Message-ID: Hello David, I had a look at the examples that you provided. I don't think that it is related to the prefix or the structure of the XSD. The use of the as one of the alternatives in a is a bit strange, but if I modify your "working_test.wsdl" to include a type with this feature then that works fine. The example that you provided that gives the error message has some other differences compared to the working example: the use of the (target-) namespace is is different (and possibly not correct), and elementFormDefault is set to "unqualified". It is more likely that the problem is related to one of these things. Also the types that are declared in the XSD are not used in the messages, which makes it difficult to do a proper test. Can you send me a "real" example of a WSDL that is giving you problems? If you want you can send it directly to me (so not via the mailing list). regards, Willem On Thu, Sep 10, 2015 at 11:08 AM, David Garc?a wrote: > Hi Wille, > > I attach you the WSDL (test.wsdl) I am using. I have tried three ways of > using this WSDL: > > - first one this same one I am sending you > - second one adding the import of the XSD file: > - > - > - schemaLocation="../wsdl/redsys/consultas/redsys_types.xsd"/> > - > - > - third one, copying all the XSD content into the WSDL file > > > I am using yaws_soap_lib.erl for doing so, along with erlsom library.I > call the method: > > yaws_soap_lib:initModel("WHOLE_PATH_TO_test.wsdl"). > > I get the message: > > exited: {{shutdown, > {failed_to_start_child,redsys_wsdl_connector, > {{badmatch, > {error,"Group definition not found > Response/SEQ1"}}, > [{yaws_soap_lib,addSchemas,4, > > [{file,"src/lib/yaws_soap_lib.erl"},{line,394}]}, > {yaws_soap_lib,parseWsdls,4, > > [{file,"src/lib/yaws_soap_lib.erl"},{line,359}]}, > {yaws_soap_lib,initModel2,5, > > [{file,"src/lib/yaws_soap_lib.erl"},{line,331}]}, > {redsys_wsdl_connector,init,1, > [{file,"src/redsys/redsys_wsdl_connector.erl"}, > {line,29}]}, > {gen_server,init_it,6, > [{file,"gen_server.erl"},{line,304}]}, > {proc_lib,init_p_do_apply,3, > [{file,"proc_lib.erl"},{line,239}]}]}}}, > > > I also have been trying creating a config file to get headers for > compiling WSDL. I also attach the config file I have prepared, but still > can't getting working. > > I also attach a WSDL that is working for me with the same source code (the > only difference is that XSD definition is embeded in the WSDL -- as in the > third option I have tried with the non working example -- ). This XSD > definition does not have sequences defined, then I don't have the problem I > have with the non working WSDL. > > Regards, > > > 2015-09-10 8:04 GMT+02:00 Willem de Jong : > >> Hello David, >> >> Yes, it should be possible to use yaws_soap_lib with a WSDL and an XSD, >> either by importing the XSD or by embedding it in the WSDL. >> >> At first sight I would say that the definition of ClassA is valid and >> should be accepted by erlsom. >> >> Can you give a bit more information about what you are doing? Perhaps >> send me the WSDL and XSD and the commands that you have been trying. >> >> Regards, >> Willem >> >> On Wed, Sep 9, 2015 at 7:56 PM, David Garc?a >> wrote: >> >>> >>> Would it be possible to combine WSDL along with XSD file when using >>> yaws_soap_lib? >>> I have already tried importing XSD from WSDL, as well as copying the >>> content of the XSD into the WSDL. >>> Apparently there's a problem with erlsom dealing with sequence >>> definitions in the XSD file: >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> I get "*Group definition not found*" for the sequence elements, because >>> it looks like erlsom uses a default prefix ("P:") for creating internal >>> structure, but it does not use the prefix in the sequence erlsom internal >>> structure, then it can not link it. >>> >>> I have already tried compiling and adding the XSD model to the WSDL >>> model using erlsom API methods, but I didn't succeed to get it working. >>> >>> Any clue? >>> >>> Thanks in advance. >>> >>> -- >>> David >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > > > -- > David > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Sat Sep 12 19:51:39 2015 From: g@REDACTED (Guilherme Andrade) Date: Sat, 12 Sep 2015 18:51:39 +0100 Subject: [erlang-questions] Deep data structures and lenses Message-ID: <55F4662B.8050707@gandrade.net> Hello list, Partially inspired by jlouis' erl-lenses[1] and an old discussion on this mailing list[2], I've been sketching a library for easier manipulation (both data fetching and updating) of deep data structures (records inside records, keylists inside records, etc.) I would like to give priority to: - Code readability; - Safety ('try to crash on unexpected data instead of producing hard-to-track misbehaviour';) - Reasonable performance (without going the parse-transform way.) Working code already public[3] but it's still a mess and requires some cooking before it's ready for human consumption. What follows is a few statements, also available on Github[4] that demonstrate the kind of syntax I am hoping to achieve; input and criticism are welcome, as I've been struggling with some design decisions and general syntactic coherence. [1]: https://github.com/jlouis/erl-lenses [2]: http://erlang.org/pipermail/erlang-questions/2013-January/071936.html [3]: https://github.com/g-andrade/monocle/tree/f1e530f15de5e4aff94dd42bfb2d572afbfc8b96 [4]: https://github.com/g-andrade/monocle/blob/f1e530f15de5e4aff94dd42bfb2d572afbfc8b96/test/monocle_tests.erl --------------------------------------------------------------------- -record(phone, { operator :: string(), number :: string(), balance :: float() }). -record(person, { name :: string(), age :: non_neg_integer(), sex :: m | f, phone :: #phone{}, lucky_numbers :: [number()] }). tuple_test() -> Phone1 = #phone{operator = "Telecom", number = "1231234", balance = 23.51}, Person1 = #person{name = "John Doe", age = 33, sex = m, phone = Phone1, lucky_numbers = [3, 75, 123, 3.2, 74]}, Lens1 = mtuple:get(#person.age), EvalA = monocle:eval(Lens1, Person1), ?assert(EvalA =:= 33), Lens2 = mtuple:get(#person.phone, mtuple:get(#phone.number)), EvalB = monocle:eval(Lens2, Person1), ?assert(EvalB =:= Phone1#phone.number), Lens3 = mtuple:get(#person.phone, mtuple:get(#phone.balance)), EvalC = monocle:eval(Lens3, Person1), ?assert(EvalC =:= Phone1#phone.balance), Lens4 = mtuple:set(#person.phone, mql:fold([mtuple:set(#phone.balance, mnum:sub(1)), mtuple:set(#phone.balance, mnum:mul(3)), mtuple:set(#phone.balance, mnum:'/'(3.0)), mtuple:set(#phone.balance, mql:fold([mnum:min(infinity), mnum:max(0), mnum:mul(-1), mnum:mul(-1)])), mtuple:set(#phone.balance, mnum:sub(1 bsl 3)), mtuple:set(#phone.balance, mnum:add(1 bsl 3))])), EvalD1 = monocle:eval(Lens4, Person1), EvalD2 = monocle:eval(Lens3, EvalD1), ?assert(EvalD2 =:= Phone1#phone.balance - 1), Lens5 = mtuple:get(#person.lucky_numbers, mlists:sum()), EvalE = monocle:eval(Lens5, Person1), ?assert(EvalE =:= lists:sum(Person1#person.lucky_numbers)), Lens6 = mql:map([mtuple:get(#person.name), mtuple:get(#person.age), mtuple:get(#person.phone, mtuple:get(#phone.number))]), EvalF = monocle:eval(Lens6, Person1), ?assert(EvalF =:= [Person1#person.name, Person1#person.age, Phone1#phone.number]), ok. -- Guilherme https://www.gandrade.net/ PGP: 0x602B2AD8 / B348 C976 CCE1 A02A 017E 4649 7A6E B621 602B 2AD8 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From benmmurphy@REDACTED Sun Sep 13 23:22:25 2015 From: benmmurphy@REDACTED (Ben Murphy) Date: Sun, 13 Sep 2015 22:22:25 +0100 Subject: [erlang-questions] bad certificate if trying to verify StartSsl certificate In-Reply-To: References: Message-ID: Just a note that this verify function is broken and accepts all self signed certificates. :( On Thu, Sep 10, 2015 at 11:44 AM, Ben Murphy wrote: > I have a verify function that hacks around this problem. It adds the > certs to a list during the 'verification' then it resorts and passes > it off to the path validation. However, this function only supports > validation from a single root cert because we are using it in > production to connect to a server that has a chain signed by a > non-public CA. You use it like: {verify_fun, > fixed_root_lenient_verifier:create_verify_function(DerCaCert, 10)} > > Use this module at your own risk it may effectively disable your SSL > security. I really think this resorting should be done in OTP or OTP > should supply a cleaner hook for resorting. A hook that gives you the > chain and the cacerts and lets you send back a new chain would be > perfect :) > > -module(fixed_root_lenient_verifier). > > -record(state, {certs = [], root_cert, max_path_length}). > > -export([verify/3]). > -export([create_verify_function/2]). > > create_verify_function(DerCertificate, MaxPathLength) -> > DecodedCert = public_key:pkix_decode_cert(DerCertificate, otp), > {fun fixed_root_lenient_verifier:verify/3, #state{root_cert = > DecodedCert, max_path_length = MaxPathLength}}. > > append_cert(State, Cert) -> > State#state{certs = [Cert | State#state.certs]}. > > default_verify(_Cert, Event, State) -> > case Event of > {bad_cert, Reason} -> {fail, Reason}; > {extension, _} -> {unknown, State}; > valid -> {valid, State}; > valid_peer -> {valid, State} > end. > > > verify(Cert, Event, State) -> > case Event of > {bad_cert, _Reason} -> > {valid, append_cert(State, Cert)}; > {extension, _} -> {unknown, State}; > valid -> {valid, append_cert(State, Cert)}; > valid_peer -> > case final_verification(State#state.certs, Cert, > State#state.max_path_length, State#state.root_cert) of > {ok, _} -> {valid, State}; > {error, Reason} -> {fail, Reason} > end > end. > > final_verification(Certs, PeerCert, MaxPathLength, RootCertificate) -> > Chain = fix_path(PeerCert, Certs), > public_key:pkix_path_validation(RootCertificate, Chain, > [{max_path_length, MaxPathLength}, {verify_fun, {fun default_verify/3, > none}}]). > > > fix_path(Peer, RestOfChain) -> > make_chain(Peer, RestOfChain, []). > > make_chain(OtpCert, CertChain, ResultChain) -> > case public_key:pkix_is_self_signed(OtpCert) of > true -> > [OtpCert | ResultChain ]; > false -> > case find_issuer(OtpCert, CertChain) of > {ok, NewCert, NewCertChain} -> > % we remove the cert that was found from the chain > % to prevent infinite loops where a chain becomes > % a loop > make_chain(NewCert, NewCertChain, [OtpCert | ResultChain]); > {error, issuer_not_found} -> > % assume it is the 'trusted' certificate > [OtpCert | ResultChain] > end > end. > > find_issuer(OtpCert, CertChain) -> > {Not, Maybe} = lists:splitwith(fun(Candidate) -> > not public_key:pkix_is_issuer(OtpCert, Candidate) > end, CertChain), > > case Maybe of > [Issuer | Rest] -> > {ok, Issuer, Not ++ Rest}; > [] -> > {error, issuer_not_found} > end. > > On Thu, Sep 10, 2015 at 10:02 AM, Benoit Chesneau wrote: >> I >> On Tue, Aug 11, 2015 at 9:54 AM Ingela Andin wrote: >>> >>> Hi! >>> >>> 2015-07-16 11:16 GMT+02:00 Alex Hudich : >>>> >>>> Hi! >>>> >>>> >>>> >>>> wget http://curl.haxx.se/ca/cacert.pem >>>> >>>> and then >>>> >>>> ssl:connect( "www.nicemine.ru", 443, >>>> [{verify,verify_peer},{server_name_indication,"www.nicemine.ru"},{depth,2},{cacertfile,"cacert.pem"}] >>>> ). >>>> >>>> gives me {error,{tls_alert,"bad certificate"}} >>>> >>>> >>>> >>> >>> This site is not sending a correct certificate chain, I get all the >>> certificates that shall be in the chain but scrambled around and not in the >>> correct order, this is breaking the >>> SSL/TLS-protocol. OpenSSL will also get the error above when trying to >>> verify that chain, but later versions of OpenSSL and also other >>> implementations obviously tries to work around this by attempting to sort >>> them and run the validation again. >>> >>> You could do that too using the verify_fun if you really want to. We would >>> rather not make that a default feature as breaking security protocols is >>> usually a bad idea that could lead to vulnerabilities. >>> >>> >>> Regards Ingela Erlang/OTP Team - Ericsson AB >>> >> >> >> I have the same issue on another host: rest-api.pay.nl: >> >> 15> ssl:connect( "rest-api.pay.nl", 443, >> [{verify,verify_peer},{server_name_indication,"rest-api.pay.nl"},{depth,2},{cacertfile, >> "priv/ca-bundle.crt"}] ). >> >> =ERROR REPORT==== 10-Sep-2015::11:01:31 === >> SSL: certify: ssl_handshake.erl:1476:Fatal error: bad certificate >> {error,{tls_alert,"bad certificate"}} >> >> >> the chain looks correct for me and curl handle it without issue. What do you >> mean by sorting certificates ? Any example? >> >> - benoit >> >> >> >>> >>> >>>> >>>> Why? Site can be opened ok in the browser. >>>> >>>> Erlang/OTP 17 [erts-6.3] >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> From jesper.louis.andersen@REDACTED Mon Sep 14 12:27:32 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 14 Sep 2015 12:27:32 +0200 Subject: [erlang-questions] Deep data structures and lenses In-Reply-To: <55F4662B.8050707@gandrade.net> References: <55F4662B.8050707@gandrade.net> Message-ID: On Sat, Sep 12, 2015 at 7:51 PM, Guilherme Andrade wrote: > Partially inspired by jlouis' erl-lenses[1] and an old discussion on this > mailing list[2], I've been sketching a library for easier manipulation > (both data fetching and updating) of deep data structures (records inside > records, keylists inside records, etc.) Awesome idea :) I think I'll lurk and see where you take this. Definitely it is one of the things we need in Erlang, but we also slightly lack the syntax expressiveness to make it really nice. One thing I'd think about is to use maps as the canonical structure to which you blit input. This allows you to "melt" any tree-like data structure and then "cast" it in another format, eventually updating data along the path. There is a nice set of algebraic properties underneath, which amends themselves to (stateless) quickchecking by the way. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ravindrapai34@REDACTED Mon Sep 14 17:58:12 2015 From: ravindrapai34@REDACTED (ravindra pai) Date: Mon, 14 Sep 2015 08:58:12 -0700 (PDT) Subject: [erlang-questions] race condition while registering to gproc Message-ID: Hi All, I am still learning erlang and have bulit a pub-sub system using cowboy websocket and gproc. While doing load testing, my code is hitting a race condition often, I have put the code to recover from race condition. I wanted to know how to avoid entering into the race condition, someone can have better answer :-) Whenever new websocket connection is made to a Room, Please see the code below. Check in gproc whether a gen_server representing Roomname, If exist subscribe to the room. Else spawn a gen_server and register with gproc with Roomname. More often, gproc:reg({n, l, {session, Roomname}}) is raising {badmatch, badarg} error as you cannot register two process with {n, l, {session, Roomname}}. So when a concurrent connections made to Roomname, two or more process is trying to do gproc:reg({n, l, {session, Roomname}}), only one process is succeeding. ws_handler.erl websocket_info({post_init, Roomname}, Req, _State) -> Pid = case gproc:where({n, l, {session, Roomname}}) of undefined -> % this will start a gen_server representing the room or session. {ok, P} = supervisor:start_child(sessions_sup, [RoomName]), P; P -> P end, session.erl -behaviour(gen_server). %% @private init([Roomname]) -> io:format("session init started for ~p~n", [Roomname]), try gproc:reg({n, l, {session, Roomname}}), gproc:reg({p, l, Roomname}) catch _ -> ok end, Thanks in advance for your advice, Ravindra M -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Tue Sep 15 02:33:51 2015 From: rvirding@REDACTED (Robert Virding) Date: Tue, 15 Sep 2015 02:33:51 +0200 Subject: [erlang-questions] ets:match_object/3 strangeness Message-ID: I have an ETS table called *sune* which is a duplicate_bag containing many elements with the same key. The *bert* elements have the structure *{bert,SequenceNo,Data}* so I can see the ordering. If I do *ets:match_object(sune, {bert,'_','_'})* I will get back a list of all the matching elements in the order they were inserted, which is what I want. However, there can be quite a few elements with the same key so I would prefer to get them in chunks instead of all at once. The solution for this is to start with *ets:match_object(sune, {bert,'_','_'}, 5)* which returns a continuation and then follow with a sequence of *ets:match_object(Continuation)* until I get all the elements in chunks of 5. That is when things start to get strange: - Now the ordering is completely messed up. Each chunk of 5 contains a sequence but some in ascending order and some in descending order, for example 440,439,438,437,436. - The chunks come in what looks like random order, for example chunks starting with 571,425,566,430 but starting with the outermost first working towards the middle. - The really weird thing is that the continuation actually contains all the matching elements which haven't been returned yet. The last is the really strange. According to the documentation for match_object/3: "Works like ets:match_object/2 but only returns a limited (Limit) number of matching objects. The Continuation term can then be used in subsequent calls to ets:match_object/1 to get the next chunk of matching objects. This is a space efficient way to work on objects in a table which is still faster than traversing the table object by object using ets:first/1 and ets:next/1." But the continuation contains them all so where is the space efficiency? I mean why bother to use this at all as it gives me no benefits. I tried the same thing using *ets:select/3/1* and got the same result. My original goal was to work how this would interact with inserting or deleting elements while I was scanning. My got my answer. Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From schneider@REDACTED Tue Sep 15 10:04:06 2015 From: schneider@REDACTED (Frans Schneider) Date: Tue, 15 Sep 2015 10:04:06 +0200 Subject: [erlang-questions] Custom behavior background information Message-ID: <55F7D0F6.7020402@xs4all.nl> Hi list, I find myself repeating code in several gen_server and gen_fsm behavior based modules. These modules should all share about eight functions, all doing exactly the same thing. On top of these modules, more elaborate modules with common functions have to be defined, adding four or so functions. To me, this is looking like defining my own custom behaviors. I had a look at the implementation of the Erlang supervisor for extending the gen_server behavior which gives enough information to get something working. However, I have no glue how to extend the gen_fsm behavior and keep the interface to defining states in the new behavior. So, what I am looking for is some reference material on writing custom behaviors on top of what Erlang central and the OTP documentation offer. TIA, Frans -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Tue Sep 15 10:53:17 2015 From: vasdeveloper@REDACTED (Theepan) Date: Tue, 15 Sep 2015 14:23:17 +0530 Subject: [erlang-questions] Custom behavior background information In-Reply-To: <55F7D0F6.7020402@xs4all.nl> References: <55F7D0F6.7020402@xs4all.nl> Message-ID: A behaviour behaves in some specific way, and importantly the behaviour is common enough (can be used repeatedly) to define it separately. A behaviour exposes two sets of APIs. One set facilitates the communication with external processes. The second one facilitates to define the behaviour of it. The second set is also called call back functions. The second set is like a contract that a call back module for a behaviour should adhere to. There may or may not a mapping between the two sets of APIs. It is the responsibility of the behaviour defining module to call the functions in the callback module when the triggering points are reached. When you are using behaviours, you can use multiple of them meaningfully in a single module. This is called cascading behaviours. Theepan On Tue, Sep 15, 2015 at 1:34 PM, Frans Schneider wrote: > Hi list, > > I find myself repeating code in several gen_server and gen_fsm behavior > based modules. These modules should all share about eight functions, all > doing exactly the same thing. On top of these modules, more elaborate > modules with common functions have to be defined, adding four or so > functions. To me, this is looking like defining my own custom behaviors. > I had a look at the implementation of the Erlang supervisor for extending > the gen_server behavior which gives enough information to get something > working. However, I have no glue how to extend the gen_fsm behavior and > keep the interface to defining states in the new behavior. > So, what I am looking for is some reference material on writing custom > behaviors on top of what Erlang central and the OTP documentation offer. > > TIA, > > Frans > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Tue Sep 15 15:18:32 2015 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Tue, 15 Sep 2015 15:18:32 +0200 Subject: [erlang-questions] ets:match_object/3 strangeness In-Reply-To: References: Message-ID: <55F81AA8.8030706@ericsson.com> ETS 'bag' and 'duplicate_bag' allow key duplicates, but are not at all optimized for large number of duplicates. Both hashing and iteration is only based on keys. All key duplicates will thus end up in the same hash bucket as a linked list. All iterations (next,prev,match,select) have to return all duplicates in one call as there is no way to yield/resume in the middle of a cluster of duplicates. That is why the continuation term contains the rest of them. The ordering strangeness seems more like a plain bug. /Sverker, Erlang/OTP Ericsson On 09/15/2015 02:33 AM, Robert Virding wrote: > I have an ETS table called *sune* which is a duplicate_bag containing > many elements with the same key. The *bert* elements have the > structure *{bert,SequenceNo,Data}* so I can see the ordering. If I do > *ets:match_object(sune, {bert,'_','_'})* I will get back a list of all > the matching elements in the order they were inserted, which is what I > want. However, there can be quite a few elements with the same key so > I would prefer to get them in chunks instead of all at once. The > solution for this is to start with *ets:match_object(sune, > {bert,'_','_'}, 5)* which returns a continuation and then follow with > a sequence of *ets:match_object(Continuation)* until I get all the > elements in chunks of 5. That is when things start to get strange: > > - Now the ordering is completely messed up. Each chunk of 5 contains a > sequence but some in ascending order and some in descending order, for > example 440,439,438,437,436. > - The chunks come in what looks like random order, for example chunks > starting with 571,425,566,430 but starting with the outermost first > working towards the middle. > - The really weird thing is that the continuation actually contains > all the matching elements which haven't been returned yet. > > The last is the really strange. According to the documentation for > match_object/3: > > "Works like ets:match_object/2 but only returns a limited (Limit) > number of matching objects. The Continuation term can then be used in > subsequent calls to ets:match_object/1 to get the next chunk of > matching objects. This is a space efficient way to work on objects in > a table which is still faster than traversing the table object by > object using ets:first/1 and ets:next/1." > > But the continuation contains them all so where is the space > efficiency? I mean why bother to use this at all as it gives me no > benefits. > > I tried the same thing using *ets:select/3/1* and got the same result. > > My original goal was to work how this would interact with inserting or > deleting elements while I was scanning. My got my answer. > > Robert > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From samjbarney@REDACTED Tue Sep 15 18:43:28 2015 From: samjbarney@REDACTED (Samuel Barney) Date: Tue, 15 Sep 2015 10:43:28 -0600 Subject: [erlang-questions] Having trouble with httpc Message-ID: I'm trying to use httpc:request/1, but every single time I call it I get a no_scheme error. I've started inets, and all of the examples I've found say I am doing it correctly. Has anyone had this error before, and how did you resolve it? Respectfully, Samuel Barney -------------- next part -------------- An HTML attachment was scrubbed... URL: From drew.varner@REDACTED Tue Sep 15 18:48:18 2015 From: drew.varner@REDACTED (Drew Varner) Date: Tue, 15 Sep 2015 18:48:18 +0200 Subject: [erlang-questions] Having trouble with httpc In-Reply-To: References: Message-ID: <06B81E16-B114-4B3F-BC77-FF35F6E719B5@redops.org> Samuel, You need to make sure you have a compete URI, for example ?http://www.google.com ?. It looks like you forget the ?http://? portion. See https://github.com/erlang/otp/blob/a2670f0822fc6729df956c8ec8c381340ff0a5fb/lib/inets/src/http_lib/http_uri.erl#L137-L139 Cheers, Drew > On Sep 15, 2015, at 6:43 PM, Samuel Barney wrote: > > I'm trying to use httpc:request/1, but every single time I call it I get a no_scheme error. I've started inets, and all of the examples I've found say I am doing it correctly. Has anyone had this error before, and how did you resolve it? > > Respectfully, > Samuel Barney > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From samjbarney@REDACTED Tue Sep 15 18:52:06 2015 From: samjbarney@REDACTED (Samuel Barney) Date: Tue, 15 Sep 2015 10:52:06 -0600 Subject: [erlang-questions] Having trouble with httpc In-Reply-To: <06B81E16-B114-4B3F-BC77-FF35F6E719B5@redops.org> References: <06B81E16-B114-4B3F-BC77-FF35F6E719B5@redops.org> Message-ID: I just tried hitting a different page, and it worked just fine. I do have the http://, but I think the error is that the url ends in '.xml'. Is there a way to get around that? Respectfully, Samuel Barney On Tue, Sep 15, 2015 at 10:48 AM, Drew Varner wrote: > Samuel, > > You need to make sure you have a compete URI, for example ? > http://www.google.com?. > > It looks like you forget the ?http://? portion. > > See > https://github.com/erlang/otp/blob/a2670f0822fc6729df956c8ec8c381340ff0a5fb/lib/inets/src/http_lib/http_uri.erl#L137-L139 > > Cheers, > Drew > > On Sep 15, 2015, at 6:43 PM, Samuel Barney wrote: > > I'm trying to use httpc:request/1, but every single time I call it I get a > no_scheme error. I've started inets, and all of the examples I've found say > I am doing it correctly. Has anyone had this error before, and how did you > resolve it? > > Respectfully, > Samuel Barney > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From samjbarney@REDACTED Tue Sep 15 18:53:00 2015 From: samjbarney@REDACTED (Samuel Barney) Date: Tue, 15 Sep 2015 10:53:00 -0600 Subject: [erlang-questions] Having trouble with httpc In-Reply-To: References: <06B81E16-B114-4B3F-BC77-FF35F6E719B5@redops.org> Message-ID: To clarify, is there a way for me to access a url that ends with '.xml' On Tue, Sep 15, 2015 at 10:52 AM, Samuel Barney wrote: > I just tried hitting a different page, and it worked just fine. I do have > the http://, but I think the error is that the url ends in '.xml'. Is > there a way to get around that? > > Respectfully, > Samuel Barney > > On Tue, Sep 15, 2015 at 10:48 AM, Drew Varner > wrote: > >> Samuel, >> >> You need to make sure you have a compete URI, for example ? >> http://www.google.com?. >> >> It looks like you forget the ?http://? portion. >> >> See >> https://github.com/erlang/otp/blob/a2670f0822fc6729df956c8ec8c381340ff0a5fb/lib/inets/src/http_lib/http_uri.erl#L137-L139 >> >> Cheers, >> Drew >> >> On Sep 15, 2015, at 6:43 PM, Samuel Barney wrote: >> >> I'm trying to use httpc:request/1, but every single time I call it I get >> a no_scheme error. I've started inets, and all of the examples I've found >> say I am doing it correctly. Has anyone had this error before, and how did >> you resolve it? >> >> Respectfully, >> Samuel Barney >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From drew.varner@REDACTED Tue Sep 15 18:59:25 2015 From: drew.varner@REDACTED (Drew Varner) Date: Tue, 15 Sep 2015 18:59:25 +0200 Subject: [erlang-questions] Having trouble with httpc In-Reply-To: References: <06B81E16-B114-4B3F-BC77-FF35F6E719B5@redops.org> Message-ID: <9DF01E58-B14B-47A6-946D-50D32C4D7C73@redops.org> Can you share an example? ?snip-- Eshell V7.0 (abort with ^G) 1> inets:start(). ok 2> ssl:start(). ok 3> httpc:request("https://raw.githubusercontent.com/processone/ejabberd/6a9c9616d04003d73a7ce2916ee03c997f66bc1d/examples/transport-configs/configs/aim-transport.xml"). ?end snip ? Works for me. > On Sep 15, 2015, at 6:53 PM, Samuel Barney wrote: > > To clarify, is there a way for me to access a url that ends with '.xml' > > On Tue, Sep 15, 2015 at 10:52 AM, Samuel Barney wrote: > I just tried hitting a different page, and it worked just fine. I do have the http://, but I think the error is that the url ends in '.xml'. Is there a way to get around that? > > Respectfully, > Samuel Barney > > On Tue, Sep 15, 2015 at 10:48 AM, Drew Varner wrote: > Samuel, > > You need to make sure you have a compete URI, for example ?http://www.google.com?. > > It looks like you forget the ?http://? portion. > > See https://github.com/erlang/otp/blob/a2670f0822fc6729df956c8ec8c381340ff0a5fb/lib/inets/src/http_lib/http_uri.erl#L137-L139 > > Cheers, > Drew > >> On Sep 15, 2015, at 6:43 PM, Samuel Barney wrote: >> >> I'm trying to use httpc:request/1, but every single time I call it I get a no_scheme error. I've started inets, and all of the examples I've found say I am doing it correctly. Has anyone had this error before, and how did you resolve it? >> >> Respectfully, >> Samuel Barney >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > From yasha32@REDACTED Tue Sep 15 18:58:33 2015 From: yasha32@REDACTED (Mr rty ff) Date: Tue, 15 Sep 2015 16:58:33 +0000 (UTC) Subject: [erlang-questions] distinguish proplists vs. lists of proplists Message-ID: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> Hi I write a function which gets?proplists?or?list?of proplists anf key. if the input is proplists I just use?proplists:get_value?to find a value corresponding to key. But if the input is lists of proplists I need to iterate over all its elements. and look for key in each of them. I amnot sure how to distinguish?lists?vs.?proplists.?lists:is_list?returns?true?for both of them. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From samjbarney@REDACTED Tue Sep 15 19:14:07 2015 From: samjbarney@REDACTED (Samuel Barney) Date: Tue, 15 Sep 2015 11:14:07 -0600 Subject: [erlang-questions] Having trouble with httpc In-Reply-To: <9DF01E58-B14B-47A6-946D-50D32C4D7C73@redops.org> References: <06B81E16-B114-4B3F-BC77-FF35F6E719B5@redops.org> <9DF01E58-B14B-47A6-946D-50D32C4D7C73@redops.org> Message-ID: Eshell V7.0.3 (abort with ^G) 1> inets:start(). ok 2> ssl:start(). ok 3> httpc:request("http://ingeniuxtest.byui.edu/d92/x107221.xml"). {error,no_scheme} On Tue, Sep 15, 2015 at 10:59 AM, Drew Varner wrote: > Can you share an example? > > ?snip-- > Eshell V7.0 (abort with ^G) > 1> inets:start(). > ok > 2> ssl:start(). > ok > 3> httpc:request(" > https://raw.githubusercontent.com/processone/ejabberd/6a9c9616d04003d73a7ce2916ee03c997f66bc1d/examples/transport-configs/configs/aim-transport.xml > "). > ?end snip ? > > Works for me. > > > On Sep 15, 2015, at 6:53 PM, Samuel Barney wrote: > > > > To clarify, is there a way for me to access a url that ends with '.xml' > > > > On Tue, Sep 15, 2015 at 10:52 AM, Samuel Barney > wrote: > > I just tried hitting a different page, and it worked just fine. I do > have the http://, but I think the error is that the url ends in '.xml'. > Is there a way to get around that? > > > > Respectfully, > > Samuel Barney > > > > On Tue, Sep 15, 2015 at 10:48 AM, Drew Varner > wrote: > > Samuel, > > > > You need to make sure you have a compete URI, for example ? > http://www.google.com?. > > > > It looks like you forget the ?http://? portion. > > > > See > https://github.com/erlang/otp/blob/a2670f0822fc6729df956c8ec8c381340ff0a5fb/lib/inets/src/http_lib/http_uri.erl#L137-L139 > > > > Cheers, > > Drew > > > >> On Sep 15, 2015, at 6:43 PM, Samuel Barney > wrote: > >> > >> I'm trying to use httpc:request/1, but every single time I call it I > get a no_scheme error. I've started inets, and all of the examples I've > found say I am doing it correctly. Has anyone had this error before, and > how did you resolve it? > >> > >> Respectfully, > >> Samuel Barney > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From samjbarney@REDACTED Tue Sep 15 19:15:51 2015 From: samjbarney@REDACTED (Samuel Barney) Date: Tue, 15 Sep 2015 11:15:51 -0600 Subject: [erlang-questions] Having trouble with httpc In-Reply-To: References: <06B81E16-B114-4B3F-BC77-FF35F6E719B5@redops.org> <9DF01E58-B14B-47A6-946D-50D32C4D7C73@redops.org> Message-ID: The thing is, it works on requests where the resulting page is in valid xml. On Tue, Sep 15, 2015 at 11:14 AM, Samuel Barney wrote: > Eshell V7.0.3 (abort with ^G) > > 1> inets:start(). > > ok > > 2> ssl:start(). > > ok > > 3> httpc:request("http://ingeniuxtest.byui.edu/d92/x107221.xml"). > > {error,no_scheme} > > On Tue, Sep 15, 2015 at 10:59 AM, Drew Varner > wrote: > >> Can you share an example? >> >> ?snip-- >> Eshell V7.0 (abort with ^G) >> 1> inets:start(). >> ok >> 2> ssl:start(). >> ok >> 3> httpc:request(" >> https://raw.githubusercontent.com/processone/ejabberd/6a9c9616d04003d73a7ce2916ee03c997f66bc1d/examples/transport-configs/configs/aim-transport.xml >> "). >> ?end snip ? >> >> Works for me. >> >> > On Sep 15, 2015, at 6:53 PM, Samuel Barney >> wrote: >> > >> > To clarify, is there a way for me to access a url that ends with '.xml' >> > >> > On Tue, Sep 15, 2015 at 10:52 AM, Samuel Barney >> wrote: >> > I just tried hitting a different page, and it worked just fine. I do >> have the http://, but I think the error is that the url ends in '.xml'. >> Is there a way to get around that? >> > >> > Respectfully, >> > Samuel Barney >> > >> > On Tue, Sep 15, 2015 at 10:48 AM, Drew Varner >> wrote: >> > Samuel, >> > >> > You need to make sure you have a compete URI, for example ? >> http://www.google.com?. >> > >> > It looks like you forget the ?http://? portion. >> > >> > See >> https://github.com/erlang/otp/blob/a2670f0822fc6729df956c8ec8c381340ff0a5fb/lib/inets/src/http_lib/http_uri.erl#L137-L139 >> > >> > Cheers, >> > Drew >> > >> >> On Sep 15, 2015, at 6:43 PM, Samuel Barney >> wrote: >> >> >> >> I'm trying to use httpc:request/1, but every single time I call it I >> get a no_scheme error. I've started inets, and all of the examples I've >> found say I am doing it correctly. Has anyone had this error before, and >> how did you resolve it? >> >> >> >> Respectfully, >> >> Samuel Barney >> >> _______________________________________________ >> >> erlang-questions mailing list >> >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> > >> > >> > >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.fitzpatrick6012@REDACTED Tue Sep 15 19:39:00 2015 From: matthew.fitzpatrick6012@REDACTED (Matthew Fitzpatrick) Date: Tue, 15 Sep 2015 12:39:00 -0500 Subject: [erlang-questions] -callback directives with Dialyzer Message-ID: Hey All, This was done with Erlang R15B02. The example code and problem can also be seen here: https://github.com/Ceryni/cuddly-octo-adventure Based on everything I've been reading and looking at, it seemed like if I have a behavior like this... ```erlang % dude.erl -module(dude). -callback hey_dude(Say :: string()) -> Res :: string(). ``` And I then implement it... ```erlang % no_spec_dude.erl -module(no_spec_dude). -behaviour(dude). -export([hey_dude/1]). hey_dude(Say) -> lists:flatten(io_lib:format("Hey Dude, ~p", [Say])). ``` And finally, I try to use it... ```erlang % runner.erl -module(runner). -export([run/0]). -spec run() -> ok. run() -> io:format("~p", [no_spec_dude:hey_dude(42)]). ``` If i run dialyzer with these 3 files, It should complain about a contract violation right? Except I'm running into the problem of that its not. I created a sample repo with the output of dialyzer here: https://github.com/Ceryni/cuddly-octo-adventure. If I explicitly define `-spec` then it works as expected and barks at me for using an integer instead of a string, but I thought dialyzer would use the -callback? Any thoughts on what I'm doing wrong here? Maybe using dialyzer incorrectly? My Erlang SDK is too old? Just trying to figure out what's going on. Thanks in advance! -Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From drew.varner@REDACTED Tue Sep 15 19:44:44 2015 From: drew.varner@REDACTED (Drew Varner) Date: Tue, 15 Sep 2015 19:44:44 +0200 Subject: [erlang-questions] Having trouble with httpc In-Reply-To: References: <06B81E16-B114-4B3F-BC77-FF35F6E719B5@redops.org> <9DF01E58-B14B-47A6-946D-50D32C4D7C73@redops.org> Message-ID: Samuel, httpc:request(get, {Url, []}, [{auto redirect, false}], []). I bet your server is giving you a redirect and the redirect URL is invalid. - Drew > On Sep 15, 2015, at 7:14 PM, Samuel Barney wrote: > > Eshell V7.0.3 (abort with ^G) > > 1> inets:start(). > > ok > > 2> ssl:start(). > > ok > > 3> httpc:request("http://ingeniuxtest.byui.edu/d92/x107221.xml "). > > {error,no_scheme} > > > On Tue, Sep 15, 2015 at 10:59 AM, Drew Varner > wrote: > Can you share an example? > > ?snip-- > Eshell V7.0 (abort with ^G) > 1> inets:start(). > ok > 2> ssl:start(). > ok > 3> httpc:request("https://raw.githubusercontent.com/processone/ejabberd/6a9c9616d04003d73a7ce2916ee03c997f66bc1d/examples/transport-configs/configs/aim-transport.xml "). > ?end snip ? > > Works for me. > > > On Sep 15, 2015, at 6:53 PM, Samuel Barney > wrote: > > > > To clarify, is there a way for me to access a url that ends with '.xml' > > > > On Tue, Sep 15, 2015 at 10:52 AM, Samuel Barney > wrote: > > I just tried hitting a different page, and it worked just fine. I do have the http://, but I think the error is that the url ends in '.xml'. Is there a way to get around that? > > > > Respectfully, > > Samuel Barney > > > > On Tue, Sep 15, 2015 at 10:48 AM, Drew Varner > wrote: > > Samuel, > > > > You need to make sure you have a compete URI, for example ?http://www.google.com ?. > > > > It looks like you forget the ?http://? portion. > > > > See https://github.com/erlang/otp/blob/a2670f0822fc6729df956c8ec8c381340ff0a5fb/lib/inets/src/http_lib/http_uri.erl#L137-L139 > > > > Cheers, > > Drew > > > >> On Sep 15, 2015, at 6:43 PM, Samuel Barney > wrote: > >> > >> I'm trying to use httpc:request/1, but every single time I call it I get a no_scheme error. I've started inets, and all of the examples I've found say I am doing it correctly. Has anyone had this error before, and how did you resolve it? > >> > >> Respectfully, > >> Samuel Barney > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akrupicka@REDACTED Tue Sep 15 19:58:01 2015 From: akrupicka@REDACTED (Adam Krupicka) Date: Tue, 15 Sep 2015 19:58:01 +0200 Subject: [erlang-questions] distinguish proplists vs. lists of proplists In-Reply-To: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> References: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1442339510-sup-9669@arch.local> Not exactly sure what you're asking, but you can use pattern matching to distinguish a list prom a proplist - assuming your ordinary lists do not contain two-element tuples: handle(List) when is_list(List) -> ...; handle([{_, _} | _] = Proplist) -> .... A. Excerpts from Mr rty ff's message of 2015-09-15 16:58:33 +0000: > > Hi > > I write a function which gets?proplists?or?list?of proplists anf key. if the input is proplists I just use?proplists:get_value?to find a value corresponding to key. But if the input is lists of proplists I need to iterate over all its elements. and look for key in each of them. I amnot sure how to distinguish?lists?vs.?proplists.?lists:is_list?returns?true?for both of them. > > Thanks From kostis@REDACTED Tue Sep 15 20:06:06 2015 From: kostis@REDACTED (Kostis Sagonas) Date: Tue, 15 Sep 2015 20:06:06 +0200 Subject: [erlang-questions] -callback directives with Dialyzer In-Reply-To: References: Message-ID: <55F85E0E.8030905@cs.ntua.gr> On 09/15/2015 07:39 PM, Matthew Fitzpatrick wrote: > > If i run dialyzer with these 3 files, It should complain about a > contract violation right? Well, not so clear... First of all, dialyzer never promised to complain about contract violations; instead, all it promised is that when it does complain, it has every right to do so. Second, the no_spec_dude module that defines the callback, defines a callback with success typing: hey_dude(any()) -> string() which does not violate the callback spec. True, there is some other module that passes an integer() argument there, but this is not a violation of the definition of the callback function. Kostis From drew.varner@REDACTED Tue Sep 15 20:26:01 2015 From: drew.varner@REDACTED (Drew Varner) Date: Tue, 15 Sep 2015 20:26:01 +0200 Subject: [erlang-questions] distinguish proplists vs. lists of proplists In-Reply-To: <1442339510-sup-9669@arch.local> References: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> <1442339510-sup-9669@arch.local> Message-ID: <7FB5964B-A263-43D0-A656-2D483CF3ED5E@redops.org> A property in a proplist can be an atom or a tuple. https://gist.github.com/NineFX/c78ce7bae5b7fc6b0304 This gist works if you are sure that you will either get a list of lists or a proplist. In Adam?s example, the first function clause and guard sequence will match against a list of lists and any proplist. > On Sep 15, 2015, at 7:58 PM, Adam Krupicka wrote: > > > Not exactly sure what you're asking, but you can use pattern matching to > distinguish a list prom a proplist - assuming your ordinary lists do > not contain two-element tuples: > > handle(List) when is_list(List) -> > ...; > handle([{_, _} | _] = Proplist) -> > .... > > > A. > > Excerpts from Mr rty ff's message of 2015-09-15 16:58:33 +0000: >> >> Hi >> >> I write a function which gets proplists or list of proplists anf key. if the input is proplists I just use proplists:get_value to find a value corresponding to key. But if the input is lists of proplists I need to iterate over all its elements. and look for key in each of them. I amnot sure how to distinguish lists vs. proplists. lists:is_list returns true for both of them. >> >> Thanks > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From akrupicka@REDACTED Tue Sep 15 22:12:16 2015 From: akrupicka@REDACTED (Adam Krupicka) Date: Tue, 15 Sep 2015 22:12:16 +0200 Subject: [erlang-questions] distinguish proplists vs. lists of proplists In-Reply-To: <7FB5964B-A263-43D0-A656-2D483CF3ED5E@redops.org> References: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> <1442339510-sup-9669@arch.local> <7FB5964B-A263-43D0-A656-2D483CF3ED5E@redops.org> Message-ID: <1442347433-sup-4917@arch.local> > In Adam?s example, the first function clause and guard sequence will match against a list of lists and any proplist. Oh yeah, missed that (: If you want to distinguish a list maybe containing two-element tuples from a proplist, you're gonna have a bad time. If it's just about telling a list of lists from a list of non-lists, that's easy enough ^^ A. From rvirding@REDACTED Tue Sep 15 22:46:56 2015 From: rvirding@REDACTED (Robert Virding) Date: Tue, 15 Sep 2015 22:46:56 +0200 Subject: [erlang-questions] distinguish proplists vs. lists of proplists In-Reply-To: <1442347433-sup-4917@arch.local> References: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> <1442339510-sup-9669@arch.local> <7FB5964B-A263-43D0-A656-2D483CF3ED5E@redops.org> <1442347433-sup-4917@arch.local> Message-ID: If you flip the order of the clauses it will work better. However there is no safe way of distinguishing them, a prop*list* is just a list where the elements should have a predefined structure. Wouldn't it be better to have two functions, one which works on proplists and one which works on lists of proplists? Or use maps instead of proplists? Robert On 15 Sep 2015 22:12, "Adam Krupicka" wrote: > > In Adam?s example, the first function clause and guard sequence will > match against a list of lists and any proplist. > > Oh yeah, missed that (: > > If you want to distinguish a list maybe containing two-element tuples > from a proplist, you're gonna have a bad time. If it's just about > telling a list of lists from a list of non-lists, that's easy enough ^^ > > A. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Tue Sep 15 23:39:25 2015 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Tue, 15 Sep 2015 23:39:25 +0200 Subject: [erlang-questions] distinguish proplists vs. lists of proplists In-Reply-To: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> Message-ID: <5dc923b0-8b74-431b-b411-cd19f9b3d2f9@email.android.com> Hi, You maybe could use lists:flatten on your entry, do that you will always have a one level depth proplist. Regards Le?15 sept. 2015 18:58, Mr rty ff a ?crit?: > > Hi > I write a function which gets?proplists?or?list?of proplists anf key. if the input is proplists I just use?proplists:get_value?to find a value corresponding to key. But if the input is lists of proplists I need to iterate over all its elements. and look for key in each of them. I amnot sure how to distinguish?lists?vs.?proplists.?lists:is_list?returns?true?for both of them. > Thanks From jasiek.stypka@REDACTED Tue Sep 15 22:06:38 2015 From: jasiek.stypka@REDACTED (Jasiek Stypka) Date: Tue, 15 Sep 2015 22:06:38 +0200 Subject: [erlang-questions] Erlang on Xeon Phi Message-ID: <55f87a4e1cc5a0.36625851@wp.pl> Hello everyone, Does anybody have any experience with running Erlang programs on Intel Xeon Phi coprocessor? I would like to carry out some experiments on this platform concerning multicore computing, however I am struggling to run Erlang VM on that architecture. It seems to be binary incompatible with x86_64 architectures. When I try to run the normal Erlang VM v17.3 on it, I get: /home/me/lib/erlang/bin/erl: line 28: /home/me/lib/erlang/erts-6.1/bin/erlexec: cannot execute binary file /home/me/lib/erlang/bin/erl: line 28: /home/me/lib/erlang/erts-6.1/bin/erlexec: Success So I figured out that I need to use cross compilation to compile the VM specially for this architecture. After some reading and numerous trials and errors, I ended up compiling the EVM with these commands: $ ./configure --host=k1om-unknown-linux-gnu --build=x86_64-pc-linux-gnu --without-termcap --prefix=$HOME/lib/my_erlang_compilation-17.3 CC=icc CFLAGS=-mmic $ make The first command somehow finishes without any errors, but when running `make` I finally get: /tmp/iccvaLP3vas_.s: Assembler messages: /tmp/iccvaLP3vas_.s:25794: Error: `mfence' is not supported on `k1om' /tmp/iccvaLP3vas_.s:25800: Error: `mfence' is not supported on `k1om' /tmp/iccvaLP3vas_.s:26388: Error: `mfence' is not supported on `k1om' /tmp/iccvaLP3vas_.s:26394: Error: `mfence' is not supported on `k1om' make[3]: *** [obj/k1om-unknown-linux-gnu/opt/smp/erl_alloc_util.o] Error 1 This seem to be connected with memory barrier instructions and looks like it's incompatible. Does this mean that the Erlang VM cannot be run on this architecture? Thanks in advance for any suggestions, Jan Stypka From ok@REDACTED Wed Sep 16 07:09:35 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 16 Sep 2015 17:09:35 +1200 Subject: [erlang-questions] distinguish proplists vs. lists of proplists In-Reply-To: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> References: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> Message-ID: <0A63F6C2-6DE2-427B-B9D3-149647EB5746@cs.otago.ac.nz> On 16/09/2015, at 4:58 am, Mr rty ff wrote: > I write a function which gets proplists or list of proplists anf key. WHY? This is just confusing. Just DON'T. As you've pointed out, your function does not know its caller's intent and has to guess. (Was [] an empty proplist or an empty list of proplists?) The caller KNOWS. Or should know. So handle_list_of_proplists(LP) -> ... handle_proplist(P) -> handle_list_of_proplists([P]). Trying to put both computations in a single function is like trying to put both feet into one shoe; succeed or fail it is going to hurt. Let's turn to http://www.erlang.org/doc/man/proplists.html to find out what a proplist is: "Property lists are ordinary lists containing entries in the form of either tuples, whose first elements are keys used for lookup and insertion, or atoms, which work as shorthand for tuples {Atom, true}." Sounds pretty clear. handle_either_case([]) -> your definition had better give the SAME answer for empty proplists and empty lists of proplists; handle_either_case([P|Ps]) when is_list(P) -> handle_list_of_proplists([P|Ps]); handle_either_case(P) -> handle_proplist(P). But wait, there's more: "(Other terms are allowed in the lists, but are ignored by this module.)" Oops. Your task just got impossible. A list of proplists *is* a proplist, just one that contains no keys. The *best* way to do this is NOT to have a single function do both jobs, but have two functions, and get the *caller* to figure it out. The nearest it's possible to get to what you want is to accept only "strict" proplists: - a strict proplist is [] - or [B|P] where P is a strict proplist and + B is an atom (treated as {B,true} or + B is a tuple {_,_}. check_strict_proplist([]) -> ok; check_strict_proplist([B|P]) when is_atom(B) -> check_strict_proplist(P); check_strict_proplist([{_,_}|P]) - check_strict_proplist(P) check_strict_proplist([X|_]) -> {bad_element,X}; check_strict_proplist(X) -> {non_list,X}. and use that to decide what you have. Or you could do something like get_values_anywhere(Key, Prop_Tree) -> get_values_anywhere(Key, Prop_Tree, []). %% get_values_anywhere(Key, Prop_Tree, Vs) = %% get_values_anywhere(Key, Prop_Tree) ++ Vs. get_values_anywhere(Key, [H|T], Vs) -> get_values_anywhere(Key, H, get_values_anywhere(Key, T, Vs)); get_values_anywhere(Key, {Key,V}, Vs) -> [V|Vs]; get_values_anywhere(Key, Key, Vs) -> [true|Vs]; get_values_anywhere(_, _, Vs) -> Vs. Note: I said something *LIKE* that. This code is not tested, and probably isn't what you want anyway. I'm pretty confident it's not what you *need*, because that's not to muddle up the two things in the first place. From ok@REDACTED Wed Sep 16 07:15:55 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 16 Sep 2015 17:15:55 +1200 Subject: [erlang-questions] distinguish proplists vs. lists of proplists In-Reply-To: <7FB5964B-A263-43D0-A656-2D483CF3ED5E@redops.org> References: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> <1442339510-sup-9669@arch.local> <7FB5964B-A263-43D0-A656-2D483CF3ED5E@redops.org> Message-ID: <2420C084-E494-4234-8D11-B7580542738C@cs.otago.ac.nz> On 16/09/2015, at 6:26 am, Drew Varner wrote: > A property in a proplist can be an atom or a tuple. The problem is that the documentation for the proplists module says that the elements of a proplist can be ANYTHING. The documentation is a little confusing. proplists:get_value/[2,3] only pays attention to pairs; other tuples might as well not be there. Does anyone know *why* the proplists module is so tolerant of junk in proplists? Is it because get_value(Key, [{Key,Val}|_], _) -> Val; get_value(Key, [Key|_], _) -> true; get_value(Key, [_|Ps], Default) -> get_value(Key, Ps, Default); get_value(Key, [], Default) -> Default. is the easiest thing to write, or is there a use case for it? From nistrigunya@REDACTED Wed Sep 16 07:11:31 2015 From: nistrigunya@REDACTED (Avinash Dhumane) Date: Wed, 16 Sep 2015 10:41:31 +0530 Subject: [erlang-questions] Tuples Vs Maps Message-ID: I find it more expressive to use maps wherever I have been using tuples. Especially, when it comes to pattern matching. It is precise to put #{some_key := some_value} instead of {_, _, _, {some_key, some_value}, _, _} as a pattern. I only fear efficiency of maps over tuples, in particular the time-cost, not so much of space-cost. How safe will I be if I resort to use only maps freely everywhere? Thanks Avinash -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Sep 16 07:39:23 2015 From: zxq9@REDACTED (zxq9) Date: Wed, 16 Sep 2015 14:39:23 +0900 Subject: [erlang-questions] Tuples Vs Maps In-Reply-To: References: Message-ID: <26232570.am982so2oj@burrito> On Wednesday 16 September 2015 10:41:31 Avinash Dhumane wrote: > I find it more expressive to use maps wherever I have been using tuples. > Especially, when it comes to pattern matching. > > It is precise to put #{some_key := some_value} instead of {_, _, _, > {some_key, some_value}, _, _} as a pattern. > > I only fear efficiency of maps over tuples, in particular the time-cost, > not so much of space-cost. > > How safe will I be if I resort to use only maps freely everywhere? Not sure what you mean by "safety". That could mean a lot of things. In any case, the syntactic difference you referenced calls for records, not maps. Records are a special syntax over tuples, losing none of the speed. Maps are Key:Value stores with generally good lookup times (but make it more annoying than lists of tuples to look up a Key from a known Value). The use cases can overlap occasionally, but more often than not are *made* to overlap arbitrarily when one's mind gets stuck in one mode of thinking or the other. -Craig From vasdeveloper@REDACTED Wed Sep 16 12:22:50 2015 From: vasdeveloper@REDACTED (Theepan) Date: Wed, 16 Sep 2015 15:52:50 +0530 Subject: [erlang-questions] Tuples Vs Maps In-Reply-To: References: Message-ID: The use cases of maps and tuples are completely different, they are efficient in their own space. Maps are efficient in what you are trying to achieve. Theepan On Wed, Sep 16, 2015 at 10:41 AM, Avinash Dhumane wrote: > I find it more expressive to use maps wherever I have been using tuples. > Especially, when it comes to pattern matching. > > It is precise to put #{some_key := some_value} instead of {_, _, _, > {some_key, some_value}, _, _} as a pattern. > > I only fear efficiency of maps over tuples, in particular the time-cost, > not so much of space-cost. > > How safe will I be if I resort to use only maps freely everywhere? > > Thanks > Avinash > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Wed Sep 16 12:52:54 2015 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 16 Sep 2015 12:52:54 +0200 Subject: [erlang-questions] Tuples Vs Maps In-Reply-To: References: Message-ID: On Wed, Sep 16, 2015 at 7:11 AM, Avinash Dhumane wrote: > I find it more expressive to use maps wherever I have been using tuples. > Especially, when it comes to pattern matching. > > It is precise to put #{some_key := some_value} instead of {_, _, _, > {some_key, some_value}, _, _} as a pattern. > > I only fear efficiency of maps over tuples, in particular the time-cost, > not so much of space-cost. > > You've set me off - I'm allergic to the word efficiency Premature worrying about efficiency is not a good idea. You should 1) Build and make correct 2) Measure if you need to 3) Optimise if necessary Step one is the tricky bit - in 38 years of programming I can only recall having to optimise a handful of programs. In the last 20 years computers have become literally millions of times faster. The Erlang of today is millions of times faster than that of 1986 (the first Erlang) - at a guess a factor 10-20 comes from better implementation techniques - the real speedup comes from hardware. (So if you want a real speedup wait 10 years - you do *nothing* it just gets faster ) in ten years time any design decisions you made about tuples vs. maps will be irrelevant - if you use any code you wrote ten years ago you will value it being written as clearly and beautifully as possible. I saw a quote on the notice board at bell labs years ago (not sure who said it) it was. " An incorrect program can be made arbitrarily fast " The tricky bit is making programs that work - and that you can still understand years after you wrote them. /Joe > How safe will I be if I resort to use only maps freely everywhere? > > Thanks > Avinash > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lfredlund@REDACTED Wed Sep 16 13:04:12 2015 From: lfredlund@REDACTED (=?UTF-8?Q?Lars-=c3=85ke_Fredlund?=) Date: Wed, 16 Sep 2015 13:04:12 +0200 Subject: [erlang-questions] Testing api with jsongen and quickcheck In-Reply-To: <55D75EC9.9090301@gmail.com> References: <55D75EC9.9090301@gmail.com> Message-ID: <55F94CAC.4020706@fi.upm.es> Yes, compiling jsongen with later versions of QuickCheck doesn't work. Unfortunately jsongen hasn't been modified to handle some API changes in QuickCheck yet. It is on the todo list, but unlikely to happen before October. Greetings, Lars-Ake On 21/08/15 19:24, Kenneth Lakin wrote: > On 08/21/2015 09:27 AM, Mark Bruch wrote: >> Unfortunately [jsongen] doesn't compile on any of >> my machines. > It fails to compile with rebar3 on my machine, too. > > Stupid question, but do you have QuickCheck installed? The missing > include files that the build whines about seem to be related to QuickCheck. > > I found an eqcmini GitHub project, but that doesn't seem to provide > enough to get jsongen to build. > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From drormein@REDACTED Wed Sep 16 13:38:58 2015 From: drormein@REDACTED (Dror Mein) Date: Wed, 16 Sep 2015 11:38:58 +0000 (UTC) Subject: [erlang-questions] tidier Message-ID: <1974707540.280671.1442403538958.JavaMail.yahoo@mail.yahoo.com> Hi, trying to download tidier, but failing. official link fails. any help will be greatly appreciated. thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Wed Sep 16 16:19:40 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 16 Sep 2015 10:19:40 -0400 Subject: [erlang-questions] distinguish proplists vs. lists of proplists In-Reply-To: <2420C084-E494-4234-8D11-B7580542738C@cs.otago.ac.nz> References: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> <1442339510-sup-9669@arch.local> <7FB5964B-A263-43D0-A656-2D483CF3ED5E@redops.org> <2420C084-E494-4234-8D11-B7580542738C@cs.otago.ac.nz> Message-ID: <20150916141939.GB9409@fhebert-ltm1> On 09/16, Richard A. O'Keefe wrote: >Does anyone know *why* the proplists module is so tolerant of >junk in proplists? Is it because > >get_value(Key, [{Key,Val}|_], _) -> Val; >get_value(Key, [Key|_], _) -> true; >get_value(Key, [_|Ps], Default) -> get_value(Key, Ps, Default); >get_value(Key, [], Default) -> Default. > >is the easiest thing to write, or is there a use case for it? > That simpler implementation wouldn't work to properly respect current type signatures or the currently (partly broken) behaviour. The reason is that it suffers from the fact that I could describe a property's key as {body, weight}, and {body, height}, or maybe full records could be used as keys currently (although they should not) Now the problem is obviously that having only a key has the implicit meaning of {Key, true}, and expansion has to be done dynamically. By virtue of having generic keys, proplists in your version should semantically be able to consider `{body, weight}' as either a key/value pair (where key=body, value=weight), or as a tuple-key that has the value 'true' depending on the specific call and context. That again is kind of fine and I would expect makes sense, but the type specifications maintain that only tuples or atoms are accepted, which yields the following funny result: 1> Proplist = [{{body, height}, 172}, {body, lives}, attribute]. [{{body,height},172},{body,lives},attribute] 2> proplists:get_value({body, height}, Proplist). 172 3> proplists:get_value(attribute, Proplist). true 4> proplists:get_value({body,lives}, Proplist). undefined The actual one instead goes for a bit of a more complex logic making assertions on keys being atoms or tuples: https://github.com/erlang/otp/blob/maint/lib/stdlib/src/proplists.erl#L226-L241 The surprising behaviour is possibly due to checks being too lax, where tuples are valid keys only in part under the current implementation, and they should probably not be (alternatively, they are too strict compared to the current type signatures). The type signatures of proplists and their implementation would probably be due an update so that the type: property() = atom() | tuple() is instead: key() :: atom() property() :: key() | {key(), term()} Currently, though, the proplists module is kind of victim of its own lax implementation and an overly permissive spec, given most of the function in the proplists module don't even expect the list passed in to even contain property() as a type -- only normalize/2 and property/1 put such restrictions. I mean, even property/2 has the documented type signature: property(Key, Value) -> Property Types: Key = Value = term() Property = atom() | {term(), term()} Where the function to create properties does not even enforce creating actual properties defined within the module. From eric.des.courtis@REDACTED Wed Sep 16 17:03:23 2015 From: eric.des.courtis@REDACTED (Eric des Courtis) Date: Wed, 16 Sep 2015 11:03:23 -0400 Subject: [erlang-questions] Erlang Cost Model Message-ID: What is the cost model of all operations in Erlang 18? For example Python has this cost model http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model . I either want a link to a single source for this information or I would like to crowd source the information here now. Thanks for your cooperation. Eric des Courtis -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Wed Sep 16 18:56:32 2015 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 16 Sep 2015 18:56:32 +0200 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: On Wed, Sep 16, 2015 at 5:03 PM, Eric des Courtis wrote: > What is the cost model of all operations in Erlang 18? > > For example Python has this cost model > http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model > . > > I either want a link to a single source for this information or I would like > to crowd source the information here now. Interesting question - the problem with Erlang is that you cannot ignore concurrency - so even if you know the costs of an atomic operation in an isolated process I suspect this is not going to tell you the cost when a few tens of thousands of processes perform that operation. In a singly threaded language (like Python) I guess you could add costs of atomic things together - but in Erlang it's more tricky. I'd love to be able to reason like this: If process P1 takes time T1 and process P2 takes time T2 then it might take time max(T1, T2) on a dual core and about T1+T2 on a single core. But as soon as you start measuring these kind of things the interpretation of the data becomes very difficult. Not much help I'm afraid /Joe > > Thanks for your cooperation. > > Eric des Courtis > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From tony@REDACTED Wed Sep 16 20:07:39 2015 From: tony@REDACTED (Tony Rogvall) Date: Wed, 16 Sep 2015 20:07:39 +0200 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: <82DA7AB6-E0C5-4AAA-8566-D9882A700D66@rogvall.se> Please crowd source the information now. > On 16 sep 2015, at 17:03, Eric des Courtis wrote: > > What is the cost model of all operations in Erlang 18? > > For example Python has this cost model http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model . > > I either want a link to a single source for this information or I would like to crowd source the information here now. > > Thanks for your cooperation. > > Eric des Courtis > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From felixgallo@REDACTED Wed Sep 16 20:37:41 2015 From: felixgallo@REDACTED (Felix Gallo) Date: Wed, 16 Sep 2015 11:37:41 -0700 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: What Joe writes is of course true, but Python programs are generally run on general purpose multitasking operating systems too, so they too have the problem of concurrency, getting their cache forcibly invalidated, etc., etc., it's just that they generally can't do it to themselves. I think it would be awesome for someone to build a site/document that helped (semi-)formally solidify intuitions for people coming to Erlang, because most of the intuitions earned in other languages are not applicable either due to immutability (e.g. "changing" element N of a list), language design (e.g. math on arbitrarilybignums), or semi-opaque foreignness (e.g. ets bag perf characteristics vs. set vs. OTP state vs. process dictionary). Not it! F. On Wed, Sep 16, 2015 at 9:56 AM, Joe Armstrong wrote: > On Wed, Sep 16, 2015 at 5:03 PM, Eric des Courtis > wrote: > > What is the cost model of all operations in Erlang 18? > > > > For example Python has this cost model > > > http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model > > . > > > > I either want a link to a single source for this information or I would > like > > to crowd source the information here now. > > Interesting question - the problem with Erlang is that you cannot > ignore concurrency - so even if you know the costs of an atomic > operation in an isolated process I suspect this is not going to tell you > the cost when a few tens of thousands of processes perform that > operation. > > In a singly threaded language (like Python) I guess you could add costs > of atomic things together - but in Erlang it's more tricky. > > I'd love to be able to reason like this: > > If process P1 takes time T1 and process P2 takes time T2 > > then it might take time max(T1, T2) on a dual core and about > T1+T2 on a single core. > > But as soon as you start measuring these kind of things the > interpretation > of the data becomes very difficult. > > Not much help I'm afraid > > /Joe > > > > > Thanks for your cooperation. > > > > Eric des Courtis > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Wed Sep 16 20:43:23 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 16 Sep 2015 20:43:23 +0200 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: On Wed, Sep 16, 2015 at 5:03 PM, Eric des Courtis < eric.des.courtis@REDACTED> wrote: > For example Python has this cost model > http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model > . > The cost model has to be evaluated against memory read times. An instruction which is in the L1 cache or perhaps even in the register bank is almost never going to give a realistic view of runtimes. So beware of simply summing these things. You need to plot the frequency distribution function to guarantee a stable number, and you should also plot the curve as data grows. Often you have cliffs where you start hitting L3 or DRAM. Also, processor intercommunication tend to be "costly". But if you want to get more than a single core doing work, you need to move work to other cores. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Wed Sep 16 21:03:48 2015 From: vasdeveloper@REDACTED (Theepan) Date: Thu, 17 Sep 2015 00:33:48 +0530 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: I have never heard of such a cost model for Erlang?s basic operations, but most of the time I profiled the code to find out the cost of a code-segment, using tools and functions provided by Erlang/OTP. The Erlang documents list do-s and don?t-s, and from the engineering school I know good programing practices. And most of the time I use common sense of how to write a good piece of code. This is my story, and wrote it to give my one cent on the crowd sourcing part. At an Erlang process level operations can be timed, as they directly map to the operations at the underlying OS. And then you will have to add some extra cost for the process management by the OTP. This extra cost is a function of many variables like number of cores, configuration of schedulers, yielding frequency, length of process queue, GC and so on. There is no straightforward way, but through a deeper analysis they can be determined. Theepan On Wed, Sep 16, 2015 at 8:33 PM, Eric des Courtis < eric.des.courtis@REDACTED> wrote: > What is the cost model of all operations in Erlang 18? > > For example Python has this cost model > http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model > . > > I either want a link to a single source for this information or I would > like to crowd source the information here now. > > Thanks for your cooperation. > > Eric des Courtis > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From darach@REDACTED Wed Sep 16 21:12:35 2015 From: darach@REDACTED (Darach Ennis) Date: Wed, 16 Sep 2015 20:12:35 +0100 Subject: [erlang-questions] Erlang on Xeon Phi In-Reply-To: <55f87a4e1cc5a0.36625851@wp.pl> References: <55f87a4e1cc5a0.36625851@wp.pl> Message-ID: Hi Jasiek, Phi is identified as an x86_64 but is not completely compatible. Looks like (s,l,m)fence need to be replaced by serializing instructions according to: http://lists.lttng.org/pipermail/lttng-dev/2013-December/022111.html Would be very interested if you make progress here! Good luck! Cheers, Darach. On 16 Sep 2015 04:40, "Jasiek Stypka" wrote: > Hello everyone, > > Does anybody have any experience with running Erlang programs on Intel > Xeon Phi coprocessor? I would like to carry out some experiments on this > platform concerning multicore computing, however I am struggling to run > Erlang VM on that architecture. > > It seems to be binary incompatible with x86_64 architectures. When I try > to run the normal Erlang VM v17.3 on it, I get: > /home/me/lib/erlang/bin/erl: line 28: > /home/me/lib/erlang/erts-6.1/bin/erlexec: cannot execute binary file > /home/me/lib/erlang/bin/erl: line 28: > /home/me/lib/erlang/erts-6.1/bin/erlexec: Success > > So I figured out that I need to use cross compilation to compile the VM > specially for this architecture. After some reading and numerous trials and > errors, I ended up compiling the EVM with these commands: > > $ ./configure --host=k1om-unknown-linux-gnu --build=x86_64-pc-linux-gnu > --without-termcap --prefix=$HOME/lib/my_erlang_compilation-17.3 CC=icc > CFLAGS=-mmic > $ make > > The first command somehow finishes without any errors, but when running > `make` I finally get: > > /tmp/iccvaLP3vas_.s: Assembler messages: > /tmp/iccvaLP3vas_.s:25794: Error: `mfence' is not supported on `k1om' > /tmp/iccvaLP3vas_.s:25800: Error: `mfence' is not supported on `k1om' > /tmp/iccvaLP3vas_.s:26388: Error: `mfence' is not supported on `k1om' > /tmp/iccvaLP3vas_.s:26394: Error: `mfence' is not supported on `k1om' > make[3]: *** [obj/k1om-unknown-linux-gnu/opt/smp/erl_alloc_util.o] Error 1 > > This seem to be connected with memory barrier instructions and looks like > it's incompatible. Does this mean that the Erlang VM cannot be run on this > architecture? > > Thanks in advance for any suggestions, > Jan Stypka > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stu.bailey@REDACTED Wed Sep 16 21:14:18 2015 From: stu.bailey@REDACTED (Stu Bailey) Date: Wed, 16 Sep 2015 12:14:18 -0700 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: Maybe the Erlang community in collaboration with the larger massive concurrency community should to come up with (or has and I don't know about them) some standard concurrency benchmarks? Such an effort may be somewhat (maybe only slightly) analogous to the various transaction processing benchmarks that the relational database community developed. Just as a starting point, here is some work specific to Scala in that direction: http://www.infoq.com/articles/benchmarking-jvm On Wed, Sep 16, 2015 at 12:03 PM, Theepan wrote: > I have never heard of such a cost model for Erlang?s basic operations, but > most of the time I profiled the code to find out the cost of a > code-segment, using tools and functions provided by Erlang/OTP. The Erlang > documents list do-s and don?t-s, and from the engineering school I know > good programing practices. And most of the time I use common sense of how > to write a good piece of code. This is my story, and wrote it to give my > one cent on the crowd sourcing part. > > > > At an Erlang process level operations can be timed, as they directly map > to the operations at the underlying OS. And then you will have to add some > extra cost for the process management by the OTP. This extra cost is a > function of many variables like number of cores, configuration of > schedulers, yielding frequency, length of process queue, GC and so on. > There is no straightforward way, but through a deeper analysis they can be > determined. > > > > Theepan > > On Wed, Sep 16, 2015 at 8:33 PM, Eric des Courtis < > eric.des.courtis@REDACTED> wrote: > >> What is the cost model of all operations in Erlang 18? >> >> For example Python has this cost model >> http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model >> . >> >> I either want a link to a single source for this information or I would >> like to crowd source the information here now. >> >> Thanks for your cooperation. >> >> Eric des Courtis >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Wed Sep 16 22:09:50 2015 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 16 Sep 2015 22:09:50 +0200 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: On Wed, Sep 16, 2015 at 9:03 PM, Theepan wrote: > I have never heard of such a cost model for Erlang?s basic operations, but > most of the time I profiled the code to find out the cost of a code-segment, > using tools and functions provided by Erlang/OTP. The Erlang documents list > do-s and don?t-s, and from the engineering school I know good programing > practices. And most of the time I use common sense of how to write a good > piece of code. This is my story, and wrote it to give my one cent on the > crowd sourcing part. > > > > At an Erlang process level operations can be timed, as they directly map to > the operations at the underlying OS. And then you will have to add some > extra cost for the process management by the OTP. This extra cost is a > function of many variables like number of cores, configuration of > schedulers, yielding frequency, length of process queue, GC and so on. Right - the problem is not measuring how long the primitives take it is making sense of the measurements. Even arithmetic is problematic - how long does an integer multiply take? - answer "it depends" - Erlang has bignums not fixnums. The python cost model gives a time for multiplying integers, but for bignums the time will depend upon the size of the bignums and this depends upon the algorithm. Just looking at code, you can't see how large the integers are. >From what I've seen extrapolation from "programs in the small" (ie answering question like - "how long does an integer multiply take") to "programs in the large" (ie millions of processes collaborating to provide a service) is very hard. Cheers /Joe There > is no straightforward way, but through a deeper analysis they can be > determined. > > > > Theepan > > > On Wed, Sep 16, 2015 at 8:33 PM, Eric des Courtis > wrote: >> >> What is the cost model of all operations in Erlang 18? >> >> For example Python has this cost model >> http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model >> . >> >> I either want a link to a single source for this information or I would >> like to crowd source the information here now. >> >> Thanks for your cooperation. >> >> Eric des Courtis >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From erlang@REDACTED Wed Sep 16 22:24:06 2015 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 16 Sep 2015 22:24:06 +0200 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: On Wed, Sep 16, 2015 at 9:14 PM, Stu Bailey wrote: > Maybe the Erlang community in collaboration with the larger massive > concurrency community should to come up with (or has and I don't know about > them) some standard concurrency benchmarks? Such an effort may be somewhat > (maybe only slightly) analogous to the various transaction processing > benchmarks that the relational database community developed. > Hi Stu, ... Again I'd not read too much into the results - Akka message passing will outperform Erlang (we know this) but they (Erlang and Akka) are not doing the same thing. Erlang code is designed for 'on-the-fly' upgrading so all exported function calls go through an indirect pointer, this slows everything down, but has implications for continuos operation. Java code can be turned for message passing so that it is far faster than Erlang - *but* a single error can crash the entire system. So sure we can measure what at first sight appears to be "the same thing" (like message passing times) - but which has entirely different behaviour. Many of the things which I think are important in designing a system like "fault tolerance" "scalability" etc. are difficult to measure. How do we measure "fault tolerance" for example? Can we say this system is twice as fault-tolerant as this other system? So yes - we can measure things like message passing times and so on, but they don't help us answer questions like "is it scalable" and so on. By all means perform concurrency benchmarking - but I'd take great care when interpreting the results. The numbers are easy to find - but the interpretation is very difficult. Cheers /Joe > Just as a starting point, here is some work specific to Scala in that > direction: > > http://www.infoq.com/articles/benchmarking-jvm > > On Wed, Sep 16, 2015 at 12:03 PM, Theepan wrote: >> >> I have never heard of such a cost model for Erlang?s basic operations, but >> most of the time I profiled the code to find out the cost of a code-segment, >> using tools and functions provided by Erlang/OTP. The Erlang documents list >> do-s and don?t-s, and from the engineering school I know good programing >> practices. And most of the time I use common sense of how to write a good >> piece of code. This is my story, and wrote it to give my one cent on the >> crowd sourcing part. >> >> >> >> At an Erlang process level operations can be timed, as they directly map >> to the operations at the underlying OS. And then you will have to add some >> extra cost for the process management by the OTP. This extra cost is a >> function of many variables like number of cores, configuration of >> schedulers, yielding frequency, length of process queue, GC and so on. There >> is no straightforward way, but through a deeper analysis they can be >> determined. >> >> >> >> Theepan >> >> >> On Wed, Sep 16, 2015 at 8:33 PM, Eric des Courtis >> wrote: >>> >>> What is the cost model of all operations in Erlang 18? >>> >>> For example Python has this cost model >>> http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model >>> . >>> >>> I either want a link to a single source for this information or I would >>> like to crowd source the information here now. >>> >>> Thanks for your cooperation. >>> >>> Eric des Courtis >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From pierrefenoll@REDACTED Wed Sep 16 22:27:23 2015 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Wed, 16 Sep 2015 13:27:23 -0700 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: On a side note I would be happy if we had a way to add a documentation clause that advertises the complexity of the documented function. Maybe the complexity can even be computed automatically. And combining cost of atomic actions could give an idea of the cost of these complexities constants. > On 16 Sep 2015, at 13:09, Joe Armstrong wrote: > >> On Wed, Sep 16, 2015 at 9:03 PM, Theepan wrote: >> I have never heard of such a cost model for Erlang?s basic operations, but >> most of the time I profiled the code to find out the cost of a code-segment, >> using tools and functions provided by Erlang/OTP. The Erlang documents list >> do-s and don?t-s, and from the engineering school I know good programing >> practices. And most of the time I use common sense of how to write a good >> piece of code. This is my story, and wrote it to give my one cent on the >> crowd sourcing part. >> >> >> >> At an Erlang process level operations can be timed, as they directly map to >> the operations at the underlying OS. And then you will have to add some >> extra cost for the process management by the OTP. This extra cost is a >> function of many variables like number of cores, configuration of >> schedulers, yielding frequency, length of process queue, GC and so on. > > Right - the problem is not measuring how long the primitives take > it is making sense of the measurements. > > Even arithmetic is problematic - how long does an integer multiply take? > - answer "it depends" - Erlang has bignums not fixnums. > The python cost model gives a time for multiplying integers, but for > bignums the time will depend upon the size of the bignums and this > depends upon the algorithm. Just looking at code, you can't see how > large the integers are. > > From what I've seen extrapolation from "programs in the small" (ie > answering question like - "how long does an integer multiply take") to > "programs in the large" (ie millions of processes collaborating to > provide a service) is very hard. > > Cheers > > /Joe > > > > There >> is no straightforward way, but through a deeper analysis they can be >> determined. >> >> >> >> Theepan >> >> >> On Wed, Sep 16, 2015 at 8:33 PM, Eric des Courtis >> wrote: >>> >>> What is the cost model of all operations in Erlang 18? >>> >>> For example Python has this cost model >>> http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model >>> . >>> >>> I either want a link to a single source for this information or I would >>> like to crowd source the information here now. >>> >>> Thanks for your cooperation. >>> >>> Eric des Courtis >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From erlang@REDACTED Wed Sep 16 22:56:37 2015 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 16 Sep 2015 22:56:37 +0200 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: On Wed, Sep 16, 2015 at 10:27 PM, Pierre Fenoll wrote: > On a side note I would be happy if we had a way to add a documentation clause that advertises the complexity of the documented function. And I would be *even happier* if the *specifications* had information like "this function must take no more than XXXX ms." - this I have virtually *never* seen. (I've even asked the people who wrote the specs. and they usually haven't a clue, and I don't view "as quickly as possible" as a good answer). With the absence of a "target" value in the specification it is pretty meaningless to measure *anything*. Mostly specification say very little about non-functional behaviour. They specify functional behaviour but not non-functional behaviour (how long, how much memory, ...) - most systems I've worked with failed because the non-functional behaviour was terrible (they crashed). I've seen huge projects that were functionally perfect - the functions computed the correct values - but oops - the system crashed after a while. All I can say is "beware measuring the wrong thing" /Joe > > Maybe the complexity can even be computed automatically. Maybe the moon is made of blue cheese. > > And combining cost of atomic actions could give an idea of the cost of these complexities constants. If computing was a science this might be so, right now it's a craft. Programs don't obey basic algebraic properties. If A takes time Ta and B Tb how long will A+B take? Ta + Tb ???? It would be nice if this were true but it's not. Good 'ol Turing's halting problem illustrated the fact that no algorithm can decide if another program will halt or not, let alone figure out how long it would take ... /Joe > >> On 16 Sep 2015, at 13:09, Joe Armstrong wrote: >> >>> On Wed, Sep 16, 2015 at 9:03 PM, Theepan wrote: >>> I have never heard of such a cost model for Erlang?s basic operations, but >>> most of the time I profiled the code to find out the cost of a code-segment, >>> using tools and functions provided by Erlang/OTP. The Erlang documents list >>> do-s and don?t-s, and from the engineering school I know good programing >>> practices. And most of the time I use common sense of how to write a good >>> piece of code. This is my story, and wrote it to give my one cent on the >>> crowd sourcing part. >>> >>> >>> >>> At an Erlang process level operations can be timed, as they directly map to >>> the operations at the underlying OS. And then you will have to add some >>> extra cost for the process management by the OTP. This extra cost is a >>> function of many variables like number of cores, configuration of >>> schedulers, yielding frequency, length of process queue, GC and so on. >> >> Right - the problem is not measuring how long the primitives take >> it is making sense of the measurements. >> >> Even arithmetic is problematic - how long does an integer multiply take? >> - answer "it depends" - Erlang has bignums not fixnums. >> The python cost model gives a time for multiplying integers, but for >> bignums the time will depend upon the size of the bignums and this >> depends upon the algorithm. Just looking at code, you can't see how >> large the integers are. >> >> From what I've seen extrapolation from "programs in the small" (ie >> answering question like - "how long does an integer multiply take") to >> "programs in the large" (ie millions of processes collaborating to >> provide a service) is very hard. >> >> Cheers >> >> /Joe >> >> >> >> There >>> is no straightforward way, but through a deeper analysis they can be >>> determined. >>> >>> >>> >>> Theepan >>> >>> >>> On Wed, Sep 16, 2015 at 8:33 PM, Eric des Courtis >>> wrote: >>>> >>>> What is the cost model of all operations in Erlang 18? >>>> >>>> For example Python has this cost model >>>> http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model >>>> . >>>> >>>> I either want a link to a single source for this information or I would >>>> like to crowd source the information here now. >>>> >>>> Thanks for your cooperation. >>>> >>>> Eric des Courtis >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From mjtruog@REDACTED Wed Sep 16 22:57:03 2015 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 16 Sep 2015 13:57:03 -0700 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: <55F9D79F.70406@gmail.com> I have been using the source code at https://github.com/okeuday/erlbench for basic micro-benchmarks, mainly for data structure comparisons. The code is a bit basic, but is able to make sure the iterations are large enough and are repeated often enough, to avoid some GC variation that can affect results. Then it may be possible to make conclusions based on the hardware used for testing. The concurrency testing would require tests be done in parallel to determine how the timing result is affected, but a basic run of low-level operations would provide a starting point. As mentioned elsewhere, making sure you are testing bignums (anything over 26bits in Erlang, likely different in Python) is important, and other details that avoid erroneous conclusions. The influence of concurrency is more important in Erlang, due to the support provided by Erlang, but latency testing of concurrency is simpler with higher-level connections to the Erlang server, rather than low-level testing that is unlikely to provide realistic GC usage. The hardware being used for the concurrency testing will then provide more variation and it is necessary to make sure the hardware is not being throttled due to excessive heat to avoid skewing the results. On 09/16/2015 08:03 AM, Eric des Courtis wrote: > What is the cost model of all operations in Erlang 18? > > For example Python has this cost model http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model . > > I either want a link to a single source for this information or I would like to crowd source the information here now. > > Thanks for your cooperation. > > Eric des Courtis > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From anders.nygren@REDACTED Wed Sep 16 22:57:52 2015 From: anders.nygren@REDACTED (Anders Nygren) Date: Wed, 16 Sep 2015 15:57:52 -0500 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: I have done a lot of measurements in my life and initially it was "easy", no caches, virtual memory, or frequency scaling. But even then You did a measurement of X and got a result. Then You thought a little about it and realized that You had actually measured Y. So You modified the test and ran it again and got a new result, that You soon realized was for Z. Repeat forever. I gave up on these activities a few years ago when I found that the response times decreased with increased load. Because of frequency scaling. On Wed, Sep 16, 2015 at 3:27 PM, Pierre Fenoll wrote: > On a side note I would be happy if we had a way to add a documentation > clause that advertises the complexity of the documented function. > > Maybe the complexity can even be computed automatically. > > And combining cost of atomic actions could give an idea of the cost of > these complexities constants. > > > On 16 Sep 2015, at 13:09, Joe Armstrong wrote: > > > >> On Wed, Sep 16, 2015 at 9:03 PM, Theepan > wrote: > >> I have never heard of such a cost model for Erlang?s basic operations, > but > >> most of the time I profiled the code to find out the cost of a > code-segment, > >> using tools and functions provided by Erlang/OTP. The Erlang documents > list > >> do-s and don?t-s, and from the engineering school I know good programing > >> practices. And most of the time I use common sense of how to write a > good > >> piece of code. This is my story, and wrote it to give my one cent on the > >> crowd sourcing part. > >> > >> > >> > >> At an Erlang process level operations can be timed, as they directly > map to > >> the operations at the underlying OS. And then you will have to add some > >> extra cost for the process management by the OTP. This extra cost is a > >> function of many variables like number of cores, configuration of > >> schedulers, yielding frequency, length of process queue, GC and so on. > > > > Right - the problem is not measuring how long the primitives take > > it is making sense of the measurements. > > > > Even arithmetic is problematic - how long does an integer multiply take? > > - answer "it depends" - Erlang has bignums not fixnums. > > The python cost model gives a time for multiplying integers, but for > > bignums the time will depend upon the size of the bignums and this > > depends upon the algorithm. Just looking at code, you can't see how > > large the integers are. > > > > From what I've seen extrapolation from "programs in the small" (ie > > answering question like - "how long does an integer multiply take") to > > "programs in the large" (ie millions of processes collaborating to > > provide a service) is very hard. > > > > Cheers > > > > /Joe > > > > > > > > There > >> is no straightforward way, but through a deeper analysis they can be > >> determined. > >> > >> > >> > >> Theepan > >> > >> > >> On Wed, Sep 16, 2015 at 8:33 PM, Eric des Courtis > >> wrote: > >>> > >>> What is the cost model of all operations in Erlang 18? > >>> > >>> For example Python has this cost model > >>> > http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model > >>> . > >>> > >>> I either want a link to a single source for this information or I would > >>> like to crowd source the information here now. > >>> > >>> Thanks for your cooperation. > >>> > >>> Eric des Courtis > >>> > >>> _______________________________________________ > >>> erlang-questions mailing list > >>> erlang-questions@REDACTED > >>> http://erlang.org/mailman/listinfo/erlang-questions > >> > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Wed Sep 16 23:00:58 2015 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 16 Sep 2015 14:00:58 -0700 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: <55F9D79F.70406@gmail.com> References: <55F9D79F.70406@gmail.com> Message-ID: <55F9D88A.9090609@gmail.com> Sorry, Erlang uses 27bits for integers before using bignums. Not sure where that is documented. On 09/16/2015 01:57 PM, Michael Truog wrote: > I have been using the source code at https://github.com/okeuday/erlbench for basic micro-benchmarks, mainly for data structure comparisons. The code is a bit basic, but is able to make sure the iterations are large enough and are repeated often enough, to avoid some GC variation that can affect results. Then it may be possible to make conclusions based on the hardware used for testing. The concurrency testing would require tests be done in parallel to determine how the timing result is affected, but a basic run of low-level operations would provide a starting point. > > As mentioned elsewhere, making sure you are testing bignums (anything over 26bits in Erlang, likely different in Python) is important, and other details that avoid erroneous conclusions. The influence of concurrency is more important in Erlang, due to the support provided by Erlang, but latency testing of concurrency is simpler with higher-level connections to the Erlang server, rather than low-level testing that is unlikely to provide realistic GC usage. The hardware being used for the concurrency testing will then provide more variation and it is necessary to make sure the hardware is not being throttled due to excessive heat to avoid skewing the results. > > On 09/16/2015 08:03 AM, Eric des Courtis wrote: >> What is the cost model of all operations in Erlang 18? >> >> For example Python has this cost model http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model . >> >> I either want a link to a single source for this information or I would like to crowd source the information here now. >> >> Thanks for your cooperation. >> >> Eric des Courtis >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony@REDACTED Wed Sep 16 23:11:45 2015 From: tony@REDACTED (Tony Rogvall) Date: Wed, 16 Sep 2015 23:11:45 +0200 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: <55F9D88A.9090609@gmail.com> References: <55F9D79F.70406@gmail.com> <55F9D88A.9090609@gmail.com> Message-ID: On a 64 bit machine that is 60 bits. Just to make things easy. /Tony > On 16 sep 2015, at 23:00, Michael Truog wrote: > > Sorry, Erlang uses 27bits for integers before using bignums. Not sure where that is documented. > > On 09/16/2015 01:57 PM, Michael Truog wrote: >> I have been using the source code at https://github.com/okeuday/erlbench for basic micro-benchmarks, mainly for data structure comparisons. The code is a bit basic, but is able to make sure the iterations are large enough and are repeated often enough, to avoid some GC variation that can affect results. Then it may be possible to make conclusions based on the hardware used for testing. The concurrency testing would require tests be done in parallel to determine how the timing result is affected, but a basic run of low-level operations would provide a starting point. >> >> As mentioned elsewhere, making sure you are testing bignums (anything over 26bits in Erlang, likely different in Python) is important, and other details that avoid erroneous conclusions. The influence of concurrency is more important in Erlang, due to the support provided by Erlang, but latency testing of concurrency is simpler with higher-level connections to the Erlang server, rather than low-level testing that is unlikely to provide realistic GC usage. The hardware being used for the concurrency testing will then provide more variation and it is necessary to make sure the hardware is not being throttled due to excessive heat to avoid skewing the results. >> >> On 09/16/2015 08:03 AM, Eric des Courtis wrote: >>> What is the cost model of all operations in Erlang 18? >>> >>> For example Python has this cost model http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model . >>> >>> I either want a link to a single source for this information or I would like to crowd source the information here now. >>> >>> Thanks for your cooperation. >>> >>> Eric des Courtis >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eric.des.courtis@REDACTED Thu Sep 17 01:25:41 2015 From: eric.des.courtis@REDACTED (Eric des Courtis) Date: Wed, 16 Sep 2015 19:25:41 -0400 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: <55F9D79F.70406@gmail.com> <55F9D88A.9090609@gmail.com> Message-ID: As a first step lets focus on the complexity of the operations (ignoring any associated constants or specific time measurements). Just so newcomers to Erlang can validate their assumptions. I will start: Adding to the beginning of a list is O(1) Adding to the end of a list is O(n) Sending a large binary to another node is O(n) Sending a large binary to other processes on the same node is O(1) etc... On Wed, Sep 16, 2015 at 5:11 PM, Tony Rogvall wrote: > On a 64 bit machine that is 60 bits. Just to make things easy. > > /Tony > > On 16 sep 2015, at 23:00, Michael Truog wrote: > > Sorry, Erlang uses 27bits for integers before using bignums. Not sure > where that is documented. > > On 09/16/2015 01:57 PM, Michael Truog wrote: > > I have been using the source code at https://github.com/okeuday/erlbench > for basic micro-benchmarks, mainly for data structure comparisons. The > code is a bit basic, but is able to make sure the iterations are large > enough and are repeated often enough, to avoid some GC variation that can > affect results. Then it may be possible to make conclusions based on the > hardware used for testing. The concurrency testing would require tests be > done in parallel to determine how the timing result is affected, but a > basic run of low-level operations would provide a starting point. > > As mentioned elsewhere, making sure you are testing bignums (anything over > 26bits in Erlang, likely different in Python) is important, and other > details that avoid erroneous conclusions. The influence of concurrency is > more important in Erlang, due to the support provided by Erlang, but > latency testing of concurrency is simpler with higher-level connections to > the Erlang server, rather than low-level testing that is unlikely to > provide realistic GC usage. The hardware being used for the concurrency > testing will then provide more variation and it is necessary to make sure > the hardware is not being throttled due to excessive heat to avoid skewing > the results. > > On 09/16/2015 08:03 AM, Eric des Courtis wrote: > > What is the cost model of all operations in Erlang 18? > > For example Python has this cost model > http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model > . > > I either want a link to a single source for this information or I would > like to crowd source the information here now. > > Thanks for your cooperation. > > Eric des Courtis > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Thu Sep 17 06:17:24 2015 From: zxq9@REDACTED (zxq9) Date: Thu, 17 Sep 2015 13:17:24 +0900 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: <2423630.4x47EVrrNS@burrito> On Wednesday 16 September 2015 19:25:41 Eric des Courtis wrote: > As a first step lets focus on the complexity of the operations (ignoring > any associated constants or specific time measurements). Just so newcomers > to Erlang can validate their assumptions. > > I will start: > > Adding to the beginning of a list is O(1) > Adding to the end of a list is O(n) > > Sending a large binary to another node is O(n) > Sending a large binary to other processes on the same node is O(1) > > etc... That is going to wind up reading very similarly to the cost model for any language that uses similar data structures and executes in a managed environment. And its not very useful. Here is why: "Just so newcomers to Erlang can validate their assumptions." Their assumptions are going to be totally invalid (which is different from being wrong) because using concurrent processes as a basic component of a model of computation is *a different model of computation* than these cost models assume. So whatever these "newcomer assumptions" are will not make sense in real programs, unless newcomers only write sequential Erlang programs. This is like trying to tune a violin by drum tempo or tune a drum to output from a spectroscope. Sure, each involves a "frequency", but the underlying concept is so different in each case they simply don't compare. The only place cost models make sense is: 1- Dealing with a technically ignorant bureaucracy that doesn't understand why this makes no sense, but have the power to can your project if you can't justify it to some Soviet standard 2- Trying to squeeze the last bit of performance out of a bottleneck you can't imagine a way around Case 1 is in error, as Erlang will fail in most competitions that measure sequential speed. Meh. Case 2 is generally in error as well, as the goal should usually be to avoid having bottlenecks to begin with, not making bottlenecks more performant. (There are standard answers for that one time when you really, genuinely need to make a pig fly.) The mapping of operations to cost in terms of scheduler reductions is the closest thing to a meaningful number I think there is. (If this is wrong someone please correct me or expand on this -- I am sort of curious, though only morbidly so. But don't go on and on about how l33t dirty schedulers and NIFs are.) In actual programs I've written (and am writing) that handle actual problems for actual users I've had to think about this exactly one time, ever, and that turned out to be a big fat waste of time once I stepped back and took a second look from above. What is the purpose of such a cost model? What is its use? Why is this needed? The concept is so different that I think this is probably a sort of X-Y problem. If, instead of asking for a cost model, the intended utility of the cost model were explained perhaps more useful things would start popping up on the ML. -Craig From ahe.sanath@REDACTED Thu Sep 17 09:09:39 2015 From: ahe.sanath@REDACTED (Sanath Prasanna) Date: Thu, 17 Sep 2015 12:39:39 +0530 Subject: [erlang-questions] Unexpected behavior of HTTPC module - tls_connection process take more memory In-Reply-To: References: Message-ID: Hi Ingela, Even apply your patch, *still problem is persist.* Any more suggestions to solve this unexpected behavior ? Br, Robert On Tue, Sep 8, 2015 at 1:52 PM, Sanath Prasanna wrote: > Hi Ingela. > Tx a lot for your help & patch related to that.I'll inform you the result > after applying & testing patch. > Br, > Robert > > On Tue, Sep 8, 2015 at 1:31 PM, Ingela Andin > wrote: > >> Hi! >> >> It could be an bug that in the ssl application that I just fixed. The >> default session cache >> was violating the API, and this in turn made the mechanism for not >> registering a lot of equivalent >> sessions in the client fail. >> >> Here is the patch: >> >> >> diff --git a/lib/ssl/src/ssl_session.erl b/lib/ssl/src/ssl_session.erl >> index 1770faf..0d6cc93 100644 >> --- a/lib/ssl/src/ssl_session.erl >> +++ b/lib/ssl/src/ssl_session.erl >> @@ -100,14 +100,14 @@ select_session([], _, _) -> >> no_session; >> select_session(Sessions, #ssl_options{ciphers = Ciphers}, OwnCert) -> >> IsNotResumable = >> - fun([_Id, Session]) -> >> + fun(Session) -> >> not (resumable(Session#session.is_resumable) andalso >> lists:member(Session#session.cipher_suite, Ciphers) >> andalso (OwnCert == Session#session.own_certificate)) >> end, >> case lists:dropwhile(IsNotResumable, Sessions) of >> [] -> no_session; >> - [[Id, _]|_] -> Id >> + [Session | _] -> Session#session.session_id >> end. >> >> is_resumable(_, _, #ssl_options{reuse_sessions = false}, _, _, _, _) -> >> diff --git a/lib/ssl/src/ssl_session_cache.erl >> b/lib/ssl/src/ssl_session_cach >> e.erl >> index 11ed310..cfc48cd 100644 >> --- a/lib/ssl/src/ssl_session_cache.erl >> +++ b/lib/ssl/src/ssl_session_cache.erl >> @@ -83,7 +83,7 @@ foldl(Fun, Acc0, Cache) -> >> %%-------------------------------------------------------------------- >> select_session(Cache, PartialKey) -> >> ets:select(Cache, >> - [{{{PartialKey,'$1'}, '$2'},[],['$$']}]). >> + [{{{PartialKey,'_'}, '$1'},[],['$1']}]). >> >> %%-------------------------------------------------------------------- >> %%% Internal functions >> >> >> Regards Ingela Erlang/OTP team - Ericsson AB >> >> >> 2015-09-07 5:52 GMT+02:00 Sanath Prasanna : >> >>> Hi all, >>> >>> I am running HTTP client using httpc Module to send both http and https >>> requests. normally sending arround 300 request per second without any >>> issue. however sometimes erlang node become very slow responsive. at that >>> time server load average is very high and using etop can identify >>> "tls_connection" process take more memory. when restart the erlang node its >>> become normal. as per my investigation normal time memory, processors, >>> loadAverage is not increasing. following is the HTTP request config >>> >>> >>> >>> httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync, false}]) >>> >>> >>> >>> below is the etop output at that time >>> >>> >>> >>> procs 1134 processes 1504844 >>> code 9309 >>> >>> runq 0 atom 420 >>> ets 29692 >>> >>> >>> >>> Pid Name or Initial Func Time Reds Memory MsgQ >>> Current Function >>> >>> >>> ---------------------------------------------------------------------------------------- >>> >>> <5490.26428.14>tls_connection:init/ '-' 733224580768 0 >>> gen_fsm:loop/7 >>> >>> <5490.26429.14>tls_connection:init/ '-' 1328524580768 0 >>> gen_fsm:loop/7 >>> >>> <5490.26430.14>tls_connection:init/ '-' 528924580768 0 >>> gen_fsm:loop/7 >>> >>> <5490.26431.14>tls_connection:init/ '-' 1432224580768 0 >>> gen_fsm:loop/7 >>> >>> <5490.26432.14>tls_connection:init/ '-' 024580768 0 >>> gen_fsm:loop/7 >>> >>> <5490.26433.14>tls_connection:init/ '-' 024580768 0 >>> gen_fsm:loop/7 >>> >>> <5490.26434.14>tls_connection:init/ '-' 024580768 0 >>> gen_fsm:loop/7 >>> >>> <5490.26435.14>tls_connection:init/ '-' 024580768 0 >>> gen_fsm:loop/7 >>> >>> <5490.26436.14>tls_connection:init/ '-' 024580768 0 >>> gen_fsm:loop/7 >>> >>> >>> can some one help me to solve this issue? >>> >>> Br, >>> >>> A.H.E. Robert >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Eric.desCourtis@REDACTED Thu Sep 17 17:22:05 2015 From: Eric.desCourtis@REDACTED (Eric des Courtis) Date: Thu, 17 Sep 2015 11:22:05 -0400 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: <2423630.4x47EVrrNS@burrito> References: <2423630.4x47EVrrNS@burrito> Message-ID: Regardless if concurrency and parallelism plays a role *I still need to know if an operation in a sequential process I am working on is O(1) or O(n)* or am I missing something? On Thu, Sep 17, 2015 at 12:17 AM, zxq9 wrote: > On Wednesday 16 September 2015 19:25:41 Eric des Courtis wrote: > > As a first step lets focus on the complexity of the operations (ignoring > > any associated constants or specific time measurements). Just so > newcomers > > to Erlang can validate their assumptions. > > > > I will start: > > > > Adding to the beginning of a list is O(1) > > Adding to the end of a list is O(n) > > > > Sending a large binary to another node is O(n) > > Sending a large binary to other processes on the same node is O(1) > > > > etc... > > That is going to wind up reading very similarly to the cost model for any > language that uses similar data structures and executes in a managed > environment. > > And its not very useful. > > Here is why: "Just so newcomers to Erlang can validate their assumptions." > > Their assumptions are going to be totally invalid (which is different from > being wrong) because using concurrent processes as a basic component of a > model of computation is *a different model of computation* than these cost > models assume. So whatever these "newcomer assumptions" are will not make > sense in real programs, unless newcomers only write sequential Erlang > programs. > > This is like trying to tune a violin by drum tempo or tune a drum to > output from a spectroscope. Sure, each involves a "frequency", but the > underlying concept is so different in each case they simply don't compare. > > The only place cost models make sense is: > 1- Dealing with a technically ignorant bureaucracy that doesn't understand > why this makes no sense, but have the power to can your project if you > can't justify it to some Soviet standard > 2- Trying to squeeze the last bit of performance out of a bottleneck you > can't imagine a way around > > Case 1 is in error, as Erlang will fail in most competitions that measure > sequential speed. Meh. > > Case 2 is generally in error as well, as the goal should usually be to > avoid having bottlenecks to begin with, not making bottlenecks more > performant. (There are standard answers for that one time when you really, > genuinely need to make a pig fly.) > > The mapping of operations to cost in terms of scheduler reductions is the > closest thing to a meaningful number I think there is. (If this is wrong > someone please correct me or expand on this -- I am sort of curious, though > only morbidly so. But don't go on and on about how l33t dirty schedulers > and NIFs are.) > > In actual programs I've written (and am writing) that handle actual > problems for actual users I've had to think about this exactly one time, > ever, and that turned out to be a big fat waste of time once I stepped back > and took a second look from above. > > What is the purpose of such a cost model? What is its use? Why is this > needed? > > The concept is so different that I think this is probably a sort of X-Y > problem. If, instead of asking for a cost model, the intended utility of > the cost model were explained perhaps more useful things would start > popping up on the ML. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Thu Sep 17 17:56:47 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 17 Sep 2015 11:56:47 -0400 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: <2423630.4x47EVrrNS@burrito> Message-ID: <20150917155645.GC9409@fhebert-ltm1> On 09/17, Eric des Courtis wrote: >Regardless if concurrency and parallelism plays a role *I still need to >know if an operation in a sequential process I am working on is O(1) or >O(n)* or am I missing something? These values are way more worthwhile as a property of libraries than language, since they're far more impacted by the choice of algorithm taken there. Let's look at a few modules: 1. Queue "all operations has an amortized O(1) running time, except len/1, join/2, split/2, filter/2 and member/2 that have O(n). To minimize the size of a queue minimizing the amount of garbage built by queue operations, the queues do not contain explicit length information, and that is why len/1 is O(n)." 2. Maps undefined in the module doc, but from the EEP (http://www.erlang.org/eeps/eep-0043.html): "has at most O(log N) time complexity in insert and lookup operations, where N is the number of key-value associations." 3. Dict undefined in docs. I know that it is a very flat tree (bunch of buckets) acting as a hash map, so O(log N) 4. Array undefined in docs. Also a very flat tree. O(Log N), where N is the size of the array, and if I recall, the tree has a branching factor of 10 to compromise between the speed of lookups and garbage generated when modifying it. It's described in the source only, sadly. 5. Gb_trees Doc compares them to AVL trees, without storage overhead and with better performance. O(log N) it is (those are the worst cases for AVL trees) 6. Sets They're using the same mechanism as dicts, so O(Log N) 7. Gb_sets Same mechanism as gb_trees, so O(log N) 8. ETS Doc says: "These provide the ability to store very large quantities of data in an Erlang runtime system, and to have constant access time to the data. (In the case of ordered_set, see below, access time is proportional to the logarithm of the number of objects stored)." 9. Orddict Doc says " An orddict is a representation of a dictionary, where a list of pairs is used to store the keys and values. The list is ordered after the keys." lists are O(N) traversal and insertion, so let's consider this the best case. 10. ordsets implemented over orddict, O(N) too. 11. sofs despite having the most cryptic documentation, it does mention: "The execution time of the functions of this module is dominated by the time it takes to sort lists. When no sorting is needed, the execution time is in the worst case proportional to the sum of the sizes of the input arguments and the returned value. A few functions execute in constant time: from_external, is_empty_set, is_set, is_sofs_set, to_external, type." O(N log N) for most (optimal sort time, where N = sum(Input)), with a list of O(1) functions 12. Digraph Unspecified, no idea 13. lists:key* functions using lists, so O(N), but they're fast given they're implemented in C 14. Proplists Using lists, so O(N). Now with these modules somewhat covered (there's a lot of subtleties in the APIs making some better than others depending on the access patterns), it's interesting to look at basic operations we may do a lot, for example: 1. Message receiving (non-selective) O(1), though accessing the mailbox when it's very large can trigger GCs and mess up expectations a bit. 2. Message receiving (selective) O(N) selection where N is the mailbox size. An optimization existsw for common access patterns where a reference (with make_ref()) is created in the scope of the receive and used in all clauses, which truncates the existing mailbox, therefore reducing N. 3. Sorting (lists:sort) O(N log N), performing a fancypants merge sort So yeah, for a lot of data structures, the information is already there. For many, it's also still missing. Regards, Fred. From zxq9@REDACTED Thu Sep 17 18:13:04 2015 From: zxq9@REDACTED (zxq9) Date: Fri, 18 Sep 2015 01:13:04 +0900 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: <20150917155645.GC9409@fhebert-ltm1> References: <20150917155645.GC9409@fhebert-ltm1> Message-ID: <2920189.nkzdrQtalV@changa> On 2015?9?17? ??? 11:56:47 you wrote: > On 09/17, Eric des Courtis wrote: > >Regardless if concurrency and parallelism plays a role *I still need to > >know if an operation in a sequential process I am working on is O(1) or > >O(n)* or am I missing something? So that comment I made about "case 1" where there are a "lot of options already"... > These values are way more worthwhile as a property of libraries than > language, since they're far more impacted by the choice of algorithm > taken there. > > Let's look at a few modules: > > 1. Queue > 2. Maps > 3. Dict > 4. Array > 5. Gb_trees > 6. Sets > 7. Gb_sets > 8. ETS > 9. Orddict > 10. ordsets > 11. sofs > 12. Digraph > 13. lists:key* functions > 14. Proplists ... > 1. Message receiving (non-selective) > 2. Message receiving (selective) > 3. Sorting (lists:sort) These are costs specific to a particular operation happening on its own time, but tell you nothing meaningful about your system as a whole because all of this stuff is going on at once in different processes at different times. Sometimes you are in the situation where doubling your processing speed just means adding more cores. Sometimes not. You will either *know* at the outset of a project or *have no idea whether this is true* until you actually have something up and running that you can measure. Usually it is the latter. Very often you will find yourself able to approach a concurrent ideal after you've already got something implemented that does basically what you want, but not before. This is true whether or not you've got months of paid time to toy with an idea (HA HA! Like *that* ever happens!). It is nearly always faster to experiment with a prototype in Erlang than to just muse about it until the concept is perfect. Once you have a prototype, tweaking it is easy, and when that exists you can already measure stuff for real. This is why I am calling the *unqualified* utility of cost models into question. -Craig From vasdeveloper@REDACTED Thu Sep 17 18:36:40 2015 From: vasdeveloper@REDACTED (Theepan) Date: Thu, 17 Sep 2015 22:06:40 +0530 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: <20150917155645.GC9409@fhebert-ltm1> References: <2423630.4x47EVrrNS@burrito> <20150917155645.GC9409@fhebert-ltm1> Message-ID: ETS - O(1) Diagraph - O(1) Theepan On Thu, Sep 17, 2015 at 9:26 PM, Fred Hebert wrote: > On 09/17, Eric des Courtis wrote: > >> Regardless if concurrency and parallelism plays a role *I still need to >> know if an operation in a sequential process I am working on is O(1) or >> O(n)* or am I missing something? >> > > These values are way more worthwhile as a property of libraries than > language, since they're far more impacted by the choice of algorithm taken > there. > > Let's look at a few modules: > > 1. Queue > > "all operations has an amortized O(1) running time, except len/1, join/2, > split/2, filter/2 and member/2 that have O(n). To minimize the size of a > queue minimizing the amount of garbage built by queue operations, the > queues do not contain explicit length information, and that is why len/1 is > O(n)." > > > 2. Maps > > undefined in the module doc, but from the EEP ( > http://www.erlang.org/eeps/eep-0043.html): > > "has at most O(log N) time complexity in insert and lookup operations, > where N is the number of key-value associations." > > 3. Dict > > undefined in docs. I know that it is a very flat tree (bunch of buckets) > acting as a hash map, so O(log N) > > 4. Array > > undefined in docs. Also a very flat tree. O(Log N), where N is the size of > the array, and if I recall, the tree has a branching factor of 10 to > compromise between the speed of lookups and garbage generated when > modifying it. It's described in the source only, sadly. > > 5. Gb_trees > > Doc compares them to AVL trees, without storage overhead and with better > performance. O(log N) it is (those are the worst cases for AVL trees) > > 6. Sets > > They're using the same mechanism as dicts, so O(Log N) > > 7. Gb_sets > > Same mechanism as gb_trees, so O(log N) > > 8. ETS > > Doc says: "These provide the ability to store very large quantities of > data in an Erlang runtime system, and to have constant access time to the > data. (In the case of ordered_set, see below, access time is proportional > to the logarithm of the number of objects stored)." > > 9. Orddict > > Doc says " An orddict is a representation of a dictionary, where a list of > pairs is used to store the keys and values. The list is ordered after the > keys." > > lists are O(N) traversal and insertion, so let's consider this the best > case. > > 10. ordsets > > implemented over orddict, O(N) too. > > 11. sofs > > despite having the most cryptic documentation, it does mention: "The > execution time of the functions of this module is dominated by the time it > takes to sort lists. When no sorting is needed, the execution time is in > the worst case proportional to the sum of the sizes of the input arguments > and the returned value. A few functions execute in constant time: > from_external, is_empty_set, is_set, is_sofs_set, to_external, type." > > O(N log N) for most (optimal sort time, where N = sum(Input)), with a list > of O(1) functions > > 12. Digraph > > Unspecified, no idea > > 13. lists:key* functions > > using lists, so O(N), but they're fast given they're implemented in C > > 14. Proplists > > Using lists, so O(N). > > > > Now with these modules somewhat covered (there's a lot of subtleties in > the APIs making some better than others depending on the access patterns), > it's interesting to look at basic operations we may do a lot, for example: > > 1. Message receiving (non-selective) > > O(1), though accessing the mailbox when it's very large can trigger GCs > and mess up expectations a bit. > > 2. Message receiving (selective) > > O(N) selection where N is the mailbox size. An optimization existsw for > common access patterns where a reference (with make_ref()) is created in > the scope of the receive and used in all clauses, which truncates the > existing mailbox, therefore reducing N. > > 3. Sorting (lists:sort) > > O(N log N), performing a fancypants merge sort > > > > So yeah, for a lot of data structures, the information is already there. > For many, it's also still missing. > > Regards, > Fred. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Thu Sep 17 19:04:32 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 17 Sep 2015 13:04:32 -0400 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: <2920189.nkzdrQtalV@changa> References: <20150917155645.GC9409@fhebert-ltm1> <2920189.nkzdrQtalV@changa> Message-ID: <20150917170431.GE9409@fhebert-ltm1> On 09/18, zxq9 wrote: >These are costs specific to a particular operation happening on its own >time, but tell you nothing meaningful about your system as a whole >because all of this stuff is going on at once in different processes at >different times. Sometimes you are in the situation where doubling your >processing speed just means adding more cores. Sometimes not. You will >either *know* at the outset of a project or *have no idea whether this >is true* until you actually have something up and running that you can >measure. Usually it is the latter. > If you have any sizeable data set, you can improve speed by orders of magnitude, which is a lot more worthwhile than by core count (dividing by a constant factor only). If you have an O(n?) program running on 48 cores, and a O(log n) one on one core, any significant input will leave your one core program going faster sooner or later. Disregarding the time and memory complexity of your algorithms because ~cores~ is just a plain terrible design mechanism. Hell, you also have to care for memory, network, etc. which are often addressed by the complexity of algorithms too. All in all, you've got N units of work to do, and processors let you do M of them at once, all cores summed up. The best way to optimize is to find how to reduce the amount of work by a lot rather than augment the number of cores. Not that you should spit on cores. >Very often you will find yourself able to approach a concurrent ideal >after you've already got something implemented that does basically what >you want, but not before. This is true whether or not you've got months >of paid time to toy with an idea (HA HA! Like *that* ever happens!). It >is nearly always faster to experiment with a prototype in Erlang than >to just muse about it until the concept is perfect. Once you have a >prototype, tweaking it is easy, and when that exists you can already >measure stuff for real. This is why I am calling the *unqualified* >utility of cost models into question. > That is 100% unrelated to knowing which data structures already available exist and what their properties are. If you've got 300,000 pieces of data to store, you don't need months of prototyping to realize orddicts are a bad idea and maps a better one. And that if you often want the smallest of these 300,000 elements, then maybe gb_trees are a better choice than maps because the doc says you can get them in O(log n) and maps require you to do an O(n) scan. You may find out you need a fancy custom data structure after prototyping, but in no way do you find yourself in a bad situation for knowing the complexity of tools you already have available. From zxq9@REDACTED Thu Sep 17 19:11:27 2015 From: zxq9@REDACTED (zxq9) Date: Fri, 18 Sep 2015 02:11:27 +0900 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: <20150917170431.GE9409@fhebert-ltm1> References: <2920189.nkzdrQtalV@changa> <20150917170431.GE9409@fhebert-ltm1> Message-ID: <4266266.K7ytP5HQ6i@changa> On 2015?9?17? ??? 13:04:32 you wrote: > On 09/18, zxq9 wrote: > >These are costs specific to a particular operation happening on its own > >time, but tell you nothing meaningful about your system as a whole > >because all of this stuff is going on at once in different processes at > >different times. Sometimes you are in the situation where doubling your > >processing speed just means adding more cores. Sometimes not. You will > >either *know* at the outset of a project or *have no idea whether this > >is true* until you actually have something up and running that you can > >measure. Usually it is the latter. > > > > If you have any sizeable data set, you can improve speed by orders of > magnitude, which is a lot more worthwhile than by core count (dividing > by a constant factor only). ... > >This is why I am calling the *unqualified* > >utility of cost models into question. > > ... > You may find out you need a fancy custom data structure after > prototyping, but in no way do you find yourself in a bad situation for > knowing the complexity of tools you already have available. These are all qualifications. Which is my whole point. I can't think of many (any?) places where I know in advance that I will have 300 thousand (or 300 million) elements in a *single* data set that I will want to manipulate them directly in Erlang. -Craig From ahe.sanath@REDACTED Thu Sep 17 20:05:34 2015 From: ahe.sanath@REDACTED (Sanath Prasanna) Date: Thu, 17 Sep 2015 23:35:34 +0530 Subject: [erlang-questions] Unexpected behavior of HTTPC module - tls_connection process take more memory In-Reply-To: References: Message-ID: Hi Ingela, Any update on this?? "Even apply your patch, *still problem is persist.* Any more suggestions to solve this unexpected behavior ? " Br, Robert On Thu, Sep 17, 2015 at 12:39 PM, Sanath Prasanna wrote: > Hi Ingela, > Even apply your patch, *still problem is persist.* Any > more suggestions to solve this unexpected behavior ? > Br, > Robert > > On Tue, Sep 8, 2015 at 1:52 PM, Sanath Prasanna > wrote: > >> Hi Ingela. >> Tx a lot for your help & patch related to that.I'll inform you the result >> after applying & testing patch. >> Br, >> Robert >> >> On Tue, Sep 8, 2015 at 1:31 PM, Ingela Andin >> wrote: >> >>> Hi! >>> >>> It could be an bug that in the ssl application that I just fixed. The >>> default session cache >>> was violating the API, and this in turn made the mechanism for not >>> registering a lot of equivalent >>> sessions in the client fail. >>> >>> Here is the patch: >>> >>> >>> diff --git a/lib/ssl/src/ssl_session.erl b/lib/ssl/src/ssl_session.erl >>> index 1770faf..0d6cc93 100644 >>> --- a/lib/ssl/src/ssl_session.erl >>> +++ b/lib/ssl/src/ssl_session.erl >>> @@ -100,14 +100,14 @@ select_session([], _, _) -> >>> no_session; >>> select_session(Sessions, #ssl_options{ciphers = Ciphers}, OwnCert) -> >>> IsNotResumable = >>> - fun([_Id, Session]) -> >>> + fun(Session) -> >>> not (resumable(Session#session.is_resumable) andalso >>> lists:member(Session#session.cipher_suite, Ciphers) >>> andalso (OwnCert == Session#session.own_certificate)) >>> end, >>> case lists:dropwhile(IsNotResumable, Sessions) of >>> [] -> no_session; >>> - [[Id, _]|_] -> Id >>> + [Session | _] -> Session#session.session_id >>> end. >>> >>> is_resumable(_, _, #ssl_options{reuse_sessions = false}, _, _, _, _) -> >>> diff --git a/lib/ssl/src/ssl_session_cache.erl >>> b/lib/ssl/src/ssl_session_cach >>> e.erl >>> index 11ed310..cfc48cd 100644 >>> --- a/lib/ssl/src/ssl_session_cache.erl >>> +++ b/lib/ssl/src/ssl_session_cache.erl >>> @@ -83,7 +83,7 @@ foldl(Fun, Acc0, Cache) -> >>> %%-------------------------------------------------------------------- >>> select_session(Cache, PartialKey) -> >>> ets:select(Cache, >>> - [{{{PartialKey,'$1'}, '$2'},[],['$$']}]). >>> + [{{{PartialKey,'_'}, '$1'},[],['$1']}]). >>> >>> %%-------------------------------------------------------------------- >>> %%% Internal functions >>> >>> >>> Regards Ingela Erlang/OTP team - Ericsson AB >>> >>> >>> 2015-09-07 5:52 GMT+02:00 Sanath Prasanna : >>> >>>> Hi all, >>>> >>>> I am running HTTP client using httpc Module to send both http and https >>>> requests. normally sending arround 300 request per second without any >>>> issue. however sometimes erlang node become very slow responsive. at that >>>> time server load average is very high and using etop can identify >>>> "tls_connection" process take more memory. when restart the erlang node its >>>> become normal. as per my investigation normal time memory, processors, >>>> loadAverage is not increasing. following is the HTTP request config >>>> >>>> >>>> >>>> httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync, >>>> false}]) >>>> >>>> >>>> >>>> below is the etop output at that time >>>> >>>> >>>> >>>> procs 1134 processes 1504844 >>>> code 9309 >>>> >>>> runq 0 atom 420 >>>> ets 29692 >>>> >>>> >>>> >>>> Pid Name or Initial Func Time Reds Memory MsgQ >>>> Current Function >>>> >>>> >>>> ---------------------------------------------------------------------------------------- >>>> >>>> <5490.26428.14>tls_connection:init/ '-' 733224580768 0 >>>> gen_fsm:loop/7 >>>> >>>> <5490.26429.14>tls_connection:init/ '-' 1328524580768 0 >>>> gen_fsm:loop/7 >>>> >>>> <5490.26430.14>tls_connection:init/ '-' 528924580768 0 >>>> gen_fsm:loop/7 >>>> >>>> <5490.26431.14>tls_connection:init/ '-' 1432224580768 0 >>>> gen_fsm:loop/7 >>>> >>>> <5490.26432.14>tls_connection:init/ '-' 024580768 0 >>>> gen_fsm:loop/7 >>>> >>>> <5490.26433.14>tls_connection:init/ '-' 024580768 0 >>>> gen_fsm:loop/7 >>>> >>>> <5490.26434.14>tls_connection:init/ '-' 024580768 0 >>>> gen_fsm:loop/7 >>>> >>>> <5490.26435.14>tls_connection:init/ '-' 024580768 0 >>>> gen_fsm:loop/7 >>>> >>>> <5490.26436.14>tls_connection:init/ '-' 024580768 0 >>>> gen_fsm:loop/7 >>>> >>>> >>>> can some one help me to solve this issue? >>>> >>>> Br, >>>> >>>> A.H.E. Robert >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Fri Sep 18 00:16:04 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Thu, 17 Sep 2015 18:16:04 -0400 (EDT) Subject: [erlang-questions] Add a field to a record Message-ID: <1442528164.751430151@apps.rackspace.com> Hello, I understand that it would be better to use maps if you anticipate adding fields to a record. But suppose you didn't anticipate and you've been merrily adding data to your db when you discover--- oops! I need another field. e.g. You have: -record anything, { field1, field2 } and you discover that you really need: -record anything, { field1, new_field, field2 } Is there a cunning trick or technique to make the change without losing data? Thanks, LRP ********************************************* My books: THE GOSPEL OF ASHES http://thegospelofashes.com Strength is not enough. Do they have the courage and the cunning? Can they survive long enough to save the lives of millions? FREEIN' PANCHO http://freeinpancho.com A community of misfits help a troubled boy find his way AYA TAKEO http://ayatakeo.com Star-crossed love, war and power in an alternative universe Available through Amazon or by request from your favorite bookstore ********************************************** From mjtruog@REDACTED Fri Sep 18 00:47:31 2015 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 17 Sep 2015 15:47:31 -0700 Subject: [erlang-questions] Add a field to a record In-Reply-To: <1442528164.751430151@apps.rackspace.com> References: <1442528164.751430151@apps.rackspace.com> Message-ID: <55FB4303.3010309@gmail.com> On 09/17/2015 03:16 PM, lloyd@REDACTED wrote: > Hello, > > I understand that it would be better to use maps if you anticipate adding fields to a record. > > But suppose you didn't anticipate and you've been merrily adding data to your db when you discover--- oops! I need another field. > > e.g. You have: > -record anything, { > field1, > field2 > } > > and you discover that you really need: > -record anything, { > field1, > new_field, > field2 > } > > Is there a cunning trick or technique to make the change without losing data? If you always add fields last, it will likely not be a problem, since the older format's indexes will remain the same, unless something accesses the new record format with an old tuple (from the old record format, especially if the tuple length is checked). It is safer to put a record version number suffix in the record name and do an explicit conversion. > > Thanks, > > LRP > > > ********************************************* > My books: > > THE GOSPEL OF ASHES > http://thegospelofashes.com > > Strength is not enough. Do they have the courage > and the cunning? Can they survive long enough to > save the lives of millions? > > FREEIN' PANCHO > http://freeinpancho.com > > A community of misfits help a troubled boy find his way > > AYA TAKEO > http://ayatakeo.com > > Star-crossed love, war and power in an alternative > universe > > Available through Amazon or by request from your > favorite bookstore > > > ********************************************** > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From ok@REDACTED Fri Sep 18 01:59:34 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 18 Sep 2015 11:59:34 +1200 Subject: [erlang-questions] distinguish proplists vs. lists of proplists In-Reply-To: <20150916141939.GB9409@fhebert-ltm1> References: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> <1442339510-sup-9669@arch.local> <7FB5964B-A263-43D0-A656-2D483CF3ED5E@redops.org> <2420C084-E494-4234-8D11-B7580542738C@cs.otago.ac.nz> <20150916141939.GB9409@fhebert-ltm1> Message-ID: Just to be clear, the code I sketched for get_value/3 was not a proposal or recommendation or preference but just an idle musing about why the proplists interface so, um, well, you know what I mean. I suspect that nobody ever thought it would hang around this long. From ok@REDACTED Fri Sep 18 02:54:57 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 18 Sep 2015 12:54:57 +1200 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: Message-ID: On 17/09/2015, at 3:03 am, Eric des Courtis wrote: > What is the cost model of all operations in Erlang 18? > > For example Python has this cost model http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Python_Cost_Model So let's look at that page, and you'll spot something interesting in the URL. (1) That page was last revised in January 2008; the results are from 2007. Looking at neighbouring pages like http://scripts.mit.edu/~6.006/fall07/wiki/index.php?title=Document_Distance_Program_Version_5 I get "Database error", which suggests that the site has not been maintained for too long. Poke around a bit further, and you'll find (2) It used Windows XP (version of 2001) on "an IBM Thinkpad T43p with a 1.86GHz Pentium M processor and 1.5GB RAM." (3) The version of Python used for the tests is not stated, but the Python version current at the time was 2.5.1. (4) The author thought "extract a byte from a string" was an interesting operation. Let's compare that with today. (1) It's September 2015, nearly 8 years since the measurements were done. (2) I'm using Mac OS X 10.9.5 on an iMac14.2 with a 3.2 GHz Intel Core i5 with 4 cores (and using those 4 cores is an important part of *system*-level efficiency) and 16 GiB of 1.6 GHz DDR3 memory. The architecture of a Core i5 is significantly different from a Pentium M, and it's not just the 64-bit ISA. (3) Python 2.7.8 was released in mid-2014 but the current version is Python 3.5.0. The benchmarks had much to say about integers and strings, and these are two areas which have seen MAJOR changes in the change to Python 3. (4) Unicode is everywhere these days; characters are no longer bytes. At least one of the benchmarks is now of questionable relevance. In short, before I used that "Python cost model" page for *anything*, I'd have to first port timing.py to Python 3 (which may or may not be trivial; no idea without trying it), and then rerun it on my machine. I'd still be wondering what difference, if any, building with recent clang, recent gcc, or recent icc would make. And it *still* wouldn't tell me anything about the cost of using NumPy or SciPy or PyQt or TKinter or .... From mrtndimitrov@REDACTED Fri Sep 18 09:04:30 2015 From: mrtndimitrov@REDACTED (Martin Koroudjiev) Date: Fri, 18 Sep 2015 10:04:30 +0300 Subject: [erlang-questions] Replacing erlang:now/0 for getting a unique number Message-ID: <55FBB77E.70307@gmail.com> Hello, I have code that users erlang:now/0 to get a unique number. What is the alternative in OTP 18? In the documentation for erlang:monotonic_time/0 says that consecutive calls might produce same results. Thanks, Martin From zxq9@REDACTED Fri Sep 18 09:24:36 2015 From: zxq9@REDACTED (zxq9) Date: Fri, 18 Sep 2015 16:24:36 +0900 Subject: [erlang-questions] Replacing erlang:now/0 for getting a unique number In-Reply-To: <55FBB77E.70307@gmail.com> References: <55FBB77E.70307@gmail.com> Message-ID: <4157002.n1BnLX5JJz@changa> On 2015?9?18? ??? 10:04:30 Martin Koroudjiev wrote: > Hello, > > I have code that users erlang:now/0 to get a unique number. What is the > alternative in OTP 18? In the documentation for erlang:monotonic_time/0 > says that consecutive calls might produce same results. It depends on what you are using the value for. If you are just getting the time, either erlang:system_time/1 (you pass what unit you want), or if you want it to work the way now/0 used to, then use erlang:timestamp/0 -- but this is not guaranteed to be unique (make two calls faster than the clock's resolution, for example). If you need to determine ordering of events using a timestamp then you need a tuple that combines monotonic_time/0 and inique_integer/1: EventStamp = {monotonic_time(), unique_integer([monotonic])}. Systems are so fast now that outrunning the system clock can actually happen more than you might expect! There is a section in the release notes for R18 that explains a how time issues have changed. -Craig From zxq9@REDACTED Fri Sep 18 09:28:08 2015 From: zxq9@REDACTED (zxq9) Date: Fri, 18 Sep 2015 16:28:08 +0900 Subject: [erlang-questions] Replacing erlang:now/0 for getting a unique number In-Reply-To: <4157002.n1BnLX5JJz@changa> References: <55FBB77E.70307@gmail.com> <4157002.n1BnLX5JJz@changa> Message-ID: <2365410.jjx97Kpb2k@changa> On 2015?9?18? ??? 16:24:36 zxq9 wrote: > There is a section in the release notes for R18 that explains a how time issues have changed. Ah, it is right up near the beginning of the new version of the Erlang User's Guide. There is a whole section worth reading through once: http://www.erlang.org/doc/apps/erts/time_correction.html The part with a short rundown of "Do's and Don'ts" is here: http://www.erlang.org/doc/apps/erts/time_correction.html#Dos_and_Donts -Craig From vasdeveloper@REDACTED Fri Sep 18 09:30:13 2015 From: vasdeveloper@REDACTED (Theepan) Date: Fri, 18 Sep 2015 13:00:13 +0530 Subject: [erlang-questions] Replacing erlang:now/0 for getting a unique number In-Reply-To: <55FBB77E.70307@gmail.com> References: <55FBB77E.70307@gmail.com> Message-ID: It depends on where you want the number to be unique (intra node/inter node), and the frequency of unique number generation (system calls are heavy), you can look at how you are generating the unique number. You can look at just having a counter too. Theepan On Fri, Sep 18, 2015 at 12:34 PM, Martin Koroudjiev wrote: > Hello, > > I have code that users erlang:now/0 to get a unique number. What is the > alternative in OTP 18? In the documentation for erlang:monotonic_time/0 > says that consecutive calls might produce same results. > > Thanks, > Martin > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrtndimitrov@REDACTED Fri Sep 18 09:35:09 2015 From: mrtndimitrov@REDACTED (Martin Koroudjiev) Date: Fri, 18 Sep 2015 10:35:09 +0300 Subject: [erlang-questions] Replacing erlang:now/0 for getting a unique number In-Reply-To: <4157002.n1BnLX5JJz@changa> References: <55FBB77E.70307@gmail.com> <4157002.n1BnLX5JJz@changa> Message-ID: <55FBBEAD.7050005@gmail.com> Thanks guys, Seems from the documentation that unique_integer([monotonic] is enough. Regards, Martin On 9/18/2015 10:24 AM, zxq9 wrote: > On 2015?9?18? ??? 10:04:30 Martin Koroudjiev wrote: >> Hello, >> >> I have code that users erlang:now/0 to get a unique number. What is the >> alternative in OTP 18? In the documentation for erlang:monotonic_time/0 >> says that consecutive calls might produce same results. > It depends on what you are using the value for. > > If you are just getting the time, either erlang:system_time/1 (you pass what unit you want), or if you want it to work the way now/0 used to, then use erlang:timestamp/0 -- but this is not guaranteed to be unique (make two calls faster than the clock's resolution, for example). > > If you need to determine ordering of events using a timestamp then you need a tuple that combines monotonic_time/0 and inique_integer/1: > > EventStamp = {monotonic_time(), unique_integer([monotonic])}. > > Systems are so fast now that outrunning the system clock can actually happen more than you might expect! > > There is a section in the release notes for R18 that explains a how time issues have changed. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From marc@REDACTED Fri Sep 18 09:54:11 2015 From: marc@REDACTED (Marc Worrell) Date: Fri, 18 Sep 2015 09:54:11 +0200 Subject: [erlang-questions] [ANN] Zotonic 0.13.4 release - upgrade your server! Message-ID: <330BCCD1-2653-4C52-8ED7-F90E36D192E3@worrell.nl> Hi, Zotonic is the Erlang Content Management System and Framework. We have released version 0.13.4 We found an issue where a survey input value was not properly escaped when displayed in the survey result editor. If you are using mod_survey and also use the survey result editor, then we urge you to upgrade. This issue is also fixed on the 0.12.x, 0.11.x and 0.10.x branches. Upgrade to the tip of those branches if you are using these older versions. Main changes in 0.13.4 are: ? fixed an issue where mod_survey answers could be shown without escaping ? fixes for admin css and template issues ? corrected handling of unsafe (escaped) characters in email addresses ? easier access rule definitions by special handling of 'meta' category ? fixed an issue where mod_video would crash if there was an empty ffmpeg configuration ? fixed an issue where the last row of an imported CSV file could be reversed See the full release notes at http://zotonic.com/docs/latest/dev/releasenotes/rel_0.13.4.html Known issue: the periodic backup of mod_backup doesn't work, use https://github.com/zotonic/zotonic/commit/132ac856b8b182230faf9dc5f3064f7ebae9498b for a fix. Kind regards, The Zotonic Team -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Fri Sep 18 10:06:03 2015 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 18 Sep 2015 10:06:03 +0200 Subject: [erlang-questions] Fwd: Unexpected behavior of HTTPC module - tls_connection process take more memory In-Reply-To: References: Message-ID: Hi! As we are very busy with the release 18.1 I have not had time to try and recreate your problem. What version of OTP and the ssl application are you using? Can you reproduce the problem with the latest on github? Regards Ingela Erlang/OTP Team - Ericsson AB On Thu, Sep 17, 2015 at 8:05 PM, Sanath Prasanna wrote: > Hi Ingela, > Any update on this?? > "Even apply your patch, *still problem is persist.* Any > more suggestions to solve this unexpected behavior ? " > Br, > Robert > > On Thu, Sep 17, 2015 at 12:39 PM, Sanath Prasanna > wrote: > >> Hi Ingela, >> Even apply your patch, *still problem is persist.* Any >> more suggestions to solve this unexpected behavior ? >> Br, >> Robert >> >> On Tue, Sep 8, 2015 at 1:52 PM, Sanath Prasanna >> wrote: >> >>> Hi Ingela. >>> Tx a lot for your help & patch related to that.I'll inform you the >>> result after applying & testing patch. >>> Br, >>> Robert >>> >>> On Tue, Sep 8, 2015 at 1:31 PM, Ingela Andin >>> wrote: >>> >>>> Hi! >>>> >>>> It could be an bug that in the ssl application that I just fixed. The >>>> default session cache >>>> was violating the API, and this in turn made the mechanism for not >>>> registering a lot of equivalent >>>> sessions in the client fail. >>>> >>>> Here is the patch: >>>> >>>> >>>> diff --git a/lib/ssl/src/ssl_session.erl b/lib/ssl/src/ssl_session.erl >>>> index 1770faf..0d6cc93 100644 >>>> --- a/lib/ssl/src/ssl_session.erl >>>> +++ b/lib/ssl/src/ssl_session.erl >>>> @@ -100,14 +100,14 @@ select_session([], _, _) -> >>>> no_session; >>>> select_session(Sessions, #ssl_options{ciphers = Ciphers}, OwnCert) -> >>>> IsNotResumable = >>>> - fun([_Id, Session]) -> >>>> + fun(Session) -> >>>> not (resumable(Session#session.is_resumable) andalso >>>> lists:member(Session#session.cipher_suite, Ciphers) >>>> andalso (OwnCert == Session#session.own_certificate)) >>>> end, >>>> case lists:dropwhile(IsNotResumable, Sessions) of >>>> [] -> no_session; >>>> - [[Id, _]|_] -> Id >>>> + [Session | _] -> Session#session.session_id >>>> end. >>>> >>>> is_resumable(_, _, #ssl_options{reuse_sessions = false}, _, _, _, _) -> >>>> diff --git a/lib/ssl/src/ssl_session_cache.erl >>>> b/lib/ssl/src/ssl_session_cach >>>> e.erl >>>> index 11ed310..cfc48cd 100644 >>>> --- a/lib/ssl/src/ssl_session_cache.erl >>>> +++ b/lib/ssl/src/ssl_session_cache.erl >>>> @@ -83,7 +83,7 @@ foldl(Fun, Acc0, Cache) -> >>>> %%-------------------------------------------------------------------- >>>> select_session(Cache, PartialKey) -> >>>> ets:select(Cache, >>>> - [{{{PartialKey,'$1'}, '$2'},[],['$$']}]). >>>> + [{{{PartialKey,'_'}, '$1'},[],['$1']}]). >>>> >>>> %%-------------------------------------------------------------------- >>>> %%% Internal functions >>>> >>>> >>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>> >>>> >>>> 2015-09-07 5:52 GMT+02:00 Sanath Prasanna : >>>> >>>>> Hi all, >>>>> >>>>> I am running HTTP client using httpc Module to send both http and >>>>> https requests. normally sending arround 300 request per second without any >>>>> issue. however sometimes erlang node become very slow responsive. at that >>>>> time server load average is very high and using etop can identify >>>>> "tls_connection" process take more memory. when restart the erlang node its >>>>> become normal. as per my investigation normal time memory, processors, >>>>> loadAverage is not increasing. following is the HTTP request config >>>>> >>>>> >>>>> >>>>> httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync, >>>>> false}]) >>>>> >>>>> >>>>> >>>>> below is the etop output at that time >>>>> >>>>> >>>>> >>>>> procs 1134 processes 1504844 >>>>> code 9309 >>>>> >>>>> runq 0 atom 420 >>>>> ets 29692 >>>>> >>>>> >>>>> >>>>> Pid Name or Initial Func Time Reds Memory MsgQ >>>>> Current Function >>>>> >>>>> >>>>> ---------------------------------------------------------------------------------------- >>>>> >>>>> <5490.26428.14>tls_connection:init/ '-' 733224580768 0 >>>>> gen_fsm:loop/7 >>>>> >>>>> <5490.26429.14>tls_connection:init/ '-' 1328524580768 0 >>>>> gen_fsm:loop/7 >>>>> >>>>> <5490.26430.14>tls_connection:init/ '-' 528924580768 0 >>>>> gen_fsm:loop/7 >>>>> >>>>> <5490.26431.14>tls_connection:init/ '-' 1432224580768 0 >>>>> gen_fsm:loop/7 >>>>> >>>>> <5490.26432.14>tls_connection:init/ '-' 024580768 0 >>>>> gen_fsm:loop/7 >>>>> >>>>> <5490.26433.14>tls_connection:init/ '-' 024580768 0 >>>>> gen_fsm:loop/7 >>>>> >>>>> <5490.26434.14>tls_connection:init/ '-' 024580768 0 >>>>> gen_fsm:loop/7 >>>>> >>>>> <5490.26435.14>tls_connection:init/ '-' 024580768 0 >>>>> gen_fsm:loop/7 >>>>> >>>>> <5490.26436.14>tls_connection:init/ '-' 024580768 0 >>>>> gen_fsm:loop/7 >>>>> >>>>> >>>>> can some one help me to solve this issue? >>>>> >>>>> Br, >>>>> >>>>> A.H.E. Robert >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Fri Sep 18 10:17:42 2015 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 18 Sep 2015 10:17:42 +0200 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: <2920189.nkzdrQtalV@changa> References: <20150917155645.GC9409@fhebert-ltm1> <2920189.nkzdrQtalV@changa> Message-ID: On Thu, Sep 17, 2015 at 6:13 PM, zxq9 wrote: > On 2015?9?17? ??? 11:56:47 you wrote: >> On 09/17, Eric des Courtis wrote: >> >Regardless if concurrency and parallelism plays a role *I still need to >> >know if an operation in a sequential process I am working on is O(1) or >> >O(n)* or am I missing something? Note that O(1) does not mean fast - it just means "constant" time without knowing *what* the constant is this information is useless. O(N) means the time depends linearly upon N - but again I doubt if anybody really knows if a function is O(N) - when you run out of cache or swap space it will certainly not be linear. Knowing that a function is O(N) is meaningless for performance estimation if you don't know the range of N AND the constant values involved AND the scale factors. If you notice *time and time again* when people say "will Erlang be fast enough to do this" and "this" is described in vague terms of "lot's of users" "lot's of data" - I reply - "give me the numbers" and "don't guess measure" You need to numerically quantity the problem AND the target computer. Even with all the numbers available predictions should be taken with a lorry-load of salt. Cheers /Joe > > So that comment I made about "case 1" where there are a "lot of options already"... > >> These values are way more worthwhile as a property of libraries than >> language, since they're far more impacted by the choice of algorithm >> taken there. >> >> Let's look at a few modules: >> >> 1. Queue >> 2. Maps >> 3. Dict >> 4. Array >> 5. Gb_trees >> 6. Sets >> 7. Gb_sets >> 8. ETS >> 9. Orddict >> 10. ordsets >> 11. sofs >> 12. Digraph >> 13. lists:key* functions >> 14. Proplists > > ... > >> 1. Message receiving (non-selective) >> 2. Message receiving (selective) >> 3. Sorting (lists:sort) > > These are costs specific to a particular operation happening on its own time, but tell you nothing meaningful about your system as a whole because all of this stuff is going on at once in different processes at different times. Sometimes you are in the situation where doubling your processing speed just means adding more cores. Sometimes not. You will either *know* at the outset of a project or *have no idea whether this is true* until you actually have something up and running that you can measure. Usually it is the latter. > > Very often you will find yourself able to approach a concurrent ideal after you've already got something implemented that does basically what you want, but not before. This is true whether or not you've got months of paid time to toy with an idea (HA HA! Like *that* ever happens!). It is nearly always faster to experiment with a prototype in Erlang than to just muse about it until the concept is perfect. Once you have a prototype, tweaking it is easy, and when that exists you can already measure stuff for real. This is why I am calling the *unqualified* utility of cost models into question. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From marc@REDACTED Fri Sep 18 10:20:39 2015 From: marc@REDACTED (Marc Worrell) Date: Fri, 18 Sep 2015 10:20:39 +0200 Subject: [erlang-questions] Add a field to a record In-Reply-To: <1442528164.751430151@apps.rackspace.com> References: <1442528164.751430151@apps.rackspace.com> Message-ID: Hi Lloyd, That is a very common situation. We have two tactics to handle that: 1) Just let it crash and then restart with the new definition 2) If (1) is not acceptable, use module/function that can ?upgrade? your record. Example of #2: upgrade(#anything{} = Rec) -> Rec; upgrade({anything, Field1, Field2}) -> #anything{field1=Field1, field2=Field2}. And then, if you receive a record from old(er) code/nodes or a backing store, first ?upgrade? your record before you do anything with it. do_something(Rec0) -> Rec = upgrade(Rec0), ? your code with Rec ? - Marc > On 18 sep. 2015, at 00:16, lloyd@REDACTED wrote: > > Hello, > > I understand that it would be better to use maps if you anticipate adding fields to a record. > > But suppose you didn't anticipate and you've been merrily adding data to your db when you discover--- oops! I need another field. > > e.g. You have: > -record anything, { > field1, > field2 > } > > and you discover that you really need: > -record anything, { > field1, > new_field, > field2 > } > > Is there a cunning trick or technique to make the change without losing data? > > Thanks, > > LRP > > > ********************************************* > My books: > > THE GOSPEL OF ASHES > http://thegospelofashes.com > > Strength is not enough. Do they have the courage > and the cunning? Can they survive long enough to > save the lives of millions? > > FREEIN' PANCHO > http://freeinpancho.com > > A community of misfits help a troubled boy find his way > > AYA TAKEO > http://ayatakeo.com > > Star-crossed love, war and power in an alternative > universe > > Available through Amazon or by request from your > favorite bookstore > > > ********************************************** > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From eranga.erl@REDACTED Fri Sep 18 10:41:20 2015 From: eranga.erl@REDACTED (Eranga Udesh) Date: Fri, 18 Sep 2015 14:11:20 +0530 Subject: [erlang-questions] Fwd: Unexpected behavior of HTTPC module - tls_connection process take more memory In-Reply-To: References: Message-ID: Hi, I am experiencing the same issue in, - Erlang version : V6.4 , (Erlang/OTP 17 [erts-6.4] [source] [64-bit]) - OS version/architecture (32/64 bit) : Red Hat Enterprise Linux Server release 6.6 (64 bit) The issue doesn't come always but after running the system for 3-10 days can experience the tls_connection instances making high reductions and consuming high memory as sent by Sanath. Since it's a production system, I wasn't able to do much experiments, but applied the patches you sent, which didn't solve this issue. There are 2 servers running Erlang VMs making SSL requests to the same HTTPS host. When this issue happens, it happens in both the VMs. So it could be triggered by some conditions in SSL or network connection. Even though I left the VMs to run for a while to see if they can recover, but no success. However if I restart the VMs, it starts to run as normal. So the summary is, - It triggers by some condition in remote SSL host or network connection - Erlang VMs don't recover itself - Once restarted, it start working, which implies the issue is local. i.e. in Erlang VM (tls_connection) We will try to recreate it and send you more details. I wonder if anybody else is experiencing such issue. Tks, - Eranga On Fri, Sep 18, 2015 at 1:36 PM, Ingela Andin wrote: > Hi! > > As we are very busy with the release 18.1 I have not had time to try and > recreate your problem. What version of OTP and the ssl application are you > using? > Can you reproduce the problem with the latest on github? > > Regards Ingela Erlang/OTP Team - Ericsson AB > > > On Thu, Sep 17, 2015 at 8:05 PM, Sanath Prasanna > wrote: > >> Hi Ingela, >> Any update on this?? >> "Even apply your patch, *still problem is persist.* Any >> more suggestions to solve this unexpected behavior ? " >> Br, >> Robert >> >> On Thu, Sep 17, 2015 at 12:39 PM, Sanath Prasanna >> wrote: >> >>> Hi Ingela, >>> Even apply your patch, *still problem is persist.* Any >>> more suggestions to solve this unexpected behavior ? >>> Br, >>> Robert >>> >>> On Tue, Sep 8, 2015 at 1:52 PM, Sanath Prasanna >>> wrote: >>> >>>> Hi Ingela. >>>> Tx a lot for your help & patch related to that.I'll inform you the >>>> result after applying & testing patch. >>>> Br, >>>> Robert >>>> >>>> On Tue, Sep 8, 2015 at 1:31 PM, Ingela Andin >>>> wrote: >>>> >>>>> Hi! >>>>> >>>>> It could be an bug that in the ssl application that I just fixed. The >>>>> default session cache >>>>> was violating the API, and this in turn made the mechanism for not >>>>> registering a lot of equivalent >>>>> sessions in the client fail. >>>>> >>>>> Here is the patch: >>>>> >>>>> >>>>> diff --git a/lib/ssl/src/ssl_session.erl b/lib/ssl/src/ssl_session.erl >>>>> index 1770faf..0d6cc93 100644 >>>>> --- a/lib/ssl/src/ssl_session.erl >>>>> +++ b/lib/ssl/src/ssl_session.erl >>>>> @@ -100,14 +100,14 @@ select_session([], _, _) -> >>>>> no_session; >>>>> select_session(Sessions, #ssl_options{ciphers = Ciphers}, OwnCert) -> >>>>> IsNotResumable = >>>>> - fun([_Id, Session]) -> >>>>> + fun(Session) -> >>>>> not (resumable(Session#session.is_resumable) andalso >>>>> lists:member(Session#session.cipher_suite, Ciphers) >>>>> andalso (OwnCert == Session#session.own_certificate)) >>>>> end, >>>>> case lists:dropwhile(IsNotResumable, Sessions) of >>>>> [] -> no_session; >>>>> - [[Id, _]|_] -> Id >>>>> + [Session | _] -> Session#session.session_id >>>>> end. >>>>> >>>>> is_resumable(_, _, #ssl_options{reuse_sessions = false}, _, _, _, _) >>>>> -> >>>>> diff --git a/lib/ssl/src/ssl_session_cache.erl >>>>> b/lib/ssl/src/ssl_session_cach >>>>> e.erl >>>>> index 11ed310..cfc48cd 100644 >>>>> --- a/lib/ssl/src/ssl_session_cache.erl >>>>> +++ b/lib/ssl/src/ssl_session_cache.erl >>>>> @@ -83,7 +83,7 @@ foldl(Fun, Acc0, Cache) -> >>>>> %%-------------------------------------------------------------------- >>>>> select_session(Cache, PartialKey) -> >>>>> ets:select(Cache, >>>>> - [{{{PartialKey,'$1'}, '$2'},[],['$$']}]). >>>>> + [{{{PartialKey,'_'}, '$1'},[],['$1']}]). >>>>> >>>>> %%-------------------------------------------------------------------- >>>>> %%% Internal functions >>>>> >>>>> >>>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>>> >>>>> >>>>> 2015-09-07 5:52 GMT+02:00 Sanath Prasanna : >>>>> >>>>>> Hi all, >>>>>> >>>>>> I am running HTTP client using httpc Module to send both http and >>>>>> https requests. normally sending arround 300 request per second without any >>>>>> issue. however sometimes erlang node become very slow responsive. at that >>>>>> time server load average is very high and using etop can identify >>>>>> "tls_connection" process take more memory. when restart the erlang node its >>>>>> become normal. as per my investigation normal time memory, processors, >>>>>> loadAverage is not increasing. following is the HTTP request config >>>>>> >>>>>> >>>>>> >>>>>> httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync, >>>>>> false}]) >>>>>> >>>>>> >>>>>> >>>>>> below is the etop output at that time >>>>>> >>>>>> >>>>>> >>>>>> procs 1134 processes 1504844 >>>>>> code 9309 >>>>>> >>>>>> runq 0 atom 420 >>>>>> ets 29692 >>>>>> >>>>>> >>>>>> >>>>>> Pid Name or Initial Func Time Reds Memory MsgQ >>>>>> Current Function >>>>>> >>>>>> >>>>>> ---------------------------------------------------------------------------------------- >>>>>> >>>>>> <5490.26428.14>tls_connection:init/ '-' 733224580768 0 >>>>>> gen_fsm:loop/7 >>>>>> >>>>>> <5490.26429.14>tls_connection:init/ '-' 1328524580768 0 >>>>>> gen_fsm:loop/7 >>>>>> >>>>>> <5490.26430.14>tls_connection:init/ '-' 528924580768 0 >>>>>> gen_fsm:loop/7 >>>>>> >>>>>> <5490.26431.14>tls_connection:init/ '-' 1432224580768 0 >>>>>> gen_fsm:loop/7 >>>>>> >>>>>> <5490.26432.14>tls_connection:init/ '-' 024580768 0 >>>>>> gen_fsm:loop/7 >>>>>> >>>>>> <5490.26433.14>tls_connection:init/ '-' 024580768 0 >>>>>> gen_fsm:loop/7 >>>>>> >>>>>> <5490.26434.14>tls_connection:init/ '-' 024580768 0 >>>>>> gen_fsm:loop/7 >>>>>> >>>>>> <5490.26435.14>tls_connection:init/ '-' 024580768 0 >>>>>> gen_fsm:loop/7 >>>>>> >>>>>> <5490.26436.14>tls_connection:init/ '-' 024580768 0 >>>>>> gen_fsm:loop/7 >>>>>> >>>>>> >>>>>> can some one help me to solve this issue? >>>>>> >>>>>> Br, >>>>>> >>>>>> A.H.E. Robert >>>>>> >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>> erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>> >>>>>> >>>>> >>>> >>> >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Fri Sep 18 10:57:46 2015 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Fri, 18 Sep 2015 10:57:46 +0200 Subject: [erlang-questions] distinguish proplists vs. lists of proplists In-Reply-To: <2420C084-E494-4234-8D11-B7580542738C@cs.otago.ac.nz> References: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> <1442339510-sup-9669@arch.local> <7FB5964B-A263-43D0-A656-2D483CF3ED5E@redops.org> <2420C084-E494-4234-8D11-B7580542738C@cs.otago.ac.nz> Message-ID: On Wed, Sep 16, 2015 at 7:15 AM, Richard A. O'Keefe wrote: > > On 16/09/2015, at 6:26 am, Drew Varner wrote: > > > A property in a proplist can be an atom or a tuple. > > The problem is that the documentation for the proplists > module says that the elements of a proplist can be ANYTHING. > > The documentation is a little confusing. proplists:get_value/[2,3] > only pays attention to pairs; other tuples might as well not be there. > > Does anyone know *why* the proplists module is so tolerant of > junk in proplists? Is it because > > get_value(Key, [{Key,Val}|_], _) -> Val; > get_value(Key, [Key|_], _) -> true; > get_value(Key, [_|Ps], Default) -> get_value(Key, Ps, Default); > get_value(Key, [], Default) -> Default. > > is the easiest thing to write, or is there a use case for it? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > The rationale of the proplists module was to make it easy for a program to handle options that could have been entered by a human, perhaps directly from the Erlang shell. (The first real use was in the option handling in the HiPE compiler.) I wanted it to be possible to use proplists as a drop-in replacement for most of the local option parsing hacks that were abundant in existing code - all slighty different. Looking at most modern code, I'd say that this was achieved - most people seem to be using proplists now. (Might be changing given that we have maps now - perhaps proplists should be updated to work on maps as well.) Not all expected uses of the module were for "options", though, so I didn't name it 'getopt' or similar, since that would have felt misleading. Since such lists in existing code often could contain random stuff apart from obvious key/value pairs or atoms (flags), and I also didn't want to block someone from using proplists just because they wanted to add some nonstandard stuff, the proplists module had to be lenient and mainly ignore anything that it didn't understand. The get_value example was not written like that because it was easiest, but because there was no obvious definition of the "value" for a key if the entry is a 3-tuple or larger. So instead, a user who calls get_value is explicitly asking proplists to ignore anything but pairs and naked keys, and needs to understand this. The proplists module was written for flexibility in how one writes the lists - not for speed (although I did make an effort to make the code as fast as possible given the constraints). Every now and then, I see someone benchmarking proplists functions and being surprised that lookups are slower than lists:keyfind() or similar. If you're having a problem with proplists performance you're using them wrong. If you need maximum speed, then extract what you need from the proplists and put the data in a form that offers faster access times, typically before you enter the main loop of your program. If you *know* that your lists are strict key/value pairs and will never be anything else, then you shouldn't be calling the proplists functions to begin with - just use lists:keyfind(), which is as fast as it can be. /Richard C -------------- next part -------------- An HTML attachment was scrubbed... URL: From akrupicka@REDACTED Fri Sep 18 11:59:53 2015 From: akrupicka@REDACTED (Adam Krupicka) Date: Fri, 18 Sep 2015 11:59:53 +0200 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: References: <20150917155645.GC9409@fhebert-ltm1> <2920189.nkzdrQtalV@changa> Message-ID: <1442565243-sup-6508@arch.local> > Note that O(1) does not mean fast - it just means "constant" time > without knowing *what* the constant is this information is useless. It's not completely useless. More than anything it tells you that this function scales perfectly, independent on the amount of data you shovel into the function. Of course, the definition of O is such that O(9999) = O(1). > O(N) means the time depends linearly upon N - but again > I doubt if anybody really knows if a function is O(N) - when you run > out of cache or swap space it will certainly not be linear. Again, the big O notation was never meant for exact measurements of how many units of time an algorithm takes to complete on an arbitrary architecture. It is a *very* theoretical tool that requires good understanding to be usable in practice. It doesn't take into account implementation-specific details like cache or swap space. It doesn't even require the definition of a general computation machine (think a Turing Machine) beforehand. The notation is defined purely on mathematical functions O(g) = {f | ?c,n0 ? N : ?n ? n0 : f(n) ? cg(n)}, and it is up to us to interpret these so that we actually get anything interesting out of it. What this notations is good for is describing the relation between the size of input and the amount of steps we can expect the algorithm to take to finish^. As such, if we have n data and an algorithms that runs in O(n^2), we know that if we double n, we can expect the running time to quadruple. Of course, this is just an estimate and in reality we can expect things to get a bit more complicated with concurrency or more frequent cache invalidation. I agree with the rest of what You said, but I think it is important to understand the big O notation before we attempt to use it at all, as then we might end up expecting a whole lot more from it than we're actually getting, and when things go south, blame the notation instead of ourselves. :) [^] This is a simplification. What we mean by the variable n can change depending on the class of algorithms we are talking about. For example, sorting algorithms define n to be the amount of elements we are to sort and define g(x) such that it describes the amount of comparison operations that will take place, e.g. Quicksort executes nlog(n) comparison operations when sorting n elements (in the average case). Cheers, A. K. From jesper.louis.andersen@REDACTED Fri Sep 18 12:00:23 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 18 Sep 2015 12:00:23 +0200 Subject: [erlang-questions] Replacing erlang:now/0 for getting a unique number In-Reply-To: <4157002.n1BnLX5JJz@changa> References: <55FBB77E.70307@gmail.com> <4157002.n1BnLX5JJz@changa> Message-ID: On Fri, Sep 18, 2015 at 9:24 AM, zxq9 wrote: > EventStamp = {monotonic_time(), unique_integer([monotonic])}. This can also repeat! monotonic_time() starts at the same point whenever the Erlang system reboots: -module(z). -export([t/0]). t() -> io:format("~p~n", [erlang:monotonic_time()]), ok. tiefling:~ jlouis$ for i in $(jot 10 1 10) ; do erl -noshell -s z t -s erlang halt; done -576460751926879092 -576460751926869976 -576460751923585212 -576460751924209286 -576460751925308499 -576460751926073063 -576460751927201176 -576460751927514165 -576460751927189332 -576460751925751652 So the more you restart the system, the grater risk of it repeating. You probably want to use erlang:system_time(). -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sperber@REDACTED Fri Sep 18 13:21:51 2015 From: sperber@REDACTED (Michael Sperber) Date: Fri, 18 Sep 2015 13:21:51 +0200 Subject: [erlang-questions] Call for Contributions: BOB 2016 - Berlin, Feb 19, 2016 Message-ID: BOB Conference 2016 "What happens when we use what's best for a change?" http://bobkonf.de/2016/en/cfp.html Berlin, February 19 Call for Contributions Deadline: October 30, 2015 You drive advanced software engineering methods, implement ambitious architectures and are open to cutting-edge innovation? Attend this conference, meet people that share your goals, and get to know the best software tools and technologies available today. We strive to offer a day full of new experiences and impressions that you can use to immediately improve your daily life as a software developer. If you share our vision and want to contribute, submit a proposal for a talk or tutorial! Topics ------ We are looking for talks about best-of-breed software technology, e.g.: - functional programming - reactive programming - persistent data structures and databases - types - formal methods for correctness and robustness - ... everything really that isn't mainstream, but you think should be. Presenters should provide the audience with information that is practically useful for software developers. This could take the form of e.g.: - experience reports - introductory talks on technical background - demos and how-tos Requirements ------------ We accept proposals for presentations of 45 minutes (40 minutes talk + 5 minutes questions), as well as 90 minute tutorials for beginners. The language of presentation should be either English or German. Your proposal should include (in your presentation language of choice): - an abstract of max. 1500 characters. - a short bio/cv - contact information (including at least email address) - a list of 3-5 concrete ideas of how your work can be applied in a developer's daily life - additional material (websites, blogs, slides, videos of past presentations, ...) Submit here: https://docs.google.com/forms/d/1IrCa3ilxMrO2h1G1WC4ywoxdz8wohxaPW3dfiB0cq-8/viewform?usp=send_form Organisation ------------ - submit your proposal here https://docs.google.com/forms/d/1IrCa3ilxMrO2h1G1WC4ywoxdz8wohxaPW3dfiB0cq-8/viewform?usp=send_form - direct questions to `bobkonf at active minus group dot de` - proposal deadline: **October 30, 2015** - notification: November 15, 2015 - program: December 1, 2015 NOTE: The conference fee will be waived for presenters, but travel expenses will not be covered. Program Committee ----------------- (more information here: http://bobkonf.de/2016/programmkomitee.html) - Matthias Fischmann, zerobuzz UG - Matthias Neubauer, SICK AG - Nicole Rauch, Softwareentwicklung und Entwicklungscoaching - Michael Sperber, Active Group - Stefan Wehr, factis research Scientific Advisory Board ------------------------- - Annette Bieniusa, TU Kaiserslautern - Peter Thiemann, Uni Freiburg From federico.carrone@REDACTED Fri Sep 18 21:02:18 2015 From: federico.carrone@REDACTED (Federico Carrone) Date: Fri, 18 Sep 2015 19:02:18 +0000 Subject: [erlang-questions] Fwd: [ANN] Spawned Shelter In-Reply-To: References: Message-ID: I have created a simple website called Spawned Shelter ( http://spawnedshelter.com/). The website tries to collect the best articles, videos and books for learning Erlang, the internals of its virtual machine and other languages that run on top of it. I think that it can be quite useful for the community to have a simple website like this for sending out to people who want to learn Erlang or are learning it. If you want to add or remove any link you only have to send a pull request that changes this markdown file: https://github.com/unbalancedparentheses/spawnedshelter/blob/master/src/content.md Also if you want to help in sharing it to the rest of the wolrd, you can vote it in reddit ( https://www.reddit.com/r/programming/comments/3lgg6v/spawned_shelter_the_best_articles_videos_and/) or hacker news (https://news.ycombinator.com/item?id=10240572). Let me know if you have any recommadation or comment. Regards, Federico. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Fri Sep 18 21:08:07 2015 From: ivan@REDACTED (Ivan Uemlianin) Date: Fri, 18 Sep 2015 20:08:07 +0100 Subject: [erlang-questions] Fwd: [ANN] Spawned Shelter In-Reply-To: References: Message-ID: <1442603289042-35d0e6fc-6d3a803f-6bae83ef@llaisdy.com> Very nice reading list! On Fri, Sep 18, 2015 at 8:02 pm, Federico Carrone wrote: I have created a simple website called Spawned Shelter ( http://spawnedshelter.com/ [http://spawnedshelter.com/] ). The website tries to collect the best articles, videos and books for learning Erlang, the internals of its virtual machine and other languages that run on top of it. I think that it can be quite useful for the community to have a simple website like this for sending out to people who want to learn Erlang or are learning it. If you want to add or remove any link you only have to send a pull request that changes this markdown file: https://github.com/unbalancedparentheses/spawnedshelter/blob/master/src/content.md [https://github.com/unbalancedparentheses/spawnedshelter/blob/master/src/content.md] Also if you want to help in sharing it to the rest of the wolrd, you can vote it in reddit ( https://www.reddit.com/r/programming/comments/3lgg6v/spawned_shelter_the_best_articles_videos_and/ [https://www.reddit.com/r/programming/comments/3lgg6v/spawned_shelter_the_best_articles_videos_and/] ) or hacker news ( https://news.ycombinator.com/item?id=10240572 [https://news.ycombinator.com/item?id=10240572] ). Let me know if you have any recommadation or comment. Regards, Federico. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Sat Sep 19 11:19:05 2015 From: zxq9@REDACTED (zxq9) Date: Sat, 19 Sep 2015 18:19:05 +0900 Subject: [erlang-questions] Fwd: [ANN] Spawned Shelter In-Reply-To: References: Message-ID: <4279207.CEpd2sK0S1@changa> On 2015?9?18? ??? 19:02:18 Federico Carrone wrote: > I have created a simple website called Spawned Shelter ( > http://spawnedshelter.com/). Was skeptical at first -- but its actually a pretty nice collection of useful links. The random horror-movie motif is an interesting choice... "a horror movie" is an accurate summary of how I've come to remember several development experiences (post-outsourcing maintenance and zomg-kitchen-sink situations in particular). One thing... you have to find some way to make the wolf's cup o' Erlang clickable! I have no idea what it should do. Play the psychobitch sequel to the Erlang movie, spew exit/2 bullets, no idea. I will make a game of randomly bothering you about that until you make it do something, anything. ;-) -Craig From eriksoe@REDACTED Sat Sep 19 17:40:34 2015 From: eriksoe@REDACTED (=?UTF-8?Q?Erik_S=C3=B8e_S=C3=B8rensen?=) Date: Sat, 19 Sep 2015 17:40:34 +0200 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: <20150917155645.GC9409@fhebert-ltm1> References: <2423630.4x47EVrrNS@burrito> <20150917155645.GC9409@fhebert-ltm1> Message-ID: As for ETS cost, it may be described in the docs as constant, but in reality for the core operations it's (size of key + size of data moved into or out of table). In many use cases, it makes no difference (because the key and value sizes are small or constant), but for some it does. Den 17/09/2015 17.57 skrev "Fred Hebert" : > On 09/17, Eric des Courtis wrote: > >> Regardless if concurrency and parallelism plays a role *I still need to >> know if an operation in a sequential process I am working on is O(1) or >> O(n)* or am I missing something? >> > > These values are way more worthwhile as a property of libraries than > language, since they're far more impacted by the choice of algorithm taken > there. > > Let's look at a few modules: > > 1. Queue > > "all operations has an amortized O(1) running time, except len/1, join/2, > split/2, filter/2 and member/2 that have O(n). To minimize the size of a > queue minimizing the amount of garbage built by queue operations, the > queues do not contain explicit length information, and that is why len/1 is > O(n)." > > > 2. Maps > > undefined in the module doc, but from the EEP ( > http://www.erlang.org/eeps/eep-0043.html): > > "has at most O(log N) time complexity in insert and lookup operations, > where N is the number of key-value associations." > > 3. Dict > > undefined in docs. I know that it is a very flat tree (bunch of buckets) > acting as a hash map, so O(log N) > > 4. Array > > undefined in docs. Also a very flat tree. O(Log N), where N is the size of > the array, and if I recall, the tree has a branching factor of 10 to > compromise between the speed of lookups and garbage generated when > modifying it. It's described in the source only, sadly. > > 5. Gb_trees > > Doc compares them to AVL trees, without storage overhead and with better > performance. O(log N) it is (those are the worst cases for AVL trees) > > 6. Sets > > They're using the same mechanism as dicts, so O(Log N) > > 7. Gb_sets > > Same mechanism as gb_trees, so O(log N) > > 8. ETS > > Doc says: "These provide the ability to store very large quantities of > data in an Erlang runtime system, and to have constant access time to the > data. (In the case of ordered_set, see below, access time is proportional > to the logarithm of the number of objects stored)." > > 9. Orddict > > Doc says " An orddict is a representation of a dictionary, where a list of > pairs is used to store the keys and values. The list is ordered after the > keys." > > lists are O(N) traversal and insertion, so let's consider this the best > case. > > 10. ordsets > > implemented over orddict, O(N) too. > > 11. sofs > > despite having the most cryptic documentation, it does mention: "The > execution time of the functions of this module is dominated by the time it > takes to sort lists. When no sorting is needed, the execution time is in > the worst case proportional to the sum of the sizes of the input arguments > and the returned value. A few functions execute in constant time: > from_external, is_empty_set, is_set, is_sofs_set, to_external, type." > > O(N log N) for most (optimal sort time, where N = sum(Input)), with a list > of O(1) functions > > 12. Digraph > > Unspecified, no idea > > 13. lists:key* functions > > using lists, so O(N), but they're fast given they're implemented in C > > 14. Proplists > > Using lists, so O(N). > > > > Now with these modules somewhat covered (there's a lot of subtleties in > the APIs making some better than others depending on the access patterns), > it's interesting to look at basic operations we may do a lot, for example: > > 1. Message receiving (non-selective) > > O(1), though accessing the mailbox when it's very large can trigger GCs > and mess up expectations a bit. > > 2. Message receiving (selective) > > O(N) selection where N is the mailbox size. An optimization existsw for > common access patterns where a reference (with make_ref()) is created in > the scope of the receive and used in all clauses, which truncates the > existing mailbox, therefore reducing N. > > 3. Sorting (lists:sort) > > O(N log N), performing a fancypants merge sort > > > > So yeah, for a lot of data structures, the information is already there. > For many, it's also still missing. > > Regards, > Fred. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kovariadam@REDACTED Sat Sep 19 17:01:04 2015 From: kovariadam@REDACTED (=?utf-8?Q?Adam_K=C3=B6v=C3=A1ri?=) Date: Sat, 19 Sep 2015 17:01:04 +0200 Subject: [erlang-questions] erlang:process_info for global processes Message-ID: <17D4B673-6B3D-4FF8-8BA6-A33A58CC438B@gmail.com> Hello I noticed erlang:process_info(, registered_name) returns []. Is this expected? What is the reason this doesn?t work for globally registered processes? Is there a built in function which would work with global processes? Example: 6> global:registered_names(). [test_dumb1,...] 7> 7> Pid = global:whereis_name(test_dumb1). <0.306.0> 8> erlang:process_info(Pid, registered_name). [] Thanks Adam Kovari From vinoski@REDACTED Sat Sep 19 18:49:15 2015 From: vinoski@REDACTED (Steve Vinoski) Date: Sat, 19 Sep 2015 12:49:15 -0400 Subject: [erlang-questions] erlang:process_info for global processes In-Reply-To: <17D4B673-6B3D-4FF8-8BA6-A33A58CC438B@gmail.com> References: <17D4B673-6B3D-4FF8-8BA6-A33A58CC438B@gmail.com> Message-ID: On Sat, Sep 19, 2015 at 11:01 AM, Adam K?v?ri wrote: > Hello > > I noticed erlang:process_info(, registered_name) > returns []. Is this expected? What is the reason this doesn?t work for > globally registered processes? The process_info call returns local process information. The global process support is entirely separate. > Is there a built in function which would work with global processes? > > > Example: > > 6> global:registered_names(). > [test_dumb1,...] > 7> > 7> Pid = global:whereis_name(test_dumb1). > <0.306.0> > 8> erlang:process_info(Pid, registered_name). > [] > I don't see an API function for what you want, but here's a hack to get it: 9> ets:lookup(global_pid_names, Pid). [{<0.306.0>,test_dumb1}] But this reaches into the internals of the global module and so isn't the best thing to do. Perhaps the global module needs a new lookup-by-pid or process_info exported function. Or then again, maybe there's a good reason it's not provided. --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From touch.sereysethy@REDACTED Sat Sep 19 18:34:18 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Sat, 19 Sep 2015 23:34:18 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL Message-ID: Hello, I just recently updated Erlang to latest version OTP 18 on Ubuntu server. It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used to work fine in OTP 17. The program is correctly compiled but during the execution the memory kept increasing. I need to restart the process every one or two hours to free some memory. I have read a post here [ http://erlang.2086793.n4.nabble.com/R18-Unbounded-SSL-Session-ETS-Table-Growth-td4713697.html] which discussed about the ssl_session_cache ETS table which can become very large. The process beam.smp can go up to more than 5G during a few hours of executions. I am not yet sure what is the root cause of this issue. Does anyone know how to fix this? Or where should I look at? Thanks, Sethy -------------- next part -------------- An HTML attachment was scrubbed... URL: From kovariadam@REDACTED Sat Sep 19 19:45:24 2015 From: kovariadam@REDACTED (=?utf-8?Q?Adam_K=C3=B6v=C3=A1ri?=) Date: Sat, 19 Sep 2015 19:45:24 +0200 Subject: [erlang-questions] erlang:process_info for global processes In-Reply-To: References: <17D4B673-6B3D-4FF8-8BA6-A33A58CC438B@gmail.com> Message-ID: <16BF5FB1-88C1-43BC-96F1-0B548DC5C306@gmail.com> Thanks Steve. > On 19.9.2015, at 18:49, Steve Vinoski wrote: > > I don't see an API function for what you want, but here's a hack to get it: > > 9> ets:lookup(global_pid_names, Pid). > [{<0.306.0>,test_dumb1}] > > But this reaches into the internals of the global module and so isn't the best thing to do. Perhaps the global module needs a new lookup-by-pid or process_info exported function. Or then again, maybe there's a good reason it's not provided. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From siraaj@REDACTED Sat Sep 19 20:34:40 2015 From: siraaj@REDACTED (Siraaj Khandkar) Date: Sat, 19 Sep 2015 14:34:40 -0400 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: On Sat, Sep 19, 2015 at 12:34 PM, Sereysethy TOUCH < touch.sereysethy@REDACTED> wrote: > Hello, > > I just recently updated Erlang to latest version OTP 18 on Ubuntu server. > It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used to work > fine in OTP 17. The program is correctly compiled but during the execution > the memory kept increasing. I need to restart the process every one or two > hours to free some memory. > > I have read a post here [ > http://erlang.2086793.n4.nabble.com/R18-Unbounded-SSL-Session-ETS-Table-Growth-td4713697.html] > which discussed about the ssl_session_cache ETS table which can become very > large. > > The process beam.smp can go up to more than 5G during a few hours of > executions. > > I am not yet sure what is the root cause of this issue. > > Does anyone know how to fix this? Or where should I look at? > We just experienced the same after upgrading from 17.0 (leaking over days/weeks) to 17.5.3 (leaking over hours). I should be able to share some data in a few of days. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sat Sep 19 21:22:39 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 19 Sep 2015 21:22:39 +0200 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy TOUCH < touch.sereysethy@REDACTED> wrote: > I just recently updated Erlang to latest version OTP 18 on Ubuntu server. > It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used to work > fine in OTP 17. The program is correctly compiled but during the execution > the memory kept increasing. I need to restart the process every one or two > hours to free some memory. 1. Get a shell console on the node 2. call 'memory().' in the shell 3. If it reports ets as the winner, call 4. ets:i(). This will give you a nice overview for where to look and verify it is the SSL problem. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Sun Sep 20 00:07:56 2015 From: ingela.andin@REDACTED (Ingela Andin) Date: Sun, 20 Sep 2015 00:07:56 +0200 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: Hi! 2015-09-19 18:34 GMT+02:00 Sereysethy TOUCH : > Hello, > > I just recently updated Erlang to latest version OTP 18 on Ubuntu server. > It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used to work > fine in OTP 17. The program is correctly compiled but during the execution > the memory kept increasing. I need to restart the process every one or two > hours to free some memory. > > I have read a post here [ > http://erlang.2086793.n4.nabble.com/R18-Unbounded-SSL-Session-ETS-Table-Growth-td4713697.html] > which discussed about the ssl_session_cache ETS table which can become very > large. > > Well that bug is now fixed for 18.1 and in maint on github. This bug however has been around for a while so it is unlikly that this bug causes 18 to behave different from 17. > The process beam.smp can go up to more than 5G during a few hours of > executions. > > I am not yet sure what is the root cause of this issue. > > Does anyone know how to fix this? Or where should I look at? > You could try using the observer application to inspect the node, makes inspecting easy. Let us know your findings. Regards Ingela Erlang/OTP team - Ericsson AB > > Thanks, Sethy > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From touch.sereysethy@REDACTED Sun Sep 20 15:24:47 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Sun, 20 Sep 2015 20:24:47 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: Hello, I did as what you told me by getting a shell console on the node. I run memory(). and see ets ate the most memory. And it increased overtime. I run ets:i(). and I found out the ssl_manager took a big chunk of what memory used. 32794 client_ssl_otp_session_cache ordered_set 0 89 ssl_manager 36891 server_ssl_otp_session_cache ordered_set 1564 67995 ssl_manager 40993 httpc_manager__session_cookie_db bag 0 299 httpc_manager So I think there is a problem with ssl_manager in Erlang OTP 18. Any workarounds? Sethy On Sun, Sep 20, 2015 at 2:22 AM, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > > On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy TOUCH < > touch.sereysethy@REDACTED> wrote: > >> I just recently updated Erlang to latest version OTP 18 on Ubuntu server. >> It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used to work >> fine in OTP 17. The program is correctly compiled but during the execution >> the memory kept increasing. I need to restart the process every one or two >> hours to free some memory. > > > 1. Get a shell console on the node > 2. call 'memory().' in the shell > 3. If it reports ets as the winner, call > 4. ets:i(). > > This will give you a nice overview for where to look and verify it is the > SSL problem. > > > -- > J. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From touch.sereysethy@REDACTED Sun Sep 20 15:30:49 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Sun, 20 Sep 2015 20:30:49 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: Dear Andin, To be honest, I am new to Erlang. So I am sorry sometimes for my naive questions. I just tried to fix and debug erlang code written by another person. I am not sure yet where to look at. Since we dont use anything special about SSL, beside we ask user browser to present certificate and pass it on to our agent backend to process it. I look at the code, it seems nothing is unusual. You said this bug exists since Erlang 17? But why there were no memory leak when it ran on Erlang 17. I did as Jesper Louis Andersen suggested, and I saw there a big amount of memory being used by ssl_manager. 32794 client_ssl_otp_session_cache ordered_set 0 89 ssl_manager 36891 server_ssl_otp_session_cache ordered_set 1564 67995 ssl_manager 40993 httpc_manager__session_cookie_db bag 0 299 httpc_manager And it keeps increasing overtime. Can you let me know or pointing me to any documents on how to use observer application? Thanks, Sethy On Sun, Sep 20, 2015 at 5:07 AM, Ingela Andin wrote: > Hi! > > 2015-09-19 18:34 GMT+02:00 Sereysethy TOUCH : > >> Hello, >> >> I just recently updated Erlang to latest version OTP 18 on Ubuntu server. >> It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used to work >> fine in OTP 17. The program is correctly compiled but during the execution >> the memory kept increasing. I need to restart the process every one or two >> hours to free some memory. >> >> I have read a post here [ >> http://erlang.2086793.n4.nabble.com/R18-Unbounded-SSL-Session-ETS-Table-Growth-td4713697.html] >> which discussed about the ssl_session_cache ETS table which can become very >> large. >> >> > Well that bug is now fixed for 18.1 and in maint on github. This bug > however has been around for a while so it is unlikly that this bug causes > 18 to behave different from 17. > > > >> The process beam.smp can go up to more than 5G during a few hours of >> executions. >> >> I am not yet sure what is the root cause of this issue. >> >> Does anyone know how to fix this? Or where should I look at? >> > > > You could try using the observer application to inspect the node, makes > inspecting easy. > Let us know your findings. > > Regards Ingela Erlang/OTP team - Ericsson AB > > > >> >> Thanks, Sethy >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sun Sep 20 16:44:50 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sun, 20 Sep 2015 16:44:50 +0200 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: Ingela suggested you try building the "maint" branch, which has a fix for the session cache problem (by Ingela :) simply git pull https://github.com/erlang/otp checkout the maint branch build the result. Alterantively, the 'kerl' application (also on github) can build Erlang. It has an experimental git-mode which may be useful here. I think you are looking at the SSL session cache problem. As for observer: observer:start() will start it: * You need wx in your Erlang installation, it is a GUI application * You need something which can run wx applications. It is somewhat easy in a local dev environment, but the shell commands can be useful in contexts with no easy GUI-line :) On Sun, Sep 20, 2015 at 3:30 PM, Sereysethy TOUCH < touch.sereysethy@REDACTED> wrote: > Dear Andin, > > To be honest, I am new to Erlang. So I am sorry sometimes for my naive > questions. > > I just tried to fix and debug erlang code written by another person. I am > not sure yet where to look at. Since we dont use anything special about > SSL, beside we ask user browser to present certificate and pass it on to > our agent backend to process it. I look at the code, it seems nothing is > unusual. > > You said this bug exists since Erlang 17? But why there were no memory > leak when it ran on Erlang 17. > > I did as Jesper Louis Andersen suggested, and I saw there a big amount of > memory being used by ssl_manager. > > 32794 client_ssl_otp_session_cache ordered_set 0 89 > ssl_manager > 36891 server_ssl_otp_session_cache ordered_set 1564 67995 > ssl_manager > 40993 httpc_manager__session_cookie_db bag 0 299 > httpc_manager > > And it keeps increasing overtime. > > Can you let me know or pointing me to any documents on how to use observer > application? > > Thanks, Sethy > > On Sun, Sep 20, 2015 at 5:07 AM, Ingela Andin > wrote: > >> Hi! >> >> 2015-09-19 18:34 GMT+02:00 Sereysethy TOUCH : >> >>> Hello, >>> >>> I just recently updated Erlang to latest version OTP 18 on Ubuntu >>> server. It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used >>> to work fine in OTP 17. The program is correctly compiled but during the >>> execution the memory kept increasing. I need to restart the process every >>> one or two hours to free some memory. >>> >>> I have read a post here [ >>> http://erlang.2086793.n4.nabble.com/R18-Unbounded-SSL-Session-ETS-Table-Growth-td4713697.html] >>> which discussed about the ssl_session_cache ETS table which can become very >>> large. >>> >>> >> Well that bug is now fixed for 18.1 and in maint on github. This bug >> however has been around for a while so it is unlikly that this bug causes >> 18 to behave different from 17. >> >> >> >>> The process beam.smp can go up to more than 5G during a few hours of >>> executions. >>> >>> I am not yet sure what is the root cause of this issue. >>> >>> Does anyone know how to fix this? Or where should I look at? >>> >> >> >> You could try using the observer application to inspect the node, makes >> inspecting easy. >> Let us know your findings. >> >> Regards Ingela Erlang/OTP team - Ericsson AB >> >> >> >>> >>> Thanks, Sethy >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From touch.sereysethy@REDACTED Sun Sep 20 17:53:43 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Sun, 20 Sep 2015 22:53:43 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: Hello, When do you think a new version erlang 18 will be released which fixes this problem? So you suggest that I should build the maint branch? And what to do with erlang installed on my server? I will stick with shell. No gui. Sorry I seem confused. Need more helps here :-) On Sunday, 20 September 2015, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > Ingela suggested you try building the "maint" branch, which has a fix for > the session cache problem (by Ingela :) > > simply git pull https://github.com/erlang/otp > > checkout the maint branch > > build the result. Alterantively, the 'kerl' application (also on github) > can build Erlang. It has an experimental git-mode which may be useful here. > > I think you are looking at the SSL session cache problem. > > As for observer: observer:start() will start it: > > * You need wx in your Erlang installation, it is a GUI application > * You need something which can run wx applications. It is somewhat easy in > a local dev environment, but the shell commands can be useful in contexts > with no easy GUI-line :) > > > > On Sun, Sep 20, 2015 at 3:30 PM, Sereysethy TOUCH < > touch.sereysethy@REDACTED > > wrote: > >> Dear Andin, >> >> To be honest, I am new to Erlang. So I am sorry sometimes for my naive >> questions. >> >> I just tried to fix and debug erlang code written by another person. I am >> not sure yet where to look at. Since we dont use anything special about >> SSL, beside we ask user browser to present certificate and pass it on to >> our agent backend to process it. I look at the code, it seems nothing is >> unusual. >> >> You said this bug exists since Erlang 17? But why there were no memory >> leak when it ran on Erlang 17. >> >> I did as Jesper Louis Andersen suggested, and I saw there a big amount >> of memory being used by ssl_manager. >> >> 32794 client_ssl_otp_session_cache ordered_set 0 89 >> ssl_manager >> 36891 server_ssl_otp_session_cache ordered_set 1564 67995 >> ssl_manager >> 40993 httpc_manager__session_cookie_db bag 0 299 >> httpc_manager >> >> And it keeps increasing overtime. >> >> Can you let me know or pointing me to any documents on how to use >> observer application? >> >> Thanks, Sethy >> >> On Sun, Sep 20, 2015 at 5:07 AM, Ingela Andin > > wrote: >> >>> Hi! >>> >>> 2015-09-19 18:34 GMT+02:00 Sereysethy TOUCH >> >: >>> >>>> Hello, >>>> >>>> I just recently updated Erlang to latest version OTP 18 on Ubuntu >>>> server. It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used >>>> to work fine in OTP 17. The program is correctly compiled but during the >>>> execution the memory kept increasing. I need to restart the process every >>>> one or two hours to free some memory. >>>> >>>> I have read a post here [ >>>> http://erlang.2086793.n4.nabble.com/R18-Unbounded-SSL-Session-ETS-Table-Growth-td4713697.html] >>>> which discussed about the ssl_session_cache ETS table which can become very >>>> large. >>>> >>>> >>> Well that bug is now fixed for 18.1 and in maint on github. This bug >>> however has been around for a while so it is unlikly that this bug causes >>> 18 to behave different from 17. >>> >>> >>> >>>> The process beam.smp can go up to more than 5G during a few hours of >>>> executions. >>>> >>>> I am not yet sure what is the root cause of this issue. >>>> >>>> Does anyone know how to fix this? Or where should I look at? >>>> >>> >>> >>> You could try using the observer application to inspect the node, makes >>> inspecting easy. >>> Let us know your findings. >>> >>> Regards Ingela Erlang/OTP team - Ericsson AB >>> >>> >>> >>>> >>>> Thanks, Sethy >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > > -- > J. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From touch.sereysethy@REDACTED Sun Sep 20 18:04:58 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Sun, 20 Sep 2015 23:04:58 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: FYI, I use erlang.mk to build my application. On Sun, Sep 20, 2015 at 10:53 PM, Sereysethy TOUCH < touch.sereysethy@REDACTED> wrote: > Hello, > > When do you think a new version erlang 18 will be released which fixes > this problem? > > So you suggest that I should build the maint branch? And what to do with > erlang installed on my server? > > I will stick with shell. No gui. > > Sorry I seem confused. Need more helps here :-) > > > > On Sunday, 20 September 2015, Jesper Louis Andersen < > jesper.louis.andersen@REDACTED> wrote: > >> Ingela suggested you try building the "maint" branch, which has a fix for >> the session cache problem (by Ingela :) >> >> simply git pull https://github.com/erlang/otp >> >> checkout the maint branch >> >> build the result. Alterantively, the 'kerl' application (also on github) >> can build Erlang. It has an experimental git-mode which may be useful here. >> >> I think you are looking at the SSL session cache problem. >> >> As for observer: observer:start() will start it: >> >> * You need wx in your Erlang installation, it is a GUI application >> * You need something which can run wx applications. It is somewhat easy >> in a local dev environment, but the shell commands can be useful in >> contexts with no easy GUI-line :) >> >> >> >> On Sun, Sep 20, 2015 at 3:30 PM, Sereysethy TOUCH < >> touch.sereysethy@REDACTED> wrote: >> >>> Dear Andin, >>> >>> To be honest, I am new to Erlang. So I am sorry sometimes for my naive >>> questions. >>> >>> I just tried to fix and debug erlang code written by another person. I >>> am not sure yet where to look at. Since we dont use anything special about >>> SSL, beside we ask user browser to present certificate and pass it on to >>> our agent backend to process it. I look at the code, it seems nothing is >>> unusual. >>> >>> You said this bug exists since Erlang 17? But why there were no memory >>> leak when it ran on Erlang 17. >>> >>> I did as Jesper Louis Andersen suggested, and I saw there a big amount >>> of memory being used by ssl_manager. >>> >>> 32794 client_ssl_otp_session_cache ordered_set 0 89 >>> ssl_manager >>> 36891 server_ssl_otp_session_cache ordered_set 1564 67995 >>> ssl_manager >>> 40993 httpc_manager__session_cookie_db bag 0 299 >>> httpc_manager >>> >>> And it keeps increasing overtime. >>> >>> Can you let me know or pointing me to any documents on how to use >>> observer application? >>> >>> Thanks, Sethy >>> >>> On Sun, Sep 20, 2015 at 5:07 AM, Ingela Andin >>> wrote: >>> >>>> Hi! >>>> >>>> 2015-09-19 18:34 GMT+02:00 Sereysethy TOUCH >>> >: >>>> >>>>> Hello, >>>>> >>>>> I just recently updated Erlang to latest version OTP 18 on Ubuntu >>>>> server. It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used >>>>> to work fine in OTP 17. The program is correctly compiled but during the >>>>> execution the memory kept increasing. I need to restart the process every >>>>> one or two hours to free some memory. >>>>> >>>>> I have read a post here [ >>>>> http://erlang.2086793.n4.nabble.com/R18-Unbounded-SSL-Session-ETS-Table-Growth-td4713697.html] >>>>> which discussed about the ssl_session_cache ETS table which can become very >>>>> large. >>>>> >>>>> >>>> Well that bug is now fixed for 18.1 and in maint on github. This bug >>>> however has been around for a while so it is unlikly that this bug causes >>>> 18 to behave different from 17. >>>> >>>> >>>> >>>>> The process beam.smp can go up to more than 5G during a few hours of >>>>> executions. >>>>> >>>>> I am not yet sure what is the root cause of this issue. >>>>> >>>>> Does anyone know how to fix this? Or where should I look at? >>>>> >>>> >>>> >>>> You could try using the observer application to inspect the node, makes >>>> inspecting easy. >>>> Let us know your findings. >>>> >>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>> >>>> >>>> >>>>> >>>>> Thanks, Sethy >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> >>>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> >> >> -- >> J. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sun Sep 20 18:18:11 2015 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Sun, 20 Sep 2015 18:18:11 +0200 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: <55FEDC43.80803@ninenines.eu> As far as I understand, 18.1 will be released very soon, maybe this week, so perhaps your problems will be gone in a few days without having to do anything. :-) On 09/20/2015 06:04 PM, Sereysethy TOUCH wrote: > FYI, I use erlang.mk to build my application. > > On Sun, Sep 20, 2015 at 10:53 PM, Sereysethy TOUCH > > wrote: > > Hello, > > When do you think a new version erlang 18 will be released which > fixes this problem? > > So you suggest that I should build the maint branch? And what to do > with erlang installed on my server? > > I will stick with shell. No gui. > > Sorry I seem confused. Need more helps here :-) > > > > On Sunday, 20 September 2015, Jesper Louis Andersen > > wrote: > > Ingela suggested you try building the "maint" branch, which has > a fix for the session cache problem (by Ingela :) > > simply git pull https://github.com/erlang/otp > > checkout the maint branch > > build the result. Alterantively, the 'kerl' application (also on > github) can build Erlang. It has an experimental git-mode which > may be useful here. > > I think you are looking at the SSL session cache problem. > > As for observer: observer:start() will start it: > > * You need wx in your Erlang installation, it is a GUI application > * You need something which can run wx applications. It is > somewhat easy in a local dev environment, but the shell commands > can be useful in contexts with no easy GUI-line :) > > > > On Sun, Sep 20, 2015 at 3:30 PM, Sereysethy TOUCH > wrote: > > Dear Andin, > > To be honest, I am new to Erlang. So I am sorry sometimes > for my naive questions. > > I just tried to fix and debug erlang code written by another > person. I am not sure yet where to look at. Since we dont > use anything special about SSL, beside we ask user browser > to present certificate and pass it on to our agent backend > to process it. I look at the code, it seems nothing is unusual. > > You said this bug exists since Erlang 17? But why there were > no memory leak when it ran on Erlang 17. > > I did as Jesper Louis Andersen suggested, and I saw there a > big amount of memory being used by ssl_manager. > > 32794 client_ssl_otp_session_cache ordered_set 0 > 89 ssl_manager > 36891 server_ssl_otp_session_cache ordered_set > 1564 67995 ssl_manager > 40993 httpc_manager__session_cookie_db bag 0 > 299 httpc_manager > > And it keeps increasing overtime. > > Can you let me know or pointing me to any documents on how > to use observer application? > > Thanks, Sethy > > On Sun, Sep 20, 2015 at 5:07 AM, Ingela Andin > wrote: > > Hi! > > 2015-09-19 18:34 GMT+02:00 Sereysethy TOUCH > : > > Hello, > > I just recently updated Erlang to latest version OTP > 18 on Ubuntu server. It uses cowboy (websocket), > ranch, ssl, erlydtl & rabbitmq. It used to work fine > in OTP 17. The program is correctly compiled but > during the execution the memory kept increasing. I > need to restart the process every one or two hours > to free some memory. > > I have read a post here > [http://erlang.2086793.n4.nabble.com/R18-Unbounded-SSL-Session-ETS-Table-Growth-td4713697.html] > which discussed about the ssl_session_cache ETS > table which can become very large. > > > Well that bug is now fixed for 18.1 and in maint on > github. This bug however has been around for a while so > it is unlikly that this bug causes 18 to behave > different from 17. > > The process beam.smp can go up to more than 5G > during a few hours of executions. > > I am not yet sure what is the root cause of this issue. > > Does anyone know how to fix this? Or where should I > look at? > > > > You could try using the observer application to inspect > the node, makes inspecting easy. > Let us know your findings. > > Regards Ingela Erlang/OTP team - Ericsson AB > > > Thanks, Sethy > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -- > J. > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu Author of The Erlanger Playbook, A book about software development using Erlang From rdm@REDACTED Sun Sep 20 18:22:35 2015 From: rdm@REDACTED (Rich Morin) Date: Sun, 20 Sep 2015 09:22:35 -0700 Subject: [erlang-questions] =?iso-8859-1?q?Elixir/Erlang/Phoen=ADix_OTP_Bo?= =?iso-8859-1?q?otcamp_in_Mountain_View=2C_CA=2C_9/22-23?= Message-ID: I haven't seen much publicity for this event and I see that there are still about two dozen spots available. So, if you're located in the SF Bay Area and are interested in this sort of thing, go sign up! http://www.meetup.com/Hackers-and-Founders/events/223275643 http://hf.cx/events/2015/09/23/hf-elixirerlangphoenix-otp-bootcamp-two-days -r -- http://www.cfcl.com/rdm Rich Morin rdm@REDACTED http://www.cfcl.com/rdm/resume San Bruno, CA, USA +1 650-873-7841 Software system design, development, and documentation From zhuo.quan@REDACTED Mon Sep 21 03:46:36 2015 From: zhuo.quan@REDACTED (=?utf-8?B?5YWo5Y2T?=) Date: Mon, 21 Sep 2015 09:46:36 +0800 Subject: [erlang-questions] multi adaptor problem. Message-ID: Hi,list: My computer has 2 adaptor, one local adaptor(ip:10.206.25.230) and another USB Wireless LAN Card(ip:192.168.1.100). Once I turned the USB Wireless LAN Card on, it will take a long time(about 5 seconds) to start the erlang vm when I type ?erl? in the terminal. However, it will behave normal once I turn the wireless card off. Anyone help? The OS is fedora 22,and OTP version 17, erts 6.3. Thanks Quan Zhuo -------------- next part -------------- An HTML attachment was scrubbed... URL: From touch.sereysethy@REDACTED Mon Sep 21 07:39:03 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Mon, 21 Sep 2015 12:39:03 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: On Sunday, 20 September 2015, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > Ingela suggested you try building the "maint" branch, which has a fix for > the session cache problem (by Ingela :) > > simply git pull https://github.com/erlang/otp > > checkout the maint branch > > build the result. Alterantively, the 'kerl' application (also on github) > can build Erlang. It has an experimental git-mode which may be useful here. > > I think you are looking at the SSL session cache problem. > > As for observer: observer:start() will start it: > > * You need wx in your Erlang installation, it is a GUI application > * You need something which can run wx applications. It is somewhat easy in > a local dev environment, but the shell commands can be useful in contexts > with no easy GUI-line :) > > > > On Sun, Sep 20, 2015 at 3:30 PM, Sereysethy TOUCH < > touch.sereysethy@REDACTED > > wrote: > >> Dear Andin, >> >> To be honest, I am new to Erlang. So I am sorry sometimes for my naive >> questions. >> >> I just tried to fix and debug erlang code written by another person. I am >> not sure yet where to look at. Since we dont use anything special about >> SSL, beside we ask user browser to present certificate and pass it on to >> our agent backend to process it. I look at the code, it seems nothing is >> unusual. >> >> You said this bug exists since Erlang 17? But why there were no memory >> leak when it ran on Erlang 17. >> >> I did as Jesper Louis Andersen suggested, and I saw there a big amount >> of memory being used by ssl_manager. >> >> 32794 client_ssl_otp_session_cache ordered_set 0 89 >> ssl_manager >> 36891 server_ssl_otp_session_cache ordered_set 1564 67995 >> ssl_manager >> 40993 httpc_manager__session_cookie_db bag 0 299 >> httpc_manager >> >> And it keeps increasing overtime. >> >> Can you let me know or pointing me to any documents on how to use >> observer application? >> >> Thanks, Sethy >> >> On Sun, Sep 20, 2015 at 5:07 AM, Ingela Andin > > wrote: >> >>> Hi! >>> >>> 2015-09-19 18:34 GMT+02:00 Sereysethy TOUCH >> >: >>> >>>> Hello, >>>> >>>> I just recently updated Erlang to latest version OTP 18 on Ubuntu >>>> server. It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used >>>> to work fine in OTP 17. The program is correctly compiled but during the >>>> execution the memory kept increasing. I need to restart the process every >>>> one or two hours to free some memory. >>>> >>>> I have read a post here [ >>>> http://erlang.2086793.n4.nabble.com/R18-Unbounded-SSL-Session-ETS-Table-Growth-td4713697.html] >>>> which discussed about the ssl_session_cache ETS table which can become very >>>> large. >>>> >>>> >>> Well that bug is now fixed for 18.1 and in maint on github. This bug >>> however has been around for a while so it is unlikly that this bug causes >>> 18 to behave different from 17. >>> >>> >>> >>>> The process beam.smp can go up to more than 5G during a few hours of >>>> executions. >>>> >>>> I am not yet sure what is the root cause of this issue. >>>> >>>> Does anyone know how to fix this? Or where should I look at? >>>> >>> >>> >>> You could try using the observer application to inspect the node, makes >>> inspecting easy. >>> Let us know your findings. >>> >>> Regards Ingela Erlang/OTP team - Ericsson AB >>> >>> >>> >>>> >>>> Thanks, Sethy >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > > -- > J. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From touch.sereysethy@REDACTED Mon Sep 21 07:42:02 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Mon, 21 Sep 2015 12:42:02 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: Hello, I tried to build erlang from maint branch on two machines, both successfully. But when I tried to compile my application, it failed to load their dependencies: ssl, crypto, etc. Any suggestions on which steps I missed during build or where to look at? I can run erl, and get the latest version of erts 18.0.3. Thanks, Sethy On Sunday, 20 September 2015, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > Ingela suggested you try building the "maint" branch, which has a fix for > the session cache problem (by Ingela :) > > simply git pull https://github.com/erlang/otp > > checkout the maint branch > > build the result. Alterantively, the 'kerl' application (also on github) > can build Erlang. It has an experimental git-mode which may be useful here. > > I think you are looking at the SSL session cache problem. > > As for observer: observer:start() will start it: > > * You need wx in your Erlang installation, it is a GUI application > * You need something which can run wx applications. It is somewhat easy in > a local dev environment, but the shell commands can be useful in contexts > with no easy GUI-line :) > > > > On Sun, Sep 20, 2015 at 3:30 PM, Sereysethy TOUCH < > touch.sereysethy@REDACTED > > wrote: > >> Dear Andin, >> >> To be honest, I am new to Erlang. So I am sorry sometimes for my naive >> questions. >> >> I just tried to fix and debug erlang code written by another person. I am >> not sure yet where to look at. Since we dont use anything special about >> SSL, beside we ask user browser to present certificate and pass it on to >> our agent backend to process it. I look at the code, it seems nothing is >> unusual. >> >> You said this bug exists since Erlang 17? But why there were no memory >> leak when it ran on Erlang 17. >> >> I did as Jesper Louis Andersen suggested, and I saw there a big amount >> of memory being used by ssl_manager. >> >> 32794 client_ssl_otp_session_cache ordered_set 0 89 >> ssl_manager >> 36891 server_ssl_otp_session_cache ordered_set 1564 67995 >> ssl_manager >> 40993 httpc_manager__session_cookie_db bag 0 299 >> httpc_manager >> >> And it keeps increasing overtime. >> >> Can you let me know or pointing me to any documents on how to use >> observer application? >> >> Thanks, Sethy >> >> On Sun, Sep 20, 2015 at 5:07 AM, Ingela Andin > > wrote: >> >>> Hi! >>> >>> 2015-09-19 18:34 GMT+02:00 Sereysethy TOUCH >> >: >>> >>>> Hello, >>>> >>>> I just recently updated Erlang to latest version OTP 18 on Ubuntu >>>> server. It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used >>>> to work fine in OTP 17. The program is correctly compiled but during the >>>> execution the memory kept increasing. I need to restart the process every >>>> one or two hours to free some memory. >>>> >>>> I have read a post here [ >>>> http://erlang.2086793.n4.nabble.com/R18-Unbounded-SSL-Session-ETS-Table-Growth-td4713697.html] >>>> which discussed about the ssl_session_cache ETS table which can become very >>>> large. >>>> >>>> >>> Well that bug is now fixed for 18.1 and in maint on github. This bug >>> however has been around for a while so it is unlikly that this bug causes >>> 18 to behave different from 17. >>> >>> >>> >>>> The process beam.smp can go up to more than 5G during a few hours of >>>> executions. >>>> >>>> I am not yet sure what is the root cause of this issue. >>>> >>>> Does anyone know how to fix this? Or where should I look at? >>>> >>> >>> >>> You could try using the observer application to inspect the node, makes >>> inspecting easy. >>> Let us know your findings. >>> >>> Regards Ingela Erlang/OTP team - Ericsson AB >>> >>> >>> >>>> >>>> Thanks, Sethy >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > > -- > J. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Sep 21 08:54:00 2015 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Mon, 21 Sep 2015 08:54:00 +0200 Subject: [erlang-questions] multi adaptor problem. In-Reply-To: References: Message-ID: <55FFA988.8080906@ninenines.eu> Check if the name of your computer is listed in your hosts file. I experienced the same as you and adding 'alice' (my computer name) on the 127.0.0.1 line in /etc/hosts fixed it for me. On 09/21/2015 03:46 AM, ?? wrote: > Hi,list: > > My computer has 2 adaptor, one local adaptor(ip:10.206.25.230) and > another USB Wireless LAN Card(ip:192.168.1.100). Once I turned the USB > Wireless LAN Card on, it will take a long time(about 5 seconds) to start > the erlang vm when I type ?erl? in the terminal. However, it will behave > normal once I turn the wireless card off. > > Anyone help? > > The OS is fedora 22,and OTP version 17, erts 6.3. > > Thanks > > Quan Zhuo > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu Author of The Erlanger Playbook, A book about software development using Erlang From touch.sereysethy@REDACTED Mon Sep 21 09:04:25 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Mon, 21 Sep 2015 14:04:25 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: <55FEDC43.80803@ninenines.eu> References: <55FEDC43.80803@ninenines.eu> Message-ID: Hello, I use erlang.mk to build my application. After I successfully built erlang from maint branch using kerl, when I compiled my application, I got this error: My erlang is Erlang/OTP 18 [erts-7.0.3] [source-5991161] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] Failed to solve release: Dependency ssl is specified as a dependency but is not reachable by the system. Any ideas? On Sun, Sep 20, 2015 at 11:18 PM, Lo?c Hoguin wrote: > As far as I understand, 18.1 will be released very soon, maybe this week, > so perhaps your problems will be gone in a few days without having to do > anything. :-) > > On 09/20/2015 06:04 PM, Sereysethy TOUCH wrote: > >> FYI, I use erlang.mk to build my application. >> >> On Sun, Sep 20, 2015 at 10:53 PM, Sereysethy TOUCH >> > wrote: >> >> Hello, >> >> When do you think a new version erlang 18 will be released which >> fixes this problem? >> >> So you suggest that I should build the maint branch? And what to do >> with erlang installed on my server? >> >> I will stick with shell. No gui. >> >> Sorry I seem confused. Need more helps here :-) >> >> >> >> On Sunday, 20 September 2015, Jesper Louis Andersen >> > > wrote: >> >> Ingela suggested you try building the "maint" branch, which has >> a fix for the session cache problem (by Ingela :) >> >> simply git pull https://github.com/erlang/otp >> >> checkout the maint branch >> >> build the result. Alterantively, the 'kerl' application (also on >> github) can build Erlang. It has an experimental git-mode which >> may be useful here. >> >> I think you are looking at the SSL session cache problem. >> >> As for observer: observer:start() will start it: >> >> * You need wx in your Erlang installation, it is a GUI application >> * You need something which can run wx applications. It is >> somewhat easy in a local dev environment, but the shell commands >> can be useful in contexts with no easy GUI-line :) >> >> >> >> On Sun, Sep 20, 2015 at 3:30 PM, Sereysethy TOUCH >> wrote: >> >> Dear Andin, >> >> To be honest, I am new to Erlang. So I am sorry sometimes >> for my naive questions. >> >> I just tried to fix and debug erlang code written by another >> person. I am not sure yet where to look at. Since we dont >> use anything special about SSL, beside we ask user browser >> to present certificate and pass it on to our agent backend >> to process it. I look at the code, it seems nothing is >> unusual. >> >> You said this bug exists since Erlang 17? But why there were >> no memory leak when it ran on Erlang 17. >> >> I did as Jesper Louis Andersen suggested, and I saw there a >> big amount of memory being used by ssl_manager. >> >> 32794 client_ssl_otp_session_cache ordered_set 0 >> 89 ssl_manager >> 36891 server_ssl_otp_session_cache ordered_set >> 1564 67995 ssl_manager >> 40993 httpc_manager__session_cookie_db bag 0 >> 299 httpc_manager >> >> And it keeps increasing overtime. >> >> Can you let me know or pointing me to any documents on how >> to use observer application? >> >> Thanks, Sethy >> >> On Sun, Sep 20, 2015 at 5:07 AM, Ingela Andin >> wrote: >> >> Hi! >> >> 2015-09-19 18:34 GMT+02:00 Sereysethy TOUCH >> : >> >> Hello, >> >> I just recently updated Erlang to latest version OTP >> 18 on Ubuntu server. It uses cowboy (websocket), >> ranch, ssl, erlydtl & rabbitmq. It used to work fine >> in OTP 17. The program is correctly compiled but >> during the execution the memory kept increasing. I >> need to restart the process every one or two hours >> to free some memory. >> >> I have read a post here >> [ >> http://erlang.2086793.n4.nabble.com/R18-Unbounded-SSL-Session-ETS-Table-Growth-td4713697.html >> ] >> which discussed about the ssl_session_cache ETS >> table which can become very large. >> >> >> Well that bug is now fixed for 18.1 and in maint on >> github. This bug however has been around for a while so >> it is unlikly that this bug causes 18 to behave >> different from 17. >> >> The process beam.smp can go up to more than 5G >> during a few hours of executions. >> >> I am not yet sure what is the root cause of this >> issue. >> >> Does anyone know how to fix this? Or where should I >> look at? >> >> >> >> You could try using the observer application to inspect >> the node, makes inspecting easy. >> Let us know your findings. >> >> Regards Ingela Erlang/OTP team - Ericsson AB >> >> >> Thanks, Sethy >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> >> -- >> J. >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > http://ninenines.eu > Author of The Erlanger Playbook, > A book about software development using Erlang > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Sep 21 09:11:30 2015 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Mon, 21 Sep 2015 09:11:30 +0200 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: <55FEDC43.80803@ninenines.eu> Message-ID: <55FFADA2.4050901@ninenines.eu> You need the dev library of OpenSSL when you compile Erlang, otherwise crypto, public_key and ssl won't be available. On 09/21/2015 09:04 AM, Sereysethy TOUCH wrote: > Hello, > > I use erlang.mk to build my application. After I > successfully built erlang from maint branch using kerl, when I compiled > my application, I got this error: > > My erlang is Erlang/OTP 18 [erts-7.0.3] [source-5991161] [64-bit] > [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] > > Failed to solve release: > > Dependency ssl is specified as a dependency but is not reachable by > the system. > > > Any ideas? > > > On Sun, Sep 20, 2015 at 11:18 PM, Lo?c Hoguin > wrote: > > As far as I understand, 18.1 will be released very soon, maybe this > week, so perhaps your problems will be gone in a few days without > having to do anything. :-) > > On 09/20/2015 06:04 PM, Sereysethy TOUCH wrote: > > FYI, I use erlang.mk to > build my application. > > On Sun, Sep 20, 2015 at 10:53 PM, Sereysethy TOUCH > > >> wrote: > > Hello, > > When do you think a new version erlang 18 will be released > which > fixes this problem? > > So you suggest that I should build the maint branch? And > what to do > with erlang installed on my server? > > I will stick with shell. No gui. > > Sorry I seem confused. Need more helps here :-) > > > > On Sunday, 20 September 2015, Jesper Louis Andersen > > >> wrote: > > Ingela suggested you try building the "maint" branch, > which has > a fix for the session cache problem (by Ingela :) > > simply git pull https://github.com/erlang/otp > > checkout the maint branch > > build the result. Alterantively, the 'kerl' application > (also on > github) can build Erlang. It has an experimental > git-mode which > may be useful here. > > I think you are looking at the SSL session cache problem. > > As for observer: observer:start() will start it: > > * You need wx in your Erlang installation, it is a GUI > application > * You need something which can run wx applications. It is > somewhat easy in a local dev environment, but the shell > commands > can be useful in contexts with no easy GUI-line :) > > > > On Sun, Sep 20, 2015 at 3:30 PM, Sereysethy TOUCH > > wrote: > > Dear Andin, > > To be honest, I am new to Erlang. So I am sorry > sometimes > for my naive questions. > > I just tried to fix and debug erlang code written > by another > person. I am not sure yet where to look at. Since > we dont > use anything special about SSL, beside we ask user > browser > to present certificate and pass it on to our agent > backend > to process it. I look at the code, it seems nothing > is unusual. > > You said this bug exists since Erlang 17? But why > there were > no memory leak when it ran on Erlang 17. > > I did as Jesper Louis Andersen suggested, and I saw > there a > big amount of memory being used by ssl_manager. > > 32794 client_ssl_otp_session_cache > ordered_set 0 > 89 ssl_manager > 36891 server_ssl_otp_session_cache > ordered_set > 1564 67995 ssl_manager > 40993 httpc_manager__session_cookie_db > bag 0 > 299 httpc_manager > > And it keeps increasing overtime. > > Can you let me know or pointing me to any documents > on how > to use observer application? > > Thanks, Sethy > > On Sun, Sep 20, 2015 at 5:07 AM, Ingela Andin > > wrote: > > Hi! > > 2015-09-19 18:34 GMT+02:00 Sereysethy TOUCH > >: > > Hello, > > I just recently updated Erlang to latest > version OTP > 18 on Ubuntu server. It uses cowboy > (websocket), > ranch, ssl, erlydtl & rabbitmq. It used to > work fine > in OTP 17. The program is correctly > compiled but > during the execution the memory kept > increasing. I > need to restart the process every one or > two hours > to free some memory. > > I have read a post here > > [http://erlang.2086793.n4.nabble.com/R18-Unbounded-SSL-Session-ETS-Table-Growth-td4713697.html] > which discussed about the ssl_session_cache ETS > table which can become very large. > > > Well that bug is now fixed for 18.1 and in maint on > github. This bug however has been around for a > while so > it is unlikly that this bug causes 18 to behave > different from 17. > > The process beam.smp can go up to more than 5G > during a few hours of executions. > > I am not yet sure what is the root cause of > this issue. > > Does anyone know how to fix this? Or where > should I > look at? > > > > You could try using the observer application to > inspect > the node, makes inspecting easy. > Let us know your findings. > > Regards Ingela Erlang/OTP team - Ericsson AB > > > Thanks, Sethy > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -- > J. > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -- > Lo?c Hoguin > http://ninenines.eu > Author of The Erlanger Playbook, > A book about software development using Erlang > > -- Lo?c Hoguin http://ninenines.eu Author of The Erlanger Playbook, A book about software development using Erlang From ingela.andin@REDACTED Mon Sep 21 09:51:38 2015 From: ingela.andin@REDACTED (Ingela Andin) Date: Mon, 21 Sep 2015 09:51:38 +0200 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: Hello, 2015-09-20 15:24 GMT+02:00 Sereysethy TOUCH : > Hello, > > I did as what you told me by getting a shell console on the node. > > I run memory(). and see ets ate the most memory. And it increased > overtime. I run ets:i(). and I found out the ssl_manager took a big chunk > of what memory used. > > 32794 client_ssl_otp_session_cache ordered_set 0 89 > ssl_manager > 36891 server_ssl_otp_session_cache ordered_set 1564 67995 > ssl_manager > 40993 httpc_manager__session_cookie_db bag 0 299 > httpc_manager > > So I think there is a problem with ssl_manager in Erlang OTP 18. > > Any workarounds? > > Ok, the bug I fixed with the session table is on the client side. But your big ets table is on the server side. Session data is by default fairly long lived (24 h that is the max recommended time to keep a session by the RFC), you can set the ssl application variable session_lifetime to make its lifetime shorter which should mitigate your problem. We have a backlog item to make a configurable threshold of max sessions allowed to be stored in the table, which I am sure will be implemented fairly soon but alas not for 18.1. Regards Ingela Erlang/OTP Team - Ericsson AB > Sethy > > On Sun, Sep 20, 2015 at 2:22 AM, Jesper Louis Andersen < > jesper.louis.andersen@REDACTED> wrote: > >> >> On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy TOUCH < >> touch.sereysethy@REDACTED> wrote: >> >>> I just recently updated Erlang to latest version OTP 18 on Ubuntu >>> server. It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used >>> to work fine in OTP 17. The program is correctly compiled but during the >>> execution the memory kept increasing. I need to restart the process every >>> one or two hours to free some memory. >> >> >> 1. Get a shell console on the node >> 2. call 'memory().' in the shell >> 3. If it reports ets as the winner, call >> 4. ets:i(). >> >> This will give you a nice overview for where to look and verify it is the >> SSL problem. >> >> >> -- >> J. >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From touch.sereysethy@REDACTED Mon Sep 21 10:09:40 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Mon, 21 Sep 2015 15:09:40 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: Hello, One question, why did my application work fine on Erlang 17.5.x branch? If I want to set this session_lifetime option, where should put it? So there is no use that I should build the erlang 18 from maint branch? Thanks, Sethy On Mon, Sep 21, 2015 at 2:51 PM, Ingela Andin wrote: > Hello, > > 2015-09-20 15:24 GMT+02:00 Sereysethy TOUCH : > >> Hello, >> >> I did as what you told me by getting a shell console on the node. >> >> I run memory(). and see ets ate the most memory. And it increased >> overtime. I run ets:i(). and I found out the ssl_manager took a big chunk >> of what memory used. >> >> 32794 client_ssl_otp_session_cache ordered_set 0 89 >> ssl_manager >> 36891 server_ssl_otp_session_cache ordered_set 1564 67995 >> ssl_manager >> 40993 httpc_manager__session_cookie_db bag 0 299 >> httpc_manager >> >> So I think there is a problem with ssl_manager in Erlang OTP 18. >> >> Any workarounds? >> >> > > Ok, the bug I fixed with the session table is on the client side. But > your big ets table is on the server side. > Session data is by default fairly long lived (24 h that is the max > recommended time to keep a session by the RFC), > you can set the ssl application variable session_lifetime to make its > lifetime shorter which should mitigate your problem. > We have a backlog item to make a configurable threshold of max sessions > allowed to be stored in the table, which I am sure will be > implemented fairly soon but alas not for 18.1. > > Regards Ingela Erlang/OTP Team - Ericsson AB > > > >> Sethy >> >> On Sun, Sep 20, 2015 at 2:22 AM, Jesper Louis Andersen < >> jesper.louis.andersen@REDACTED> wrote: >> >>> >>> On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy TOUCH < >>> touch.sereysethy@REDACTED> wrote: >>> >>>> I just recently updated Erlang to latest version OTP 18 on Ubuntu >>>> server. It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used >>>> to work fine in OTP 17. The program is correctly compiled but during the >>>> execution the memory kept increasing. I need to restart the process every >>>> one or two hours to free some memory. >>> >>> >>> 1. Get a shell console on the node >>> 2. call 'memory().' in the shell >>> 3. If it reports ets as the winner, call >>> 4. ets:i(). >>> >>> This will give you a nice overview for where to look and verify it is >>> the SSL problem. >>> >>> >>> -- >>> J. >>> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From touch.sereysethy@REDACTED Mon Sep 21 10:11:44 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Mon, 21 Sep 2015 15:11:44 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: <55FFADA2.4050901@ninenines.eu> References: <55FEDC43.80803@ninenines.eu> <55FFADA2.4050901@ninenines.eu> Message-ID: Hello, It works. But Ingela suggested that I should set session_lifetime option to ssl which by default limit to 24h. I wonder where should I put this option? % erlang.mk make file for application {application, ws_user, [ {description, "Cowboy websocket for users."}, {vsn, "1"}, {modules, []}, {registered, [ws_user_sup]}, {applications, [ kernel, stdlib, cowboy, ssl, erlydtl, xmerl ]}, {mod, {ws_user_app, []}}, {env, []} ]}. Thanks, Sethy On Mon, Sep 21, 2015 at 2:11 PM, Lo?c Hoguin wrote: > You need the dev library of OpenSSL when you compile Erlang, otherwise > crypto, public_key and ssl won't be available. > > On 09/21/2015 09:04 AM, Sereysethy TOUCH wrote: > >> Hello, >> >> I use erlang.mk to build my application. After I >> successfully built erlang from maint branch using kerl, when I compiled >> my application, I got this error: >> >> My erlang is Erlang/OTP 18 [erts-7.0.3] [source-5991161] [64-bit] >> [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] >> >> Failed to solve release: >> >> Dependency ssl is specified as a dependency but is not reachable by >> the system. >> >> >> Any ideas? >> >> >> On Sun, Sep 20, 2015 at 11:18 PM, Lo?c Hoguin > > wrote: >> >> As far as I understand, 18.1 will be released very soon, maybe this >> week, so perhaps your problems will be gone in a few days without >> having to do anything. :-) >> >> On 09/20/2015 06:04 PM, Sereysethy TOUCH wrote: >> >> FYI, I use erlang.mk to >> build my application. >> >> On Sun, Sep 20, 2015 at 10:53 PM, Sereysethy TOUCH >> >> > >> wrote: >> >> Hello, >> >> When do you think a new version erlang 18 will be released >> which >> fixes this problem? >> >> So you suggest that I should build the maint branch? And >> what to do >> with erlang installed on my server? >> >> I will stick with shell. No gui. >> >> Sorry I seem confused. Need more helps here :-) >> >> >> >> On Sunday, 20 September 2015, Jesper Louis Andersen >> > >> > >> wrote: >> >> Ingela suggested you try building the "maint" branch, >> which has >> a fix for the session cache problem (by Ingela :) >> >> simply git pull https://github.com/erlang/otp >> >> checkout the maint branch >> >> build the result. Alterantively, the 'kerl' application >> (also on >> github) can build Erlang. It has an experimental >> git-mode which >> may be useful here. >> >> I think you are looking at the SSL session cache problem. >> >> As for observer: observer:start() will start it: >> >> * You need wx in your Erlang installation, it is a GUI >> application >> * You need something which can run wx applications. It is >> somewhat easy in a local dev environment, but the shell >> commands >> can be useful in contexts with no easy GUI-line :) >> >> >> >> On Sun, Sep 20, 2015 at 3:30 PM, Sereysethy TOUCH >> > > wrote: >> >> Dear Andin, >> >> To be honest, I am new to Erlang. So I am sorry >> sometimes >> for my naive questions. >> >> I just tried to fix and debug erlang code written >> by another >> person. I am not sure yet where to look at. Since >> we dont >> use anything special about SSL, beside we ask user >> browser >> to present certificate and pass it on to our agent >> backend >> to process it. I look at the code, it seems nothing >> is unusual. >> >> You said this bug exists since Erlang 17? But why >> there were >> no memory leak when it ran on Erlang 17. >> >> I did as Jesper Louis Andersen suggested, and I saw >> there a >> big amount of memory being used by ssl_manager. >> >> 32794 client_ssl_otp_session_cache >> ordered_set 0 >> 89 ssl_manager >> 36891 server_ssl_otp_session_cache >> ordered_set >> 1564 67995 ssl_manager >> 40993 httpc_manager__session_cookie_db >> bag 0 >> 299 httpc_manager >> >> And it keeps increasing overtime. >> >> Can you let me know or pointing me to any documents >> on how >> to use observer application? >> >> Thanks, Sethy >> >> On Sun, Sep 20, 2015 at 5:07 AM, Ingela Andin >> > > wrote: >> >> Hi! >> >> 2015-09-19 18:34 GMT+02:00 Sereysethy TOUCH >> > >: >> >> >> Hello, >> >> I just recently updated Erlang to latest >> version OTP >> 18 on Ubuntu server. It uses cowboy >> (websocket), >> ranch, ssl, erlydtl & rabbitmq. It used to >> work fine >> in OTP 17. The program is correctly >> compiled but >> during the execution the memory kept >> increasing. I >> need to restart the process every one or >> two hours >> to free some memory. >> >> I have read a post here >> >> [ >> http://erlang.2086793.n4.nabble.com/R18-Unbounded-SSL-Session-ETS-Table-Growth-td4713697.html >> ] >> which discussed about the ssl_session_cache >> ETS >> table which can become very large. >> >> >> Well that bug is now fixed for 18.1 and in maint >> on >> github. This bug however has been around for a >> while so >> it is unlikly that this bug causes 18 to behave >> different from 17. >> >> The process beam.smp can go up to more than >> 5G >> during a few hours of executions. >> >> I am not yet sure what is the root cause of >> this issue. >> >> Does anyone know how to fix this? Or where >> should I >> look at? >> >> >> >> You could try using the observer application to >> inspect >> the node, makes inspecting easy. >> Let us know your findings. >> >> Regards Ingela Erlang/OTP team - Ericsson AB >> >> >> Thanks, Sethy >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> >> -- >> J. >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> -- >> Lo?c Hoguin >> http://ninenines.eu >> Author of The Erlanger Playbook, >> A book about software development using Erlang >> >> >> > -- > Lo?c Hoguin > http://ninenines.eu > Author of The Erlanger Playbook, > A book about software development using Erlang > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Mon Sep 21 10:52:05 2015 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 21 Sep 2015 10:52:05 +0200 Subject: [erlang-questions] Erlang tracing Message-ID: Hello everyone. As you may know, one of the OTP teams focus areas for the coming year is make tracing better. At the moment we are gathering ideas and attempting to put together a vision of what we would like to have, before deciding what we can make. I'm pretty sure that many of you have much more experience with using Erlang tracing while developing and in production than I do, which is we would love to have your input as to what you would like to change about tracing. To set the scope of the discussion, when I say tracing I include; erlang tracing, dtrace/systemtap, trace outputs (stdout/file/IP), filtering through match specs, sequence tracing, tool integration (dbg, fprof, redbug, recon to mention some) and probably more. To start the discussion, here are a few of my ideas in no particular order: * Allow multiple tracers. Today only one port/process can be the receiver of trace data. * Create a couple of scalable high throughput tracing output backends with different overflow mechanics. Today all tracing is funneled through one bottleneck and has no overflow handling at all. * Expose vm probes (today dtrace probes) to the erlang tracer. * Better integration of dtrace/lttngt/systemtap into the erlang trace. * Allow the erlang tracer to be an Erlang callback module. Today only ports/processes are allowed. * Optimize trace output to file/ip. Maybe use something like the Common Trace Format ( http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.md;hb=master), instead of the term_to_binary that we have today. * Write much much better documentation for dbg :) We are looking for feedback from both beginners as well as seasoned veterans to make erlang tracing the best it can be. So if you have any thoughts or ideas, join the discussion to make Erlang tracing better for you and everyone else. Thanks in advance, Lukas Erlang/OTP team -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Mon Sep 21 11:54:13 2015 From: ingela.andin@REDACTED (Ingela Andin) Date: Mon, 21 Sep 2015 11:54:13 +0200 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: Hi! 2015-09-21 10:09 GMT+02:00 Sereysethy TOUCH : > Hello, > > One question, why did my application work fine on Erlang 17.5.x branch? > > Well I am not sure what your application does. If you have a client application that spawns a lot of parallel connections to the same host, not waiting for one to succeed before spawning, the client session table could grow a lot. However I have implemented a feature to only save unique TLS-sessions in this senario to avoid this, and there was a bug so that this feature did not work, that I have fix in the maint branch. This bug however was in 17 as well as 18. However the ets info you sent suggest you have a server application and then setting session_liftime can help keeping the servers session table smaller. This also is true for clients, that can get a big table also through connecting to many diffrent servers. Maybe your application can be both a TLS-client and server? There is no obvious reason from what you told us so far that it should work better in 17 then 18. If I want to set this session_lifetime option, where should put it? > > It is an application environment setting. Search for app in the docs. So there is no use that I should build the erlang 18 from maint branch? > > It is always a good thing to try the latest version, and if you still have a problem it would be good if you could try providing a way to reproduce the problem. Regards Ingela Erlang/OTP tema Ericsson AB > Thanks, > Sethy > > On Mon, Sep 21, 2015 at 2:51 PM, Ingela Andin > wrote: > >> Hello, >> >> 2015-09-20 15:24 GMT+02:00 Sereysethy TOUCH : >> >>> Hello, >>> >>> I did as what you told me by getting a shell console on the node. >>> >>> I run memory(). and see ets ate the most memory. And it increased >>> overtime. I run ets:i(). and I found out the ssl_manager took a big chunk >>> of what memory used. >>> >>> 32794 client_ssl_otp_session_cache ordered_set 0 89 >>> ssl_manager >>> 36891 server_ssl_otp_session_cache ordered_set 1564 67995 >>> ssl_manager >>> 40993 httpc_manager__session_cookie_db bag 0 299 >>> httpc_manager >>> >>> So I think there is a problem with ssl_manager in Erlang OTP 18. >>> >>> Any workarounds? >>> >>> >> >> Ok, the bug I fixed with the session table is on the client side. But >> your big ets table is on the server side. >> Session data is by default fairly long lived (24 h that is the max >> recommended time to keep a session by the RFC), >> you can set the ssl application variable session_lifetime to make its >> lifetime shorter which should mitigate your problem. >> We have a backlog item to make a configurable threshold of max sessions >> allowed to be stored in the table, which I am sure will be >> implemented fairly soon but alas not for 18.1. >> >> Regards Ingela Erlang/OTP Team - Ericsson AB >> >> >> >>> Sethy >>> >>> On Sun, Sep 20, 2015 at 2:22 AM, Jesper Louis Andersen < >>> jesper.louis.andersen@REDACTED> wrote: >>> >>>> >>>> On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy TOUCH < >>>> touch.sereysethy@REDACTED> wrote: >>>> >>>>> I just recently updated Erlang to latest version OTP 18 on Ubuntu >>>>> server. It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used >>>>> to work fine in OTP 17. The program is correctly compiled but during the >>>>> execution the memory kept increasing. I need to restart the process every >>>>> one or two hours to free some memory. >>>> >>>> >>>> 1. Get a shell console on the node >>>> 2. call 'memory().' in the shell >>>> 3. If it reports ets as the winner, call >>>> 4. ets:i(). >>>> >>>> This will give you a nice overview for where to look and verify it is >>>> the SSL problem. >>>> >>>> >>>> -- >>>> J. >>>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Mon Sep 21 12:02:31 2015 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 21 Sep 2015 12:02:31 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: <8DF48FD1-5D3E-445E-AE04-D41A5E3886BE@feuerlabs.com> > On 21 Sep 2015, at 10:52, Lukas Larsson wrote: > > To start the discussion, here are a few of my ideas in no particular order: > * Allow multiple tracers. Today only one port/process can be the receiver of trace data. > * Create a couple of scalable high throughput tracing output backends with different overflow mechanics. Today all tracing is funneled through one bottleneck and has no overflow handling at all. > * Expose vm probes (today dtrace probes) to the erlang tracer. > * Better integration of dtrace/lttngt/systemtap into the erlang trace. > * Allow the erlang tracer to be an Erlang callback module. Today only ports/processes are allowed. > * Optimize trace output to file/ip. Maybe use something like the Common Trace Format (http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.md;hb=master ), instead of the term_to_binary that we have today. > * Write much much better documentation for dbg :) +1 on the suggestions, esp. multiple tracers. The one-tracer restriction makes it hard to explore some very interesting monitoring and debugging ideas. I have had some success using an event/[2,3] function for lightweight debugging, where the functions event(Line, Event) and event(Line Event, State) exist only to be traced. A simple way to generalize this would be to create ?null? BIFs: erlang:trace_event(Info, Event) -> ok, erlang:trace_event(Info, Event, State) -> ok (just a trace_event/1 would of course suffice). If not traced, these functions do nothing; when traced, they emit trace info similar to io:format debugging. See e.g. the locks application: https://github.com/uwiger/locks/blob/master/src/locks_leader.erl#L140 https://github.com/uwiger/locks/blob/master/src/locks_leader.erl#L823 The ?debug/[1,2] macro is used to also include line numbers. One could perhaps imagine such a macro existing in dbg, but putting an event() function in dbg as well forces a remote call even when tracing is not enabled. So basically, the features are: - a ?debug? function that is as lightweight as possible when not traced, just existing for the purpose of being traced - including line numbers I?ve found this extremely useful not least when enabled during multi-node test suite execution, using ttb to merge logs and a final pretty-printing step to produce a text log best viewed with emacs's Erlang syntax highlighting. See https://github.com/uwiger/locks/blob/master/src/locks_ttb.erl (Logs fetched only if test case fails). One might fantasize about enabling migration from io:format() debugging to this, ideally by simply rewriting the debug macro: a handy wrapper that simulates the original printouts. There are some problems with this, so I?ve chosen not to go there. To further generalize the issue ? What we ended up with at Ericsson was a variety of techniques for getting the most out of tracing: - A script that inserted a ?DBG macro after each ->. This was a conditional debug printout, but could have been a trace event as above. The feature was to log the ?branching? in the code, e.g. noting which case function or receive clause was actually selected. - Consistently breaking out case clauses etc as separate functions (since function calls can be traced). See e.g. the diameter application in OTP for examples of this style of coding. I think there is room for revisiting the ?et? concept in this context. The thing about ?et? was that it allowed for a ?model-level? trace, complete with sequence diagram presentation. Its main drawback was that few figured out how to use it. But I think the idea is brilliant: to be able to dynamically extract ?model markers? from executing code and displaying it in a way that can tie back to the conceptual design. http://jlouisramblings.blogspot.se/2011/10/using-event-tracer-tool-set-in-erlang.html http://souja.net/2009/04/making-sense-of-erlangs-event-tracer http://sysmagazine.com/posts/131140/ Lastly, TTB has been greatly improved (ahem?) and is extremely useful for multi-node tracing. When improving the DBG documentation, make sure to show how TTB and DBG work together and what their respective roles are. BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From touch.sereysethy@REDACTED Mon Sep 21 12:05:52 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Mon, 21 Sep 2015 17:05:52 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: Hello, My application is both TLS-client & server. It is a web server which allows client browser to connect to server using websocket. I use cowboy as web server & web socket server. In erlang 17, our application runs for weeks and there is no memory issue. We only restart it whenever we have a new update. But since we update to elang 18, we need to restart our application every 2 hours. I use erlang.mk to compile our application, I am not sure how to set application env. settings. Is it the same if I use erlang.mk? Thanks, Sethy On Mon, Sep 21, 2015 at 4:54 PM, Ingela Andin wrote: > Hi! > > > 2015-09-21 10:09 GMT+02:00 Sereysethy TOUCH : > >> Hello, >> >> One question, why did my application work fine on Erlang 17.5.x branch? >> >> > Well I am not sure what your application does. > > If you have a client application that spawns a lot of parallel connections > to the same host, not waiting for one to succeed before spawning, > the client session table could grow a lot. However I have implemented a > feature to only save unique TLS-sessions in this senario to avoid this, and > there was a bug so that this feature did not work, that I have fix in the > maint branch. This bug however was in 17 as well as 18. > > However the ets info you sent suggest you have a server application and > then setting session_liftime can help keeping the servers session table > smaller. This also is true for clients, that can get a big table also > through connecting to many diffrent servers. > > Maybe your application can be both a TLS-client and server? There is no > obvious reason from what you told us so far that it should work better in > 17 then 18. > > > If I want to set this session_lifetime option, where should put it? >> >> > It is an application environment setting. Search for app in the docs. > > > So there is no use that I should build the erlang 18 from maint branch? >> >> > > It is always a good thing to try the latest version, and if you still have > a problem it would be good if you could try providing a way to reproduce > the problem. > > Regards Ingela Erlang/OTP tema Ericsson AB > > > > >> Thanks, >> Sethy >> >> On Mon, Sep 21, 2015 at 2:51 PM, Ingela Andin >> wrote: >> >>> Hello, >>> >>> 2015-09-20 15:24 GMT+02:00 Sereysethy TOUCH >>> : >>> >>>> Hello, >>>> >>>> I did as what you told me by getting a shell console on the node. >>>> >>>> I run memory(). and see ets ate the most memory. And it increased >>>> overtime. I run ets:i(). and I found out the ssl_manager took a big chunk >>>> of what memory used. >>>> >>>> 32794 client_ssl_otp_session_cache ordered_set 0 89 >>>> ssl_manager >>>> 36891 server_ssl_otp_session_cache ordered_set 1564 67995 >>>> ssl_manager >>>> 40993 httpc_manager__session_cookie_db bag 0 299 >>>> httpc_manager >>>> >>>> So I think there is a problem with ssl_manager in Erlang OTP 18. >>>> >>>> Any workarounds? >>>> >>>> >>> >>> Ok, the bug I fixed with the session table is on the client side. But >>> your big ets table is on the server side. >>> Session data is by default fairly long lived (24 h that is the max >>> recommended time to keep a session by the RFC), >>> you can set the ssl application variable session_lifetime to make its >>> lifetime shorter which should mitigate your problem. >>> We have a backlog item to make a configurable threshold of max sessions >>> allowed to be stored in the table, which I am sure will be >>> implemented fairly soon but alas not for 18.1. >>> >>> Regards Ingela Erlang/OTP Team - Ericsson AB >>> >>> >>> >>>> Sethy >>>> >>>> On Sun, Sep 20, 2015 at 2:22 AM, Jesper Louis Andersen < >>>> jesper.louis.andersen@REDACTED> wrote: >>>> >>>>> >>>>> On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy TOUCH < >>>>> touch.sereysethy@REDACTED> wrote: >>>>> >>>>>> I just recently updated Erlang to latest version OTP 18 on Ubuntu >>>>>> server. It uses cowboy (websocket), ranch, ssl, erlydtl & rabbitmq. It used >>>>>> to work fine in OTP 17. The program is correctly compiled but during the >>>>>> execution the memory kept increasing. I need to restart the process every >>>>>> one or two hours to free some memory. >>>>> >>>>> >>>>> 1. Get a shell console on the node >>>>> 2. call 'memory().' in the shell >>>>> 3. If it reports ets as the winner, call >>>>> 4. ets:i(). >>>>> >>>>> This will give you a nice overview for where to look and verify it is >>>>> the SSL problem. >>>>> >>>>> >>>>> -- >>>>> J. >>>>> >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gguthrie@REDACTED Mon Sep 21 12:09:09 2015 From: gguthrie@REDACTED (Gordon Guthrie) Date: Mon, 21 Sep 2015 11:09:09 +0100 Subject: [erlang-questions] Erlang tracing In-Reply-To: <8DF48FD1-5D3E-445E-AE04-D41A5E3886BE@feuerlabs.com> References: <8DF48FD1-5D3E-445E-AE04-D41A5E3886BE@feuerlabs.com> Message-ID: There is a good graphical tool for tracing called Erlyberly that you should look at: https://github.com/andytill/erlyberly It would be good to get some Elixir users on it too to work out how to do tracing into the Elixir function name space properly? Gordon > Le 21 sept. 2015 ? 11:02, Ulf Wiger a ?crit : > > >> On 21 Sep 2015, at 10:52, Lukas Larsson > wrote: >> >> To start the discussion, here are a few of my ideas in no particular order: >> * Allow multiple tracers. Today only one port/process can be the receiver of trace data. >> * Create a couple of scalable high throughput tracing output backends with different overflow mechanics. Today all tracing is funneled through one bottleneck and has no overflow handling at all. >> * Expose vm probes (today dtrace probes) to the erlang tracer. >> * Better integration of dtrace/lttngt/systemtap into the erlang trace. >> * Allow the erlang tracer to be an Erlang callback module. Today only ports/processes are allowed. >> * Optimize trace output to file/ip. Maybe use something like the Common Trace Format (http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.md;hb=master ), instead of the term_to_binary that we have today. >> * Write much much better documentation for dbg :) > > +1 on the suggestions, esp. multiple tracers. The one-tracer restriction makes it hard to explore some very interesting monitoring and debugging ideas. > > I have had some success using an event/[2,3] function for lightweight debugging, where the functions event(Line, Event) and event(Line Event, State) exist only to be traced. A simple way to generalize this would be to create ?null? BIFs: erlang:trace_event(Info, Event) -> ok, erlang:trace_event(Info, Event, State) -> ok (just a trace_event/1 would of course suffice). If not traced, these functions do nothing; when traced, they emit trace info similar to io:format debugging. > > See e.g. the locks application: > https://github.com/uwiger/locks/blob/master/src/locks_leader.erl#L140 > https://github.com/uwiger/locks/blob/master/src/locks_leader.erl#L823 > > The ?debug/[1,2] macro is used to also include line numbers. One could perhaps imagine such a macro existing in dbg, but putting an event() function in dbg as well forces a remote call even when tracing is not enabled. > > So basically, the features are: > - a ?debug? function that is as lightweight as possible when not traced, just existing for the purpose of being traced > - including line numbers > > I?ve found this extremely useful not least when enabled during multi-node test suite execution, using ttb to merge logs and a final pretty-printing step to produce a text log best viewed with emacs's Erlang syntax highlighting. See https://github.com/uwiger/locks/blob/master/src/locks_ttb.erl (Logs fetched only if test case fails). > > One might fantasize about enabling migration from io:format() debugging to this, ideally by simply rewriting the debug macro: a handy wrapper that simulates the original printouts. There are some problems with this, so I?ve chosen not to go there. > > To further generalize the issue ? > > What we ended up with at Ericsson was a variety of techniques for getting the most out of tracing: > > - A script that inserted a ?DBG macro after each ->. This was a conditional debug printout, but could have been a trace event as above. The feature was to log the ?branching? in the code, e.g. noting which case function or receive clause was actually selected. > - Consistently breaking out case clauses etc as separate functions (since function calls can be traced). See e.g. the diameter application in OTP for examples of this style of coding. > > I think there is room for revisiting the ?et? concept in this context. The thing about ?et? was that it allowed for a ?model-level? trace, complete with sequence diagram presentation. Its main drawback was that few figured out how to use it. But I think the idea is brilliant: to be able to dynamically extract ?model markers? from executing code and displaying it in a way that can tie back to the conceptual design. > > http://jlouisramblings.blogspot.se/2011/10/using-event-tracer-tool-set-in-erlang.html > http://souja.net/2009/04/making-sense-of-erlangs-event-tracer > http://sysmagazine.com/posts/131140/ > > Lastly, TTB has been greatly improved (ahem?) and is extremely useful for multi-node tracing. When improving the DBG documentation, make sure to show how TTB and DBG work together and what their respective roles are. > > BR, > Ulf W > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Sep 21 12:09:51 2015 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Mon, 21 Sep 2015 12:09:51 +0200 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: Message-ID: <55FFD76F.8080903@ninenines.eu> On 09/21/2015 12:05 PM, Sereysethy TOUCH wrote: > Hello, > > My application is both TLS-client & server. It is a web server which > allows client browser to connect to server using websocket. I use cowboy > as web server & web socket server. > > In erlang 17, our application runs for weeks and there is no memory > issue. We only restart it whenever we have a new update. But since we > update to elang 18, we need to restart our application every 2 hours. > > I use erlang.mk to compile our application, I am not > sure how to set application env. settings. Is it the same if I use > erlang.mk ? If you run a release, you can put the configuration in the sys.config file (by default rel/sys.config) and run make again. See http://www.erlang.org/doc/man/config.html > Thanks, > Sethy > > On Mon, Sep 21, 2015 at 4:54 PM, Ingela Andin > wrote: > > Hi! > > > 2015-09-21 10:09 GMT+02:00 Sereysethy TOUCH > >: > > Hello, > > One question, why did my application work fine on Erlang 17.5.x > branch? > > > Well I am not sure what your application does. > > If you have a client application that spawns a lot of parallel > connections to the same host, not waiting for one to succeed before > spawning, > the client session table could grow a lot. However I have > implemented a feature to only save unique TLS-sessions in this > senario to avoid this, and there was a bug so that this feature did > not work, that I have fix in the maint branch. This bug however was > in 17 as well as 18. > > However the ets info you sent suggest you have a server application > and then setting session_liftime can help keeping the servers > session table > smaller. This also is true for clients, that can get a big table > also through connecting to many diffrent servers. > > Maybe your application can be both a TLS-client and server? There is > no obvious reason from what you told us so far that it should work > better in 17 then 18. > > > If I want to set this session_lifetime option, where should put it? > > > It is an application environment setting. Search for app in the docs. > > > So there is no use that I should build the erlang 18 from maint > branch? > > > > It is always a good thing to try the latest version, and if you > still have a problem it would be good if you could try providing a > way to reproduce the problem. > > Regards Ingela Erlang/OTP tema Ericsson AB > > > Thanks, > Sethy > > On Mon, Sep 21, 2015 at 2:51 PM, Ingela Andin > > wrote: > > Hello, > > 2015-09-20 15:24 GMT+02:00 Sereysethy TOUCH > >: > > Hello, > > I did as what you told me by getting a shell console on > the node. > > I run memory(). and see ets ate the most memory. And it > increased overtime. I run ets:i(). and I found out the > ssl_manager took a big chunk of what memory used. > > 32794 client_ssl_otp_session_cache ordered_set > 0 89 ssl_manager > 36891 server_ssl_otp_session_cache ordered_set > 1564 67995 ssl_manager > 40993 httpc_manager__session_cookie_db bag 0 > 299 httpc_manager > > So I think there is a problem with ssl_manager in Erlang > OTP 18. > > Any workarounds? > > > > Ok, the bug I fixed with the session table is on the client > side. But your big ets table is on the server side. > Session data is by default fairly long lived (24 h that is > the max recommended time to keep a session by the RFC), > you can set the ssl application variable session_lifetime to > make its lifetime shorter which should mitigate your problem. > We have a backlog item to make a configurable threshold of > max sessions allowed to be stored in the table, which I am > sure will be > implemented fairly soon but alas not for 18.1. > > Regards Ingela Erlang/OTP Team - Ericsson AB > > Sethy > > On Sun, Sep 20, 2015 at 2:22 AM, Jesper Louis Andersen > > wrote: > > > On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy TOUCH > > wrote: > > I just recently updated Erlang to latest version > OTP 18 on Ubuntu server. It uses cowboy > (websocket), ranch, ssl, erlydtl & rabbitmq. It > used to work fine in OTP 17. The program is > correctly compiled but during the execution the > memory kept increasing. I need to restart the > process every one or two hours to free some memory. > > > 1. Get a shell console on the node > 2. call 'memory().' in the shell > 3. If it reports ets as the winner, call > 4. ets:i(). > > This will give you a nice overview for where to look > and verify it is the SSL problem. > > > -- > J. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu Author of The Erlanger Playbook, A book about software development using Erlang From touch.sereysethy@REDACTED Mon Sep 21 12:44:09 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Mon, 21 Sep 2015 17:44:09 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: <55FFD76F.8080903@ninenines.eu> References: <55FFD76F.8080903@ninenines.eu> Message-ID: Hello, You meant _rel or I need to create a folder "rel" and put sys.config inside? Sethy On Mon, Sep 21, 2015 at 5:09 PM, Lo?c Hoguin wrote: > On 09/21/2015 12:05 PM, Sereysethy TOUCH wrote: > >> Hello, >> >> My application is both TLS-client & server. It is a web server which >> allows client browser to connect to server using websocket. I use cowboy >> as web server & web socket server. >> >> In erlang 17, our application runs for weeks and there is no memory >> issue. We only restart it whenever we have a new update. But since we >> update to elang 18, we need to restart our application every 2 hours. >> >> I use erlang.mk to compile our application, I am not >> sure how to set application env. settings. Is it the same if I use >> erlang.mk ? >> > > If you run a release, you can put the configuration in the sys.config file > (by default rel/sys.config) and run make again. See > http://www.erlang.org/doc/man/config.html > > Thanks, >> Sethy >> >> On Mon, Sep 21, 2015 at 4:54 PM, Ingela Andin > > wrote: >> >> Hi! >> >> >> 2015-09-21 10:09 GMT+02:00 Sereysethy TOUCH >> >: >> >> >> Hello, >> >> One question, why did my application work fine on Erlang 17.5.x >> branch? >> >> >> Well I am not sure what your application does. >> >> If you have a client application that spawns a lot of parallel >> connections to the same host, not waiting for one to succeed before >> spawning, >> the client session table could grow a lot. However I have >> implemented a feature to only save unique TLS-sessions in this >> senario to avoid this, and there was a bug so that this feature did >> not work, that I have fix in the maint branch. This bug however was >> in 17 as well as 18. >> >> However the ets info you sent suggest you have a server application >> and then setting session_liftime can help keeping the servers >> session table >> smaller. This also is true for clients, that can get a big table >> also through connecting to many diffrent servers. >> >> Maybe your application can be both a TLS-client and server? There is >> no obvious reason from what you told us so far that it should work >> better in 17 then 18. >> >> >> If I want to set this session_lifetime option, where should put >> it? >> >> >> It is an application environment setting. Search for app in the docs. >> >> >> So there is no use that I should build the erlang 18 from maint >> branch? >> >> >> >> It is always a good thing to try the latest version, and if you >> still have a problem it would be good if you could try providing a >> way to reproduce the problem. >> >> Regards Ingela Erlang/OTP tema Ericsson AB >> >> >> Thanks, >> Sethy >> >> On Mon, Sep 21, 2015 at 2:51 PM, Ingela Andin >> > wrote: >> >> Hello, >> >> 2015-09-20 15:24 GMT+02:00 Sereysethy TOUCH >> > >: >> >> >> Hello, >> >> I did as what you told me by getting a shell console on >> the node. >> >> I run memory(). and see ets ate the most memory. And it >> increased overtime. I run ets:i(). and I found out the >> ssl_manager took a big chunk of what memory used. >> >> 32794 client_ssl_otp_session_cache ordered_set >> 0 89 ssl_manager >> 36891 server_ssl_otp_session_cache ordered_set >> 1564 67995 ssl_manager >> 40993 httpc_manager__session_cookie_db bag 0 >> 299 httpc_manager >> >> So I think there is a problem with ssl_manager in Erlang >> OTP 18. >> >> Any workarounds? >> >> >> >> Ok, the bug I fixed with the session table is on the client >> side. But your big ets table is on the server side. >> Session data is by default fairly long lived (24 h that is >> the max recommended time to keep a session by the RFC), >> you can set the ssl application variable session_lifetime to >> make its lifetime shorter which should mitigate your problem. >> We have a backlog item to make a configurable threshold of >> max sessions allowed to be stored in the table, which I am >> sure will be >> implemented fairly soon but alas not for 18.1. >> >> Regards Ingela Erlang/OTP Team - Ericsson AB >> >> Sethy >> >> On Sun, Sep 20, 2015 at 2:22 AM, Jesper Louis Andersen >> > > wrote: >> >> >> On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy TOUCH >> > > wrote: >> >> I just recently updated Erlang to latest version >> OTP 18 on Ubuntu server. It uses cowboy >> (websocket), ranch, ssl, erlydtl & rabbitmq. It >> used to work fine in OTP 17. The program is >> correctly compiled but during the execution the >> memory kept increasing. I need to restart the >> process every one or two hours to free some >> memory. >> >> >> 1. Get a shell console on the node >> 2. call 'memory().' in the shell >> 3. If it reports ets as the winner, call >> 4. ets:i(). >> >> This will give you a nice overview for where to look >> and verify it is the SSL problem. >> >> >> -- >> J. >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- > Lo?c Hoguin > http://ninenines.eu > Author of The Erlanger Playbook, > A book about software development using Erlang > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Sep 21 12:46:04 2015 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Mon, 21 Sep 2015 12:46:04 +0200 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: <55FFD76F.8080903@ninenines.eu> Message-ID: <55FFDFEC.20701@ninenines.eu> Add {sys_config, "path/to/sys.config"} to your relx.config, then create that sys.config file with the right options and run make. On 09/21/2015 12:44 PM, Sereysethy TOUCH wrote: > Hello, > > You meant _rel or I need to create a folder "rel" and put sys.config inside? > > Sethy > > On Mon, Sep 21, 2015 at 5:09 PM, Lo?c Hoguin > wrote: > > On 09/21/2015 12:05 PM, Sereysethy TOUCH wrote: > > Hello, > > My application is both TLS-client & server. It is a web server which > allows client browser to connect to server using websocket. I > use cowboy > as web server & web socket server. > > In erlang 17, our application runs for weeks and there is no memory > issue. We only restart it whenever we have a new update. But > since we > update to elang 18, we need to restart our application every 2 > hours. > > I use erlang.mk to compile > our application, I am not > sure how to set application env. settings. Is it the same if I use > erlang.mk ? > > > If you run a release, you can put the configuration in the > sys.config file (by default rel/sys.config) and run make again. See > http://www.erlang.org/doc/man/config.html > > > > Thanks, > Sethy > > On Mon, Sep 21, 2015 at 4:54 PM, Ingela Andin > > >> > wrote: > > Hi! > > > 2015-09-21 10:09 GMT+02:00 Sereysethy TOUCH > > >>: > > > Hello, > > One question, why did my application work fine on > Erlang 17.5.x > branch? > > > Well I am not sure what your application does. > > If you have a client application that spawns a lot of parallel > connections to the same host, not waiting for one to > succeed before > spawning, > the client session table could grow a lot. However I have > implemented a feature to only save unique TLS-sessions in this > senario to avoid this, and there was a bug so that this > feature did > not work, that I have fix in the maint branch. This bug > however was > in 17 as well as 18. > > However the ets info you sent suggest you have a server > application > and then setting session_liftime can help keeping the servers > session table > smaller. This also is true for clients, that can get a big > table > also through connecting to many diffrent servers. > > Maybe your application can be both a TLS-client and server? > There is > no obvious reason from what you told us so far that it > should work > better in 17 then 18. > > > If I want to set this session_lifetime option, where > should put it? > > > It is an application environment setting. Search for app in > the docs. > > > So there is no use that I should build the erlang 18 > from maint > branch? > > > > It is always a good thing to try the latest version, and if you > still have a problem it would be good if you could try > providing a > way to reproduce the problem. > > Regards Ingela Erlang/OTP tema Ericsson AB > > > Thanks, > Sethy > > On Mon, Sep 21, 2015 at 2:51 PM, Ingela Andin > > >> > wrote: > > Hello, > > 2015-09-20 15:24 GMT+02:00 Sereysethy TOUCH > > >>: > > > Hello, > > I did as what you told me by getting a shell > console on > the node. > > I run memory(). and see ets ate the most > memory. And it > increased overtime. I run ets:i(). and I found > out the > ssl_manager took a big chunk of what memory used. > > 32794 client_ssl_otp_session_cache > ordered_set > 0 89 ssl_manager > 36891 server_ssl_otp_session_cache > ordered_set > 1564 67995 ssl_manager > 40993 > httpc_manager__session_cookie_db bag 0 > 299 httpc_manager > > So I think there is a problem with ssl_manager > in Erlang > OTP 18. > > Any workarounds? > > > > Ok, the bug I fixed with the session table is on > the client > side. But your big ets table is on the server side. > Session data is by default fairly long lived (24 h > that is > the max recommended time to keep a session by the RFC), > you can set the ssl application variable > session_lifetime to > make its lifetime shorter which should mitigate > your problem. > We have a backlog item to make a configurable > threshold of > max sessions allowed to be stored in the table, > which I am > sure will be > implemented fairly soon but alas not for 18.1. > > Regards Ingela Erlang/OTP Team - Ericsson AB > > Sethy > > On Sun, Sep 20, 2015 at 2:22 AM, Jesper Louis > Andersen > > >> wrote: > > > On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy > TOUCH > > >> wrote: > > I just recently updated Erlang to > latest version > OTP 18 on Ubuntu server. It uses cowboy > (websocket), ranch, ssl, erlydtl & > rabbitmq. It > used to work fine in OTP 17. The program is > correctly compiled but during the > execution the > memory kept increasing. I need to > restart the > process every one or two hours to free > some memory. > > > 1. Get a shell console on the node > 2. call 'memory().' in the shell > 3. If it reports ets as the winner, call > 4. ets:i(). > > This will give you a nice overview for > where to look > and verify it is the SSL problem. > > > -- > J. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -- > Lo?c Hoguin > http://ninenines.eu > Author of The Erlanger Playbook, > A book about software development using Erlang > > -- Lo?c Hoguin http://ninenines.eu Author of The Erlanger Playbook, A book about software development using Erlang From touch.sereysethy@REDACTED Mon Sep 21 12:52:07 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Mon, 21 Sep 2015 17:52:07 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: <55FFDFEC.20701@ninenines.eu> References: <55FFD76F.8080903@ninenines.eu> <55FFDFEC.20701@ninenines.eu> Message-ID: Something like this? {release, {ws_user, "1"}, [ws_user]}. {sys_config, "./config/sys.config"}. %{paths, ["../deps/"]}. % generate startup script, yes {extended_start_script, true}. On Mon, Sep 21, 2015 at 5:46 PM, Lo?c Hoguin wrote: > Add {sys_config, "path/to/sys.config"} to your relx.config, then create > that sys.config file with the right options and run make. > > On 09/21/2015 12:44 PM, Sereysethy TOUCH wrote: > >> Hello, >> >> You meant _rel or I need to create a folder "rel" and put sys.config >> inside? >> >> Sethy >> >> On Mon, Sep 21, 2015 at 5:09 PM, Lo?c Hoguin > > wrote: >> >> On 09/21/2015 12:05 PM, Sereysethy TOUCH wrote: >> >> Hello, >> >> My application is both TLS-client & server. It is a web server >> which >> allows client browser to connect to server using websocket. I >> use cowboy >> as web server & web socket server. >> >> In erlang 17, our application runs for weeks and there is no >> memory >> issue. We only restart it whenever we have a new update. But >> since we >> update to elang 18, we need to restart our application every 2 >> hours. >> >> I use erlang.mk to compile >> our application, I am not >> sure how to set application env. settings. Is it the same if I use >> erlang.mk ? >> >> >> If you run a release, you can put the configuration in the >> sys.config file (by default rel/sys.config) and run make again. See >> http://www.erlang.org/doc/man/config.html >> >> >> >> Thanks, >> Sethy >> >> On Mon, Sep 21, 2015 at 4:54 PM, Ingela Andin >> >> >> >> wrote: >> >> Hi! >> >> >> 2015-09-21 10:09 GMT+02:00 Sereysethy TOUCH >> > >> > >> >>: >> >> >> Hello, >> >> One question, why did my application work fine on >> Erlang 17.5.x >> branch? >> >> >> Well I am not sure what your application does. >> >> If you have a client application that spawns a lot of >> parallel >> connections to the same host, not waiting for one to >> succeed before >> spawning, >> the client session table could grow a lot. However I have >> implemented a feature to only save unique TLS-sessions in >> this >> senario to avoid this, and there was a bug so that this >> feature did >> not work, that I have fix in the maint branch. This bug >> however was >> in 17 as well as 18. >> >> However the ets info you sent suggest you have a server >> application >> and then setting session_liftime can help keeping the servers >> session table >> smaller. This also is true for clients, that can get a big >> table >> also through connecting to many diffrent servers. >> >> Maybe your application can be both a TLS-client and server? >> There is >> no obvious reason from what you told us so far that it >> should work >> better in 17 then 18. >> >> >> If I want to set this session_lifetime option, where >> should put it? >> >> >> It is an application environment setting. Search for app in >> the docs. >> >> >> So there is no use that I should build the erlang 18 >> from maint >> branch? >> >> >> >> It is always a good thing to try the latest version, and if >> you >> still have a problem it would be good if you could try >> providing a >> way to reproduce the problem. >> >> Regards Ingela Erlang/OTP tema Ericsson AB >> >> >> Thanks, >> Sethy >> >> On Mon, Sep 21, 2015 at 2:51 PM, Ingela Andin >> >> >> >> wrote: >> >> Hello, >> >> 2015-09-20 15:24 GMT+02:00 Sereysethy TOUCH >> > >> > >> >>: >> >> >> Hello, >> >> I did as what you told me by getting a shell >> console on >> the node. >> >> I run memory(). and see ets ate the most >> memory. And it >> increased overtime. I run ets:i(). and I found >> out the >> ssl_manager took a big chunk of what memory used. >> >> 32794 client_ssl_otp_session_cache >> ordered_set >> 0 89 ssl_manager >> 36891 server_ssl_otp_session_cache >> ordered_set >> 1564 67995 ssl_manager >> 40993 >> httpc_manager__session_cookie_db bag 0 >> 299 httpc_manager >> >> So I think there is a problem with ssl_manager >> in Erlang >> OTP 18. >> >> Any workarounds? >> >> >> >> Ok, the bug I fixed with the session table is on >> the client >> side. But your big ets table is on the server side. >> Session data is by default fairly long lived (24 h >> that is >> the max recommended time to keep a session by the >> RFC), >> you can set the ssl application variable >> session_lifetime to >> make its lifetime shorter which should mitigate >> your problem. >> We have a backlog item to make a configurable >> threshold of >> max sessions allowed to be stored in the table, >> which I am >> sure will be >> implemented fairly soon but alas not for 18.1. >> >> Regards Ingela Erlang/OTP Team - Ericsson AB >> >> Sethy >> >> On Sun, Sep 20, 2015 at 2:22 AM, Jesper Louis >> Andersen >> > >> > >> wrote: >> >> >> On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy >> TOUCH >> > >> > >> >> wrote: >> >> I just recently updated Erlang to >> latest version >> OTP 18 on Ubuntu server. It uses cowboy >> (websocket), ranch, ssl, erlydtl & >> rabbitmq. It >> used to work fine in OTP 17. The program >> is >> correctly compiled but during the >> execution the >> memory kept increasing. I need to >> restart the >> process every one or two hours to free >> some memory. >> >> >> 1. Get a shell console on the node >> 2. call 'memory().' in the shell >> 3. If it reports ets as the winner, call >> 4. ets:i(). >> >> This will give you a nice overview for >> where to look >> and verify it is the SSL problem. >> >> >> -- >> J. >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> > > >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> -- >> Lo?c Hoguin >> http://ninenines.eu >> Author of The Erlanger Playbook, >> A book about software development using Erlang >> >> >> > -- > Lo?c Hoguin > http://ninenines.eu > Author of The Erlanger Playbook, > A book about software development using Erlang > -------------- next part -------------- An HTML attachment was scrubbed... URL: From touch.sereysethy@REDACTED Mon Sep 21 13:00:49 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Mon, 21 Sep 2015 18:00:49 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: <55FFD76F.8080903@ninenines.eu> <55FFDFEC.20701@ninenines.eu> Message-ID: I got it Lo?c. Thanks. On Mon, Sep 21, 2015 at 5:52 PM, Sereysethy TOUCH < touch.sereysethy@REDACTED> wrote: > Something like this? > > {release, {ws_user, "1"}, > > [ws_user]}. > > {sys_config, "./config/sys.config"}. > > %{paths, ["../deps/"]}. > > % generate startup script, yes > > {extended_start_script, true}. > > On Mon, Sep 21, 2015 at 5:46 PM, Lo?c Hoguin wrote: > >> Add {sys_config, "path/to/sys.config"} to your relx.config, then create >> that sys.config file with the right options and run make. >> >> On 09/21/2015 12:44 PM, Sereysethy TOUCH wrote: >> >>> Hello, >>> >>> You meant _rel or I need to create a folder "rel" and put sys.config >>> inside? >>> >>> Sethy >>> >>> On Mon, Sep 21, 2015 at 5:09 PM, Lo?c Hoguin >> > wrote: >>> >>> On 09/21/2015 12:05 PM, Sereysethy TOUCH wrote: >>> >>> Hello, >>> >>> My application is both TLS-client & server. It is a web server >>> which >>> allows client browser to connect to server using websocket. I >>> use cowboy >>> as web server & web socket server. >>> >>> In erlang 17, our application runs for weeks and there is no >>> memory >>> issue. We only restart it whenever we have a new update. But >>> since we >>> update to elang 18, we need to restart our application every 2 >>> hours. >>> >>> I use erlang.mk to compile >>> our application, I am not >>> sure how to set application env. settings. Is it the same if I >>> use >>> erlang.mk ? >>> >>> >>> If you run a release, you can put the configuration in the >>> sys.config file (by default rel/sys.config) and run make again. See >>> http://www.erlang.org/doc/man/config.html >>> >>> >>> >>> Thanks, >>> Sethy >>> >>> On Mon, Sep 21, 2015 at 4:54 PM, Ingela Andin >>> >>> >> >>> wrote: >>> >>> Hi! >>> >>> >>> 2015-09-21 10:09 GMT+02:00 Sereysethy TOUCH >>> >> >>> >> >>> >>: >>> >>> >>> Hello, >>> >>> One question, why did my application work fine on >>> Erlang 17.5.x >>> branch? >>> >>> >>> Well I am not sure what your application does. >>> >>> If you have a client application that spawns a lot of >>> parallel >>> connections to the same host, not waiting for one to >>> succeed before >>> spawning, >>> the client session table could grow a lot. However I have >>> implemented a feature to only save unique TLS-sessions in >>> this >>> senario to avoid this, and there was a bug so that this >>> feature did >>> not work, that I have fix in the maint branch. This bug >>> however was >>> in 17 as well as 18. >>> >>> However the ets info you sent suggest you have a server >>> application >>> and then setting session_liftime can help keeping the >>> servers >>> session table >>> smaller. This also is true for clients, that can get a big >>> table >>> also through connecting to many diffrent servers. >>> >>> Maybe your application can be both a TLS-client and server? >>> There is >>> no obvious reason from what you told us so far that it >>> should work >>> better in 17 then 18. >>> >>> >>> If I want to set this session_lifetime option, where >>> should put it? >>> >>> >>> It is an application environment setting. Search for app in >>> the docs. >>> >>> >>> So there is no use that I should build the erlang 18 >>> from maint >>> branch? >>> >>> >>> >>> It is always a good thing to try the latest version, and if >>> you >>> still have a problem it would be good if you could try >>> providing a >>> way to reproduce the problem. >>> >>> Regards Ingela Erlang/OTP tema Ericsson AB >>> >>> >>> Thanks, >>> Sethy >>> >>> On Mon, Sep 21, 2015 at 2:51 PM, Ingela Andin >>> >>> >> >>> wrote: >>> >>> Hello, >>> >>> 2015-09-20 15:24 GMT+02:00 Sereysethy TOUCH >>> >> >>> >> >>> >>: >>> >>> >>> Hello, >>> >>> I did as what you told me by getting a shell >>> console on >>> the node. >>> >>> I run memory(). and see ets ate the most >>> memory. And it >>> increased overtime. I run ets:i(). and I found >>> out the >>> ssl_manager took a big chunk of what memory >>> used. >>> >>> 32794 client_ssl_otp_session_cache >>> ordered_set >>> 0 89 ssl_manager >>> 36891 server_ssl_otp_session_cache >>> ordered_set >>> 1564 67995 ssl_manager >>> 40993 >>> httpc_manager__session_cookie_db bag 0 >>> 299 httpc_manager >>> >>> So I think there is a problem with ssl_manager >>> in Erlang >>> OTP 18. >>> >>> Any workarounds? >>> >>> >>> >>> Ok, the bug I fixed with the session table is on >>> the client >>> side. But your big ets table is on the server side. >>> Session data is by default fairly long lived (24 h >>> that is >>> the max recommended time to keep a session by the >>> RFC), >>> you can set the ssl application variable >>> session_lifetime to >>> make its lifetime shorter which should mitigate >>> your problem. >>> We have a backlog item to make a configurable >>> threshold of >>> max sessions allowed to be stored in the table, >>> which I am >>> sure will be >>> implemented fairly soon but alas not for 18.1. >>> >>> Regards Ingela Erlang/OTP Team - Ericsson AB >>> >>> Sethy >>> >>> On Sun, Sep 20, 2015 at 2:22 AM, Jesper Louis >>> Andersen >>> >> >>> >> >> wrote: >>> >>> >>> On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy >>> TOUCH >>> >> >>> >> >>> >> wrote: >>> >>> I just recently updated Erlang to >>> latest version >>> OTP 18 on Ubuntu server. It uses cowboy >>> (websocket), ranch, ssl, erlydtl & >>> rabbitmq. It >>> used to work fine in OTP 17. The >>> program is >>> correctly compiled but during the >>> execution the >>> memory kept increasing. I need to >>> restart the >>> process every one or two hours to free >>> some memory. >>> >>> >>> 1. Get a shell console on the node >>> 2. call 'memory().' in the shell >>> 3. If it reports ets as the winner, call >>> 4. ets:i(). >>> >>> This will give you a nice overview for >>> where to look >>> and verify it is the SSL problem. >>> >>> >>> -- >>> J. >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> >> > >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> -- >>> Lo?c Hoguin >>> http://ninenines.eu >>> Author of The Erlanger Playbook, >>> A book about software development using Erlang >>> >>> >>> >> -- >> Lo?c Hoguin >> http://ninenines.eu >> Author of The Erlanger Playbook, >> A book about software development using Erlang >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From magnus@REDACTED Mon Sep 21 13:25:18 2015 From: magnus@REDACTED (Magnus Henoch) Date: Mon, 21 Sep 2015 12:25:18 +0100 Subject: [erlang-questions] Erlang tracing In-Reply-To: Lukas Larsson's message of "Mon\, 21 Sep 2015 10\:52\:05 +0200 \(2 hours\, 28 minutes\, 17 seconds ago\)" Message-ID: I had a vaguely related idea the other day. When you trace a function that takes an anonymous function as an argument, it's printed like this: 8> lists:all(foo:foo(3), [1, 2]). (<0.33.0>) call lists:all(#Fun,[1,2]) (<0.33.0>) returned from lists:all/2 -> false That tells you that anonymous function number 0 in module foo is passed as an argument. It would be convenient to have the 0 be the line number where the anonymous function is defined instead, so you can quickly figure out exactly which function is being passed, in case your module creates several different ones. Regards, Magnus From mrtndimitrov@REDACTED Mon Sep 21 13:26:35 2015 From: mrtndimitrov@REDACTED (Martin Koroudjiev) Date: Mon, 21 Sep 2015 14:26:35 +0300 Subject: [erlang-questions] -include with define macro Message-ID: <55FFE96B.5000303@gmail.com> Hello, I am working on an app that will often be included into bigger applications. In an .hrl file, my app defines some debug macros. But I would like to allow the enclosing app to provide a better, more specific to its needs, implementation of these macros. So in the .hrl file I do: -ifndef(DEBUG). -define(DEBUG(Format, Args), io:format("DEBUG: ~p ~p: "++Format, [?MODULE, ?LINE | Args])). -endif. In all my source files, I include the .hrl file. But how should I include the enclosing app .hrl file where the DEBUG macro is defined? I tried this: -ifdef(CONTAINING_APP_INCLUDE). -include(?CONTAINING_APP_INCLUDE). -endif. and to compile the source files as: erlc -Werror -Wall -DCONTAINING_APP_INCLUDE="\"/.hrl\"" .erl but I get the following error: badly formed 'include' Thanks in advance for taking the time to look at this. Regards, Martin From lukas@REDACTED Mon Sep 21 13:46:39 2015 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 21 Sep 2015 13:46:39 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: <8DF48FD1-5D3E-445E-AE04-D41A5E3886BE@feuerlabs.com> References: <8DF48FD1-5D3E-445E-AE04-D41A5E3886BE@feuerlabs.com> Message-ID: Hello, On Mon, Sep 21, 2015 at 12:02 PM, Ulf Wiger wrote: > > I have had some success using an event/[2,3] function for lightweight > debugging, where the functions event(Line, Event) and event(Line Event, > State) exist only to be traced. A simple way to generalize this would be to > create ?null? BIFs: erlang:trace_event(Info, Event) -> ok, > erlang:trace_event(Info, Event, State) -> ok (just a trace_event/1 would of > course suffice). If not traced, these functions do nothing; when traced, > they emit trace info similar to io:format debugging. > > See e.g. the locks application: > https://github.com/uwiger/locks/blob/master/src/locks_leader.erl#L140 > https://github.com/uwiger/locks/blob/master/src/locks_leader.erl#L823 > > So it should be possible define your own trace probes (for lack of a better word) in erlang code? That is a very good point. For dtrace we have the dyntrace module that you can put calls to with very little overhead. Maybe something similar can be done for Erlang tracing? I don't think we should step on the toes of logging though, as those (in my opinion) should be two different things. > The ?debug/[1,2] macro is used to also include line numbers. One could > perhaps imagine such a macro existing in dbg, but putting an event() > function in dbg as well forces a remote call even when tracing is not > enabled. > Including the line number could be useful. Maybe adding it in the match spec for the given trace pattern? Much in the same way as {message,{process_dump}} works today? > > I?ve found this extremely useful not least when enabled during multi-node > test suite execution, using ttb to merge logs and a final pretty-printing > step to produce a text log best viewed with emacs's Erlang syntax > highlighting. See > https://github.com/uwiger/locks/blob/master/src/locks_ttb.erl (Logs > fetched only if test case fails). > > I had completely forgot about the ttb. Will have to read up on what you can do with it. > One might fantasize about enabling migration from io:format() debugging to > this, ideally by simply rewriting the debug macro: a handy wrapper that > simulates the original printouts. There are some problems with this, so > I?ve chosen not to go there. > > To further generalize the issue ? > > What we ended up with at Ericsson was a variety of techniques for getting > the most out of tracing: > > - A script that inserted a ?DBG macro after each ->. This was a > conditional debug printout, but could have been a trace event as above. The > feature was to log the ?branching? in the code, e.g. noting which case > function or receive clause was actually selected. > - Consistently breaking out case clauses etc as separate functions (since > function calls can be traced). See e.g. the diameter application in OTP for > examples of this style of coding. > Tracing on all branches would indeed be neat to have. Not sure if it ever will be possible to do in the emulator without instrumenting the code beforehand. > > I think there is room for revisiting the ?et? concept in this context. The > thing about ?et? was that it allowed for a ?model-level? trace, complete > with sequence diagram presentation. Its main drawback was that few figured > out how to use it. But I think the idea is brilliant: to be able to > dynamically extract ?model markers? from executing code and displaying it > in a way that can tie back to the conceptual design. > > > http://jlouisramblings.blogspot.se/2011/10/using-event-tracer-tool-set-in-erlang.html > http://souja.net/2009/04/making-sense-of-erlangs-event-tracer > http://sysmagazine.com/posts/131140/ > yet another tool I had forgotten about :) thanks for reminder! > > Lastly, TTB has been greatly improved (ahem?) and is extremely useful for > multi-node tracing. When improving the DBG documentation, make sure to show > how TTB and DBG work together and what their respective roles are. > We'll keep that in mind. Thanks for your input! -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Sep 21 13:54:53 2015 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Mon, 21 Sep 2015 13:54:53 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: <8DF48FD1-5D3E-445E-AE04-D41A5E3886BE@feuerlabs.com> Message-ID: <55FFF00D.7090305@ninenines.eu> On 09/21/2015 01:46 PM, Lukas Larsson wrote: > Hello, > > On Mon, Sep 21, 2015 at 12:02 PM, Ulf Wiger > wrote: > > > I have had some success using an event/[2,3] function for > lightweight debugging, where the functions event(Line, Event) and > event(Line Event, State) exist only to be traced. A simple way to > generalize this would be to create ?null? BIFs: > erlang:trace_event(Info, Event) -> ok, erlang:trace_event(Info, > Event, State) -> ok (just a trace_event/1 would of course suffice). > If not traced, these functions do nothing; when traced, they emit > trace info similar to io:format debugging. > > See e.g. the locks application: > https://github.com/uwiger/locks/blob/master/src/locks_leader.erl#L140 > https://github.com/uwiger/locks/blob/master/src/locks_leader.erl#L823 > > > So it should be possible define your own trace probes (for lack of a > better word) in erlang code? That is a very good point. For dtrace we > have the dyntrace module that you can put calls to with very little > overhead. Maybe something similar can be done for Erlang tracing? I > don't think we should step on the toes of logging though, as those (in > my opinion) should be two different things. I would quickly add dyntrace calls all over my projects if it didn't require me to have a dtrace/systemtap/.. enabled kernel to use it (and most importantly, confirm that it works). > I?ve found this extremely useful not least when enabled during > multi-node test suite execution, using ttb to merge logs and a final > pretty-printing step to produce a text log best viewed with emacs's > Erlang syntax highlighting. See > https://github.com/uwiger/locks/blob/master/src/locks_ttb.erl (Logs > fetched only if test case fails). > > > I had completely forgot about the ttb. Will have to read up on what you > can do with it. It would be great to at least be able to use the ttb at the same time as dbg. I ran into this in the past, though I can't recall exactly what my issue was. ttb also needs some love with regard to documentation. It's probably worse than dbg, and I would bet that much fewer people know about it. -- Lo?c Hoguin http://ninenines.eu Author of The Erlanger Playbook, A book about software development using Erlang From zxq9@REDACTED Mon Sep 21 13:56:16 2015 From: zxq9@REDACTED (zxq9) Date: Mon, 21 Sep 2015 20:56:16 +0900 Subject: [erlang-questions] -include with define macro In-Reply-To: <55FFE96B.5000303@gmail.com> References: <55FFE96B.5000303@gmail.com> Message-ID: <4760812.XdNxuBg2Ug@changa> On 2015?9?21? ??? 14:26:35 Martin Koroudjiev wrote: > -ifdef(CONTAINING_APP_INCLUDE). > -include(?CONTAINING_APP_INCLUDE). > -endif. > > and to compile the source files as: > > erlc -Werror -Wall -DCONTAINING_APP_INCLUDE="\"/.hrl\"" > .erl > > but I get the following error: > > badly formed 'include' I think includes run before macros, so your include is trying to include a very odd-looking thing that starts with a question-mark instead of a filename. Usually things like debug (or other selective behavior definitions) expand to reference a module based on the debug level (or whatever). So ?DEBUG might look like ?DEBUG:report(Foo) in code and one time expand to: `my_logger:report(Foo)` or `my_full_blown_debugger:reort(Foo)` or `my_do_nothing_module:report(Foo)`. In the last case the function might be optimized out entirely if it really does nothing. -Craig From alex.arnon@REDACTED Mon Sep 21 14:09:10 2015 From: alex.arnon@REDACTED (Alex Arnon) Date: Mon, 21 Sep 2015 15:09:10 +0300 Subject: [erlang-questions] Fwd: [ANN] Spawned Shelter In-Reply-To: References: Message-ID: Very very nice, thank you! On Fri, Sep 18, 2015 at 10:02 PM, Federico Carrone < federico.carrone@REDACTED> wrote: > > I have created a simple website called Spawned Shelter ( > http://spawnedshelter.com/). > > The website tries to collect the best articles, videos and books for > learning Erlang, the internals of its virtual machine and other languages > that run on top of it. I think that it can be quite useful for the > community to have a simple website like this for sending out to people who > want to learn Erlang or are learning it. > > If you want to add or remove any link you only have to send a pull request > that changes this markdown file: > > https://github.com/unbalancedparentheses/spawnedshelter/blob/master/src/content.md > > Also if you want to help in sharing it to the rest of the wolrd, you can > vote it in reddit ( > https://www.reddit.com/r/programming/comments/3lgg6v/spawned_shelter_the_best_articles_videos_and/) > or hacker news (https://news.ycombinator.com/item?id=10240572). > > Let me know if you have any recommadation or comment. > > Regards, > Federico. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Mon Sep 21 14:11:55 2015 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 21 Sep 2015 14:11:55 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: On Mon, Sep 21, 2015 at 1:25 PM, Magnus Henoch wrote: > I had a vaguely related idea the other day. When you trace a function > that takes an anonymous function as an argument, it's printed like this: > > 8> lists:all(foo:foo(3), [1, 2]). > (<0.33.0>) call lists:all(#Fun,[1,2]) > (<0.33.0>) returned from lists:all/2 -> false > > That tells you that anonymous function number 0 in module foo is passed as > an argument. It would be convenient to have the 0 be the line number where > the anonymous function is defined instead, so you can quickly figure out > exactly which function is being passed, in case your module creates several > different ones. > > That is more of a formatting thing, as io:format does the same thing. I'll try to keep it in mind though. Adding {line,No} tuple to the return of erlang:fun_info should be simple enough and from there io_lib:format can do what it wants. Just hope we don't step on any backwards compatibility toes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Mon Sep 21 14:14:19 2015 From: rvirding@REDACTED (Robert Virding) Date: Mon, 21 Sep 2015 14:14:19 +0200 Subject: [erlang-questions] Abstract code in .beam files Message-ID: When you compile a file with debug_info then the Erlang AST of the file is included in the .beam file. Does anyone know if the actual structure which is put in the file is checked for format? Can I put anything in there, for example the core code? Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Mon Sep 21 14:22:12 2015 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 21 Sep 2015 14:22:12 +0200 Subject: [erlang-questions] Deep data structures and lenses In-Reply-To: <55F4662B.8050707@gandrade.net> References: <55F4662B.8050707@gandrade.net> Message-ID: <231D943C-08E4-44F8-BC8F-4153083F83F0@feuerlabs.com> > On 12 Sep 2015, at 19:51, Guilherme Andrade wrote: > > Partially inspired by jlouis' erl-lenses[1] and an old discussion on this mailing list[2], I've been sketching a library for easier manipulation (both data fetching and updating) of deep data structures Another library that?s been inspired by erl-lenses (although this has perhaps not been well communicated) is exprecs in parse_trans. Example: -module(lens). -include_lib("parse_trans/include/exprecs.hrl"). -export([t/0]). -record(r, {a, b, c}). -export_records([r]). t() -> R = '#new-r'([{a,1},{b,2},{c,3}]), {GetA, SetA} = Lens = '#lens-r'(a), GetA1 = GetA(R), R1 = SetA(17, R), GetA2 = GetA(R1), [{lens, Lens}, {get_a, GetA1}, {set_a, 17, R1}, {get_a, GetA2}]. ... Eshell V5.10.4 (abort with ^G) 1> c(lens). {ok,lens} 2> lens:t(). [{lens,{#Fun,#Fun}}, {get_a,1}, {set_a,17,{r,17,2,3}}, {get_a,17}] BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Mon Sep 21 14:33:03 2015 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Mon, 21 Sep 2015 14:33:03 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: > That is more of a formatting thing, as io:format does the same thing. I'll > try to keep it in mind though. Adding {line,No} tuple to the return of > erlang:fun_info should be simple enough and from there io_lib:format can do > what it wants. Just hope we don't step on any backwards compatibility toes. > For what is worth, we include the module, function and arity of where the anonymous function was defined when inspecting them in Elixir: #Function<20.90072148/0 in :erl_eval.expr/5> Including the line would be a helpful addition. *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Mon Sep 21 14:46:21 2015 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Mon, 21 Sep 2015 14:46:21 +0200 Subject: [erlang-questions] Abstract code in .beam files In-Reply-To: References: Message-ID: Robert, the .beam file is made of a bunch of chunks. The abstract code is stored in a chunk named "Abst", where it is versioned and term_to_binary'ed. Literally: erlang:term_to_binary({raw_abstract_v1, AST}) You can't/shouldn't store the core format here but you can create as many chunks as you want in the BEAM file. We actually do this in Elixir to store the documentation. At EUC, I talked to Bj?rn Gustavsson about officially supporting a core chunk. Here is what we agreed on to the best of my memory: 1. We will have a new chunk named "Core" (they are all four letters) 2. The core AST will be versioned as well (raw_core_v1) and term_to_binary'ed, like this: erlang:term_to_binary({raw_core_v1, CoreAST}) I also talked to Kostis so dialyzer starts reading the core information from it too. Let me know if you go ahead with this so we can sync before I eventually send a PR to OTP and dialyzer. *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D On Mon, Sep 21, 2015 at 2:14 PM, Robert Virding wrote: > When you compile a file with debug_info then the Erlang AST of the file is > included in the .beam file. Does anyone know if the actual structure which > is put in the file is checked for format? Can I put anything in there, for > example the core code? > > Robert > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From spawn.think@REDACTED Mon Sep 21 15:25:34 2015 From: spawn.think@REDACTED (Ahmed Omar) Date: Mon, 21 Sep 2015 15:25:34 +0200 Subject: [erlang-questions] segfault erts-5.10.4 (R16B03-1) In-Reply-To: References: Message-ID: The crash occurred again, this time I ran a locally compiled gdb on the build environment pointing to the beam.smp file from the build used to generate the release. This worked better.. Here's a link for the bt full output https://gist.github.com/spawnthink/4af73b8e5d6b7f58ec4c#file-btfull-log Any hints would be appreciated Best Regards, - Ahmed Omar http://about.me/spawn.think/ On Wed, Sep 9, 2015 at 2:52 PM, Lukas Larsson wrote: > On Wed, Sep 9, 2015 at 2:19 PM, Ahmed Omar wrote: > >> ### >> [Thread debugging using libthread_db enabled] >> Using host libthread_db library "/lib64/libthread_db.so.1". >> Core was generated by `/var/lib/ejabberd/erts-5.10.4/bin/beam.smp -K true >> -A 128 -P 2500000 -Q 500000'. >> Program terminated with signal SIGSEGV, Segmentation fault. >> #0 0x000000000044d299 in bf_get_free_block () >> [Current thread is 1 (Thread 0x7fe077fbc700 (LWP 24141))] >> (gdb) bt full >> #0 0x000000000044d299 in bf_get_free_block () >> No symbol table info available. >> #1 0x0000000000442fa6 in mbc_alloc () >> No symbol table info available. >> #2 0x0000000000448803 in erts_alcu_alloc_thr_pref () >> No symbol table info available. >> #3 0x0000000000509508 in erts_alloc () >> No symbol table info available. >> #4 0x00000000005096de in erts_bs_append () >> No symbol table info available. >> #5 0x000000000053c1f8 in process_main () >> No symbol table info available. >> #6 0x000000000049da8b in sched_thread_func () >> No symbol table info available. >> #7 0x00000000005bd146 in thr_wrapper () >> No symbol table info available. >> #8 0x00000039a80079d1 in start_thread () from /lib64/libpthread.so.0 >> No symbol table info available. >> #9 0x00000039a78e88fd in clone () from /lib64/libc.so.6 >> No symbol table info available. >> ### >> >> I'll keep digging ... >> >> > It looks like you have a memory corruption issue. Normally when something > goes wrong in the memory allocators, it is because a badly written > nif/linked-in driver has written outside the buffer it requested by calling > driver/enif_alloc. I would take a deep long look at any native code that > you have running and see if you can find any such bugs. If you can > reproduce the error, I would attempt to run the same scenario in valgrind > as it is quite good at spotting these types of errors. > > Lukas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Mon Sep 21 15:35:26 2015 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 21 Sep 2015 15:35:26 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: <55FFF00D.7090305@ninenines.eu> References: <8DF48FD1-5D3E-445E-AE04-D41A5E3886BE@feuerlabs.com> <55FFF00D.7090305@ninenines.eu> Message-ID: <6704DEEC-56D3-4922-97FD-7B9CA6800434@feuerlabs.com> > On 21 Sep 2015, at 13:54, Lo?c Hoguin wrote: > > It would be great to at least be able to use the ttb at the same time as dbg. I ran into this in the past, though I can't recall exactly what my issue was. > > ttb also needs some love with regard to documentation. It's probably worse than dbg, and I would bet that much fewer people know about it. During the PROTEST project, we (ESL et al) did some work on using trace-based tools combined with property validation. We had initially thought to use Inviso, but found that the better option would be to fix some things that were broken in TTB (basically, TTB did most of what we wanted, except it obviously hadn?t been tested, and didn?t work). So some of the outcomes of this work were that OTP retired Inviso, and ESL fixed a number of things in TTB. TTB actually uses DBG, and the things that I think are most useful to people are: - Easily enabling trace on multiple nodes - Fetching and merging trace logs - User-defined formatting of the merged log - Being able to save the command history and replaying it later. BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From vladdu55@REDACTED Mon Sep 21 15:41:23 2015 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Mon, 21 Sep 2015 15:41:23 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: <8DF48FD1-5D3E-445E-AE04-D41A5E3886BE@feuerlabs.com> References: <8DF48FD1-5D3E-445E-AE04-D41A5E3886BE@feuerlabs.com> Message-ID: Hi! On Mon, Sep 21, 2015 at 12:02 PM, Ulf Wiger wrote: > On 21 Sep 2015, at 10:52, Lukas Larsson wrote: > > To start the discussion, here are a few of my ideas in no particular order: > * Allow multiple tracers. Today only one port/process can be the > receiver of trace data. > * Create a couple of scalable high throughput tracing output backends > with different overflow mechanics. Today all tracing is funneled through > one bottleneck and has no overflow handling at all. > * Expose vm probes (today dtrace probes) to the erlang tracer. > * Better integration of dtrace/lttngt/systemtap into the erlang trace. > * Allow the erlang tracer to be an Erlang callback module. Today only > ports/processes are allowed. > * Optimize trace output to file/ip. Maybe use something like the Common > Trace Format ( > http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.md;hb=master), > instead of the term_to_binary that we have today. > * Write much much better documentation for dbg :) > > > +1 on the suggestions, esp. multiple tracers. The one-tracer restriction > makes it hard to explore some very interesting monitoring and debugging > ideas. > Those are very good starting points. For multiple tracers, I would assume that they will be independent - i.e. not all of them get all tracing events, but only the ones they are interested in. The question is if the tracing engine can be made fast enough, possibly only if the filtering criteria are restricted to term matching (as opposed to arbitrary predicates provided by users). Also, it might be interesting to consider having the configuration in a declarative format that is easier to understand and maintain, while also allowing to keep the configuration separate from the code (in an external file). It's not difficult to implement that on top the current API, but a common and official format would feel a lot better. > One might fantasize about enabling migration from io:format() debugging to > this, ideally by simply rewriting the debug macro: a handy wrapper that > simulates the original printouts. There are some problems with this, so > I?ve chosen not to go there. > Ulf, could you please sketch the issues that you encountered? best regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrtndimitrov@REDACTED Mon Sep 21 15:46:34 2015 From: mrtndimitrov@REDACTED (Martin Koroudjiev) Date: Mon, 21 Sep 2015 16:46:34 +0300 Subject: [erlang-questions] -include with define macro In-Reply-To: <4760812.XdNxuBg2Ug@changa> References: <55FFE96B.5000303@gmail.com> <4760812.XdNxuBg2Ug@changa> Message-ID: <56000A3A.9070703@gmail.com> Thanks for the info. I will organize the code as you suggested. Regards, Martin On 9/21/2015 2:56 PM, zxq9 wrote: > On 2015?9?21? ??? 14:26:35 Martin Koroudjiev wrote: >> -ifdef(CONTAINING_APP_INCLUDE). >> -include(?CONTAINING_APP_INCLUDE). >> -endif. >> >> and to compile the source files as: >> >> erlc -Werror -Wall -DCONTAINING_APP_INCLUDE="\"/.hrl\"" >> .erl >> >> but I get the following error: >> >> badly formed 'include' > I think includes run before macros, so your include is trying to include a very odd-looking thing that starts with a question-mark instead of a filename. > > Usually things like debug (or other selective behavior definitions) expand to reference a module based on the debug level (or whatever). So ?DEBUG might look like > > ?DEBUG:report(Foo) > > in code and one time expand to: `my_logger:report(Foo)` or `my_full_blown_debugger:reort(Foo)` or `my_do_nothing_module:report(Foo)`. In the last case the function might be optimized out entirely if it really does nothing. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ulf@REDACTED Mon Sep 21 15:46:51 2015 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 21 Sep 2015 15:46:51 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: <8DF48FD1-5D3E-445E-AE04-D41A5E3886BE@feuerlabs.com> Message-ID: > On 21 Sep 2015, at 15:41, Vlad Dumitrescu wrote: > > One might fantasize about enabling migration from io:format() debugging to this, ideally by simply rewriting the debug macro: a handy wrapper that simulates the original printouts. There are some problems with this, so I?ve chosen not to go there. > > Ulf, could you please sketch the issues that you encountered? Well, the main issue would be that most debug macros are versions of io:format/2, and for a high-level ?trace event? function, I?d like a different convention than (Format, Args). BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladdu55@REDACTED Mon Sep 21 16:06:01 2015 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Mon, 21 Sep 2015 16:06:01 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: <8DF48FD1-5D3E-445E-AE04-D41A5E3886BE@feuerlabs.com> Message-ID: On Mon, Sep 21, 2015 at 3:46 PM, Ulf Wiger wrote: > On 21 Sep 2015, at 15:41, Vlad Dumitrescu wrote: > > One might fantasize about enabling migration from io:format() debugging to >> this, ideally by simply rewriting the debug macro: a handy wrapper that >> simulates the original printouts. There are some problems with this, so >> I?ve chosen not to go there. >> > > Ulf, could you please sketch the issues that you encountered? > > Well, the main issue would be that most debug macros are versions of > io:format/2, and for a high-level ?trace event? function, I?d like a > different convention than (Format, Args). > > Ah, I see, I thought there were issues of a more technical type. Thanks. regards, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Mon Sep 21 16:12:01 2015 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 21 Sep 2015 16:12:01 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: <8DF48FD1-5D3E-445E-AE04-D41A5E3886BE@feuerlabs.com> Message-ID: On Mon, Sep 21, 2015 at 3:41 PM, Vlad Dumitrescu wrote: > Hi! > > On Mon, Sep 21, 2015 at 12:02 PM, Ulf Wiger wrote: > >> On 21 Sep 2015, at 10:52, Lukas Larsson wrote: >> >> To start the discussion, here are a few of my ideas in no particular >> order: >> * Allow multiple tracers. Today only one port/process can be the >> receiver of trace data. >> * Create a couple of scalable high throughput tracing output backends >> with different overflow mechanics. Today all tracing is funneled through >> one bottleneck and has no overflow handling at all. >> * Expose vm probes (today dtrace probes) to the erlang tracer. >> * Better integration of dtrace/lttngt/systemtap into the erlang trace. >> * Allow the erlang tracer to be an Erlang callback module. Today only >> ports/processes are allowed. >> * Optimize trace output to file/ip. Maybe use something like the Common >> Trace Format ( >> http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.md;hb=master), >> instead of the term_to_binary that we have today. >> * Write much much better documentation for dbg :) >> >> >> +1 on the suggestions, esp. multiple tracers. The one-tracer restriction >> makes it hard to explore some very interesting monitoring and debugging >> ideas. >> > > Those are very good starting points. > > For multiple tracers, I would assume that they will be independent - i.e. > not all of them get all tracing events, but only the ones they are > interested in. The question is if the tracing engine can be made fast > enough, possibly only if the filtering criteria are restricted to term > matching (as opposed to arbitrary predicates provided by users). > Yes, they would be independent. I envision filtering being done as today, i.e. through match specs that are run for when a trace pattern that has been triggered. Each tracer would have it's own filter that has to be executed before a decision is made whether a trace should be generated or not, so having multiple tracers will be more expensive than one. Luckily it is quite hard to write long match specs, so normally each filter is very fast to be checked :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From touch.sereysethy@REDACTED Mon Sep 21 16:33:07 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Mon, 21 Sep 2015 21:33:07 +0700 Subject: [erlang-questions] Erlang OTP 18 memory leak / SSL In-Reply-To: References: <55FFD76F.8080903@ninenines.eu> <55FFDFEC.20701@ninenines.eu> Message-ID: Dear all, Finally I know what is the difference between Erlang 17 and Erlang 18 in ssl module which caused a problem of memory. It is not really a memory leak but simply the caching problem. I take a look at the source code of ssl module (ssl_manager.erl) in OTP-17 & OTP-18. It is how they define the 24h in seconds. In OTP-17, somebody did a wrong multiplication. While in OTP-18, the calculations were correct for both 24h-in-sec & 24h-in-msec! By the way, thanks to Ingela, Lo?c and Jesper for helping me finding the problems and solution. Thanks, T. `Erlang OTP 17` -define('24H_in_msec', 8640000). -define('24H_in_sec', 8640). -define(GEN_UNIQUE_ID_MAX_TRIES, 10). -define(SESSION_VALIDATION_INTERVAL, 60000). -define(CLEAR_PEM_CACHE, 120000). -define(CLEAN_SESSION_DB, 60000). -define(CLEAN_CERT_DB, 500). -define(NOT_TO_BIG, 10). `Erlang OTP 18` -define('24H_in_msec', 86400000). -define('24H_in_sec', 86400). -define(GEN_UNIQUE_ID_MAX_TRIES, 10). -define(SESSION_VALIDATION_INTERVAL, 60000). -define(CLEAR_PEM_CACHE, 120000). -define(CLEAN_SESSION_DB, 60000). -define(CLEAN_CERT_DB, 500). -define(NOT_TO_BIG, 10). On Mon, Sep 21, 2015 at 6:00 PM, Sereysethy TOUCH < touch.sereysethy@REDACTED> wrote: > I got it Lo?c. > > Thanks. > > On Mon, Sep 21, 2015 at 5:52 PM, Sereysethy TOUCH < > touch.sereysethy@REDACTED> wrote: > >> Something like this? >> >> {release, {ws_user, "1"}, >> >> [ws_user]}. >> >> {sys_config, "./config/sys.config"}. >> >> %{paths, ["../deps/"]}. >> >> % generate startup script, yes >> >> {extended_start_script, true}. >> >> On Mon, Sep 21, 2015 at 5:46 PM, Lo?c Hoguin wrote: >> >>> Add {sys_config, "path/to/sys.config"} to your relx.config, then create >>> that sys.config file with the right options and run make. >>> >>> On 09/21/2015 12:44 PM, Sereysethy TOUCH wrote: >>> >>>> Hello, >>>> >>>> You meant _rel or I need to create a folder "rel" and put sys.config >>>> inside? >>>> >>>> Sethy >>>> >>>> On Mon, Sep 21, 2015 at 5:09 PM, Lo?c Hoguin >>> > wrote: >>>> >>>> On 09/21/2015 12:05 PM, Sereysethy TOUCH wrote: >>>> >>>> Hello, >>>> >>>> My application is both TLS-client & server. It is a web server >>>> which >>>> allows client browser to connect to server using websocket. I >>>> use cowboy >>>> as web server & web socket server. >>>> >>>> In erlang 17, our application runs for weeks and there is no >>>> memory >>>> issue. We only restart it whenever we have a new update. But >>>> since we >>>> update to elang 18, we need to restart our application every 2 >>>> hours. >>>> >>>> I use erlang.mk to >>>> compile >>>> our application, I am not >>>> sure how to set application env. settings. Is it the same if I >>>> use >>>> erlang.mk ? >>>> >>>> >>>> If you run a release, you can put the configuration in the >>>> sys.config file (by default rel/sys.config) and run make again. See >>>> http://www.erlang.org/doc/man/config.html >>>> >>>> >>>> >>>> Thanks, >>>> Sethy >>>> >>>> On Mon, Sep 21, 2015 at 4:54 PM, Ingela Andin >>>> >>>> >>> >>> >>>> wrote: >>>> >>>> Hi! >>>> >>>> >>>> 2015-09-21 10:09 GMT+02:00 Sereysethy TOUCH >>>> >>> >>>> >>> >>>> >>: >>>> >>>> >>>> Hello, >>>> >>>> One question, why did my application work fine on >>>> Erlang 17.5.x >>>> branch? >>>> >>>> >>>> Well I am not sure what your application does. >>>> >>>> If you have a client application that spawns a lot of >>>> parallel >>>> connections to the same host, not waiting for one to >>>> succeed before >>>> spawning, >>>> the client session table could grow a lot. However I have >>>> implemented a feature to only save unique TLS-sessions in >>>> this >>>> senario to avoid this, and there was a bug so that this >>>> feature did >>>> not work, that I have fix in the maint branch. This bug >>>> however was >>>> in 17 as well as 18. >>>> >>>> However the ets info you sent suggest you have a server >>>> application >>>> and then setting session_liftime can help keeping the >>>> servers >>>> session table >>>> smaller. This also is true for clients, that can get a big >>>> table >>>> also through connecting to many diffrent servers. >>>> >>>> Maybe your application can be both a TLS-client and server? >>>> There is >>>> no obvious reason from what you told us so far that it >>>> should work >>>> better in 17 then 18. >>>> >>>> >>>> If I want to set this session_lifetime option, where >>>> should put it? >>>> >>>> >>>> It is an application environment setting. Search for app in >>>> the docs. >>>> >>>> >>>> So there is no use that I should build the erlang 18 >>>> from maint >>>> branch? >>>> >>>> >>>> >>>> It is always a good thing to try the latest version, and >>>> if you >>>> still have a problem it would be good if you could try >>>> providing a >>>> way to reproduce the problem. >>>> >>>> Regards Ingela Erlang/OTP tema Ericsson AB >>>> >>>> >>>> Thanks, >>>> Sethy >>>> >>>> On Mon, Sep 21, 2015 at 2:51 PM, Ingela Andin >>>> >>> > >>>> >>> >>> >>>> wrote: >>>> >>>> Hello, >>>> >>>> 2015-09-20 15:24 GMT+02:00 Sereysethy TOUCH >>>> >>> >>>> >>> >>>> >>: >>>> >>>> >>>> Hello, >>>> >>>> I did as what you told me by getting a shell >>>> console on >>>> the node. >>>> >>>> I run memory(). and see ets ate the most >>>> memory. And it >>>> increased overtime. I run ets:i(). and I found >>>> out the >>>> ssl_manager took a big chunk of what memory >>>> used. >>>> >>>> 32794 client_ssl_otp_session_cache >>>> ordered_set >>>> 0 89 ssl_manager >>>> 36891 server_ssl_otp_session_cache >>>> ordered_set >>>> 1564 67995 ssl_manager >>>> 40993 >>>> httpc_manager__session_cookie_db bag 0 >>>> 299 httpc_manager >>>> >>>> So I think there is a problem with ssl_manager >>>> in Erlang >>>> OTP 18. >>>> >>>> Any workarounds? >>>> >>>> >>>> >>>> Ok, the bug I fixed with the session table is on >>>> the client >>>> side. But your big ets table is on the server >>>> side. >>>> Session data is by default fairly long lived (24 h >>>> that is >>>> the max recommended time to keep a session by the >>>> RFC), >>>> you can set the ssl application variable >>>> session_lifetime to >>>> make its lifetime shorter which should mitigate >>>> your problem. >>>> We have a backlog item to make a configurable >>>> threshold of >>>> max sessions allowed to be stored in the table, >>>> which I am >>>> sure will be >>>> implemented fairly soon but alas not for 18.1. >>>> >>>> Regards Ingela Erlang/OTP Team - Ericsson AB >>>> >>>> Sethy >>>> >>>> On Sun, Sep 20, 2015 at 2:22 AM, Jesper Louis >>>> Andersen >>>> >>> >>>> >>> >> wrote: >>>> >>>> >>>> On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy >>>> TOUCH >>>> >>> >>>> >>> >>>> >> wrote: >>>> >>>> I just recently updated Erlang to >>>> latest version >>>> OTP 18 on Ubuntu server. It uses cowboy >>>> (websocket), ranch, ssl, erlydtl & >>>> rabbitmq. It >>>> used to work fine in OTP 17. The >>>> program is >>>> correctly compiled but during the >>>> execution the >>>> memory kept increasing. I need to >>>> restart the >>>> process every one or two hours to free >>>> some memory. >>>> >>>> >>>> 1. Get a shell console on the node >>>> 2. call 'memory().' in the shell >>>> 3. If it reports ets as the winner, call >>>> 4. ets:i(). >>>> >>>> This will give you a nice overview for >>>> where to look >>>> and verify it is the SSL problem. >>>> >>>> >>>> -- >>>> J. >>>> >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>> > >>>> >>> > >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>> > >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>>> -- >>>> Lo?c Hoguin >>>> http://ninenines.eu >>>> Author of The Erlanger Playbook, >>>> A book about software development using Erlang >>>> >>>> >>>> >>> -- >>> Lo?c Hoguin >>> http://ninenines.eu >>> Author of The Erlanger Playbook, >>> A book about software development using Erlang >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Mon Sep 21 17:36:18 2015 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 21 Sep 2015 17:36:18 +0200 Subject: [erlang-questions] -include with define macro In-Reply-To: <4760812.XdNxuBg2Ug@changa> References: <55FFE96B.5000303@gmail.com> <4760812.XdNxuBg2Ug@changa> Message-ID: On Mon, Sep 21, 2015 at 1:56 PM, zxq9 wrote: > On 2015?9?21? ??? 14:26:35 Martin Koroudjiev wrote: > > -ifdef(CONTAINING_APP_INCLUDE). > > -include(?CONTAINING_APP_INCLUDE). > > -endif. > > > > and to compile the source files as: > > > > erlc -Werror -Wall -DCONTAINING_APP_INCLUDE="\"/.hrl\"" > > .erl > > > > but I get the following error: > > > > badly formed 'include' > > I think includes run before macros, so your include is trying to include a > very odd-looking thing that starts with a question-mark instead of a > filename. > Not quite. The -include and -include_lib directives require that the thing between the parentheses is a string literal, so the tokens '?' 'CONTAINING_APP_INCLUDE' are not even subject to macro expansion in this case. (This could probably be easily fixed, if it is important to anyone.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Mon Sep 21 17:44:46 2015 From: g@REDACTED (Garrett Smith) Date: Mon, 21 Sep 2015 10:44:46 -0500 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: My wish for tracing in Erlang is that it be dead simple to use. Maybe this has changed or my impression is just wrong, but I've found the tracing facilities to be quite complex to jump in with. e2 has e2_debug, which I use all the time: https://github.com/gar1t/e2/blob/master/src/e2_debug.erl This simplifies the interface to tracing (at least it's suitable for my general needs) and cleans up the output (IMO). I really like redbug's ability to install a trace for a period of time and automatically remove it. redbug is a damn nice tool that's seen heavy production use (from what I understand). It could perhaps serve as a point of inspiration. On Mon, Sep 21, 2015 at 3:52 AM, Lukas Larsson wrote: > Hello everyone. > > As you may know, one of the OTP teams focus areas for the coming year is > make tracing better. At the moment we are gathering ideas and attempting to > put together a vision of what we would like to have, before deciding what we > can make. > > I'm pretty sure that many of you have much more experience with using Erlang > tracing while developing and in production than I do, which is we would love > to have your input as to what you would like to change about tracing. > > To set the scope of the discussion, when I say tracing I include; erlang > tracing, dtrace/systemtap, trace outputs (stdout/file/IP), filtering through > match specs, sequence tracing, tool integration (dbg, fprof, redbug, recon > to mention some) and probably more. > > To start the discussion, here are a few of my ideas in no particular order: > * Allow multiple tracers. Today only one port/process can be the receiver > of trace data. > * Create a couple of scalable high throughput tracing output backends with > different overflow mechanics. Today all tracing is funneled through one > bottleneck and has no overflow handling at all. > * Expose vm probes (today dtrace probes) to the erlang tracer. > * Better integration of dtrace/lttngt/systemtap into the erlang trace. > * Allow the erlang tracer to be an Erlang callback module. Today only > ports/processes are allowed. > * Optimize trace output to file/ip. Maybe use something like the Common > Trace Format > (http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.md;hb=master), > instead of the term_to_binary that we have today. > * Write much much better documentation for dbg :) > > We are looking for feedback from both beginners as well as seasoned veterans > to make erlang tracing the best it can be. So if you have any thoughts or > ideas, join the discussion to make Erlang tracing better for you and > everyone else. > > Thanks in advance, > Lukas > Erlang/OTP team > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From mononcqc@REDACTED Mon Sep 21 17:58:14 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 21 Sep 2015 11:58:14 -0400 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: <20150921155813.GF9409@fhebert-ltm1> On 09/21, Lukas Larsson wrote: >I'm pretty sure that many of you have much more experience with using >Erlang tracing while developing and in production than I do, which is we >would love to have your input as to what you would like to change about >tracing. > >To set the scope of the discussion, when I say tracing I include; erlang >tracing, dtrace/systemtap, trace outputs (stdout/file/IP), filtering >through match specs, sequence tracing, tool integration (dbg, fprof, >redbug, recon to mention some) and probably more. > I love the idea of multiple tracers. One thing I think would be nifty is to be able to trace the output of a function, but only if it matches a given pattern, rather than only being able to do it with the function arguments. Otherwise, just generally decreasing the impact of Erlang tracing on a prod system would be good. In my "let's handwave away all complexity", a thing I'd love to be able to do is know where a function call comes from. Too often I can trace the function call that fails or goes bad, but I have no manageable way to trace or extract which function call sits above it in the call stack (even if LCO makes it less useful). Regards, Fred. From lukas@REDACTED Mon Sep 21 19:03:04 2015 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 21 Sep 2015 19:03:04 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: On Mon, Sep 21, 2015 at 5:44 PM, Garrett Smith wrote: > My wish for tracing in Erlang is that it be dead simple to use. Maybe > this has changed or my impression is just wrong, but I've found the > tracing facilities to be quite complex to jump in with. > I believe that better documentation, especially more example usages, should help with this. Together with some simplification of the APIs in dbg and ttb, I might even be able to persuade some of the hard-core io:formatters here at OTP to use tracing... -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Mon Sep 21 19:10:59 2015 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 21 Sep 2015 19:10:59 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: <20150921155813.GF9409@fhebert-ltm1> References: <20150921155813.GF9409@fhebert-ltm1> Message-ID: On Mon, Sep 21, 2015 at 5:58 PM, Fred Hebert wrote: > One thing I think would be nifty is to be able to trace the output of a > function, but only if it matches a given pattern, rather than only being > able to do it with the function arguments. > Hmm, I hadn't thought about wanting to do that.... I'll add it to my list of things to think about. I've never looked at how the return_trace stuff is implemented, so could either be very simple to do, or very very hard... In my "let's handwave away all complexity", a thing I'd love to be able to > do is know where a function call comes from. Too often I can trace the > function call that fails or goes bad, but I have no manageable way to trace > or extract which function call sits above it in the call stack (even if LCO > makes it less useful). > Do you mean something like what is possible today with {message,{caller}} and/or {message,{process_dump}}? i.e. dbg:tracer(),dbg:p(all,c), dbg:tp(lists,map, [{'_',[],[{message,{caller}}]}]). -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Mon Sep 21 19:47:13 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 21 Sep 2015 13:47:13 -0400 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: <20150921155813.GF9409@fhebert-ltm1> Message-ID: <20150921174711.GG9409@fhebert-ltm1> On 09/21, Lukas Larsson wrote: > >Do you mean something like what is possible today with {message,{caller}} >and/or {message,{process_dump}}? i.e. dbg:tracer(),dbg:p(all,c), >dbg:tp(lists,map, [{'_',[],[{message,{caller}}]}]). I used mostly the trace BIFs directly. I'm guessing this would be the 'meta' specs if it exists? I didn't think it possible given the documentation: meta Return the meta trace tracer process or port for this function, if it has one. If the function is not meta traced the returned value is false, and if the function is meta traced but has once detected that the tracer proc is invalid, the returned value is []. meta_match_spec Return the meta trace match specification for this function, if it has one. If the function is meta traced but has no match specification defined, the returned value is []. If that stuff lets me trace parents of a given call, then I'm already happy with that -- the doc just isn't clear what its intended use is. From lukas@REDACTED Mon Sep 21 19:58:48 2015 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 21 Sep 2015 19:58:48 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: <20150921174711.GG9409@fhebert-ltm1> References: <20150921155813.GF9409@fhebert-ltm1> <20150921174711.GG9409@fhebert-ltm1> Message-ID: On Mon, Sep 21, 2015 at 7:47 PM, Fred Hebert wrote: > On 09/21, Lukas Larsson wrote: > >> >> Do you mean something like what is possible today with {message,{caller}} >> and/or {message,{process_dump}}? i.e. dbg:tracer(),dbg:p(all,c), >> dbg:tp(lists,map, [{'_',[],[{message,{caller}}]}]). >> > > I used mostly the trace BIFs directly. I'm guessing this would be the > 'meta' specs if it exists? > No, it uses the message part of the match spec syntax: http://www.erlang.org/doc/apps/erts/match_spec.html The match spec is just handed to erlang:trace_pattern, so erlang:trace_pattern({lists,map,'_'}, [{'_',[],[{message,{caller}}]}], []) gives the caller (not the entire call chain). I'm not really sure when the meta stuff is used outside of {return_trace}. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Mon Sep 21 20:08:14 2015 From: g@REDACTED (Garrett Smith) Date: Mon, 21 Sep 2015 13:08:14 -0500 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: On Mon, Sep 21, 2015 at 12:03 PM, Lukas Larsson wrote: > On Mon, Sep 21, 2015 at 5:44 PM, Garrett Smith wrote: >> >> My wish for tracing in Erlang is that it be dead simple to use. Maybe >> this has changed or my impression is just wrong, but I've found the >> tracing facilities to be quite complex to jump in with. > > I believe that better documentation, especially more example usages, should > help with this. Together with some simplification of the APIs in dbg and > ttb, I might even be able to persuade some of the hard-core io:formatters > here at OTP to use tracing... Tracing is so much easier and better than io format! Though it needs to be as simple as: trace_function(Mod, Fun) The stuff in e2_debug took a fair amount of effort for me to divine - but even with that knowledge, I surely don't want to mess with anything more than a single function call. I also spent a fair amount of effort in getting the output to be usable (for me). Small details perhaps, but if I didn't have that facility as it is, I'd be an ioformatter :) From mmartin4242@REDACTED Mon Sep 21 20:58:31 2015 From: mmartin4242@REDACTED (Michael L Martin) Date: Mon, 21 Sep 2015 13:58:31 -0500 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: <56005357.5020309@gmail.com> I recently tried tracing, and really couldn't figure out how to make it work for my needs. What I would like to be able to do is to turn tracing on for particular modules/functions in my top level app.erl module, rather than attach to a running application and starting tracing there. If that's already possible, I couldn't find a way to do it. It would also be very, very helpful to see from where the function was called - I think someone else has already requested that. Thanks, On 09/21/2015 03:52 AM, Lukas Larsson wrote: > Hello everyone. > > As you may know, one of the OTP teams focus areas for the coming year > is make tracing better. At the moment we are gathering ideas and > attempting to put together a vision of what we would like to have, > before deciding what we can make. > > I'm pretty sure that many of you have much more experience with using > Erlang tracing while developing and in production than I do, which is > we would love to have your input as to what you would like to change > about tracing. > > To set the scope of the discussion, when I say tracing I include; > erlang tracing, dtrace/systemtap, trace outputs (stdout/file/IP), > filtering through match specs, sequence tracing, tool integration > (dbg, fprof, redbug, recon to mention some) and probably more. > > To start the discussion, here are a few of my ideas in no particular > order: > * Allow multiple tracers. Today only one port/process can be the > receiver of trace data. > * Create a couple of scalable high throughput tracing output > backends with different overflow mechanics. Today all tracing is > funneled through one bottleneck and has no overflow handling at all. > * Expose vm probes (today dtrace probes) to the erlang tracer. > * Better integration of dtrace/lttngt/systemtap into the erlang trace. > * Allow the erlang tracer to be an Erlang callback module. Today > only ports/processes are allowed. > * Optimize trace output to file/ip. Maybe use something like the > Common Trace Format > (http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.md;hb=master), > instead of the term_to_binary that we have today. > * Write much much better documentation for dbg :) > > We are looking for feedback from both beginners as well as seasoned > veterans to make erlang tracing the best it can be. So if you have any > thoughts or ideas, join the discussion to make Erlang tracing better > for you and everyone else. > > Thanks in advance, > Lukas > Erlang/OTP team > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony@REDACTED Mon Sep 21 21:48:01 2015 From: tony@REDACTED (Tony Rogvall) Date: Mon, 21 Sep 2015 21:48:01 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: Hi! Here is my input: While thinking about how to trace erlang program execution I came to think about how the ?line? beam op code works. The idea is to add a new opcode called ?pint? ( short for print. ) The idea has been around for a long time and is also called printf debugging :-) ( heavily inspired by Mats Cronqvist ) The ?pint? opcode has the feature that it can be removed while loading a module, or it can stay lurking for later debugging opportunities. The ?pint? opcode has an argument, and it is the bound variables at the line in the source code. If the compiler could preserve the the variables names ( transform them a bit more gently through the core and kernel passes ) it could be enough just print the module, line and bound variabels to get a sense of what is going on sometimes. Example Given the program: %% %% Simple test program %% -module(fib). -compile(export_all). nth(0) -> 1; nth(1) -> 1; nth(N) -> N1 = N-1, N2 = N-2, nth(N1) + nth(N2). %% ???? My patched compiler generates the following beam code: {module, fib}. %% version = 0 {exports, [{module_info,0},{module_info,1},{nth,1}]}. {attributes, []}. {labels, 9}. {function, nth, 1, 2}. {label,1}. {line,[{location,"fib.erl",8}]}. {func_info,{atom,fib},{atom,nth},1}. {label,2}. {pint,{literal,[{cor0,{x,0}}]}}. {test,is_integer,{f,4},[{x,0}]}. {select_val,{x,0},{f,4},{list,[{integer,1},{f,3},{integer,0},{f,3}]}}. {label,3}. {move,{integer,1},{x,0}}. return. {label,4}. {allocate_zero,1,1}. {line,[{location,"fib.erl",11}]}. {pint,{literal,[{cor0,{x,0}}]}}. {gc_bif,'-',{f,0},1,[{x,0},{integer,1}],{x,1}}. {line,[{location,"fib.erl",12}]}. {pint,{literal,[{cor0,{x,0}},{'N1',{x,1}}]}}. {gc_bif,'-',{f,0},2,[{x,0},{integer,2}],{y,0}}. {move,{x,1},{x,0}}. {line,[{location,"fib.erl",13}]}. {pint,{literal,[{cor4,{x,0}}]}}. {call,1,{f,2}}. {move,{x,0},{x,1}}. {move,{y,0},{x,0}}. {move,{x,1},{y,0}}. {line,[{location,"fib.erl",13}]}. {pint,{literal,[{cor3,{x,0}}]}}. {call,1,{f,2}}. {line,[{location,"fib.erl",13}]}. {pint,{literal,[{cor3,{x,0}}]}}. {gc_bif,'+',{f,0},1,[{y,0},{x,0}],{x,0}}. {deallocate,1}. return. And the output when running a modified runtime system currently look like this: Erlang/OTP 17 [erts-6.3.1] [source-f8b6ef6] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] Eshell V6.3.1 (abort with ^G) 1> fib:nth(4). fib.erl:8: [fib:nth/1] {x,0}=cor0=4 fib.erl:11: [fib:nth/1] {x,0}=cor0=4 fib.erl:12: [fib:nth/1] {x,0}=cor0=4 {x,1}=N1=3 fib.erl:13: [fib:nth/1] {x,0}=cor4=3 fib.erl:8: [fib:nth/1] {x,0}=cor0=3 fib.erl:11: [fib:nth/1] {x,0}=cor0=3 fib.erl:12: [fib:nth/1] {x,0}=cor0=3 {x,1}=N1=2 fib.erl:13: [fib:nth/1] {x,0}=cor4=2 fib.erl:8: [fib:nth/1] {x,0}=cor0=2 fib.erl:11: [fib:nth/1] {x,0}=cor0=2 fib.erl:12: [fib:nth/1] {x,0}=cor0=2 {x,1}=N1=1 fib.erl:13: [fib:nth/1] {x,0}=cor4=1 fib.erl:8: [fib:nth/1] {x,0}=cor0=1 fib.erl:13: [fib:nth/1] {x,0}=cor3=0 fib.erl:8: [fib:nth/1] {x,0}=cor0=0 fib.erl:13: [fib:nth/1] {x,0}=cor3=1 fib.erl:13: [fib:nth/1] {x,0}=cor3=1 fib.erl:8: [fib:nth/1] {x,0}=cor0=1 fib.erl:13: [fib:nth/1] {x,0}=cor3=1 fib.erl:13: [fib:nth/1] {x,0}=cor3=2 fib.erl:8: [fib:nth/1] {x,0}=cor0=2 fib.erl:11: [fib:nth/1] {x,0}=cor0=2 fib.erl:12: [fib:nth/1] {x,0}=cor0=2 {x,1}=N1=1 fib.erl:13: [fib:nth/1] {x,0}=cor4=1 fib.erl:8: [fib:nth/1] {x,0}=cor0=1 fib.erl:13: [fib:nth/1] {x,0}=cor3=0 fib.erl:8: [fib:nth/1] {x,0}=cor0=0 fib.erl:13: [fib:nth/1] {x,0}=cor3=1 fib.erl:13: [fib:nth/1] {x,0}=cor3=2 5 Yes it uses fib 1,1 definition and the output is not good enough yet, but with some effort in the compiler to tracking variables names and possibly be able to output return values etc. But even more important is a system that can turn the pints on or off, either by line, function, module and pid etc. Also a system that can collect the output from the traces and deliver them to remote systems, filter them etc. Well, for what its worth... /Tony > On 21 sep 2015, at 10:52, Lukas Larsson wrote: > > Hello everyone. > > As you may know, one of the OTP teams focus areas for the coming year is make tracing better. At the moment we are gathering ideas and attempting to put together a vision of what we would like to have, before deciding what we can make. > > I'm pretty sure that many of you have much more experience with using Erlang tracing while developing and in production than I do, which is we would love to have your input as to what you would like to change about tracing. > > To set the scope of the discussion, when I say tracing I include; erlang tracing, dtrace/systemtap, trace outputs (stdout/file/IP), filtering through match specs, sequence tracing, tool integration (dbg, fprof, redbug, recon to mention some) and probably more. > > To start the discussion, here are a few of my ideas in no particular order: > * Allow multiple tracers. Today only one port/process can be the receiver of trace data. > * Create a couple of scalable high throughput tracing output backends with different overflow mechanics. Today all tracing is funneled through one bottleneck and has no overflow handling at all. > * Expose vm probes (today dtrace probes) to the erlang tracer. > * Better integration of dtrace/lttngt/systemtap into the erlang trace. > * Allow the erlang tracer to be an Erlang callback module. Today only ports/processes are allowed. > * Optimize trace output to file/ip. Maybe use something like the Common Trace Format (http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.md;hb=master), instead of the term_to_binary that we have today. > * Write much much better documentation for dbg :) > > We are looking for feedback from both beginners as well as seasoned veterans to make erlang tracing the best it can be. So if you have any thoughts or ideas, join the discussion to make Erlang tracing better for you and everyone else. > > Thanks in advance, > Lukas > Erlang/OTP team > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From s.j.thompson@REDACTED Mon Sep 21 22:08:01 2015 From: s.j.thompson@REDACTED (Simon Thompson) Date: Mon, 21 Sep 2015 22:08:01 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: <6DC236FB-01AF-49C7-8432-39B3199AEDED@kent.ac.uk> Lukas - that?s great news. I see that discussions have already aired filtering, and I think that the main lesson for me from RELEASE was it?s crucial to find ways to deal with torrents of potential data. It?s no good to generate trace info which is filtered out downstream: if it can be filtered at source, so only generated when required, then that would be great. Another facility which may be problematic to implement, but which could be really helpful, would be mechanisms to aggregate information as part of the BEAM: e.g. to be able to emit a trace message every time 1000 messages have passed from process A to process B, or node A to node B. Again this would push into the infrastructure something that currently needs to be done in trace processing. Simon > On 21 Sep 2015, at 10:52, Lukas Larsson wrote: > > Hello everyone. > > As you may know, one of the OTP teams focus areas for the coming year is make tracing better. At the moment we are gathering ideas and attempting to put together a vision of what we would like to have, before deciding what we can make. > > I'm pretty sure that many of you have much more experience with using Erlang tracing while developing and in production than I do, which is we would love to have your input as to what you would like to change about tracing. > > To set the scope of the discussion, when I say tracing I include; erlang tracing, dtrace/systemtap, trace outputs (stdout/file/IP), filtering through match specs, sequence tracing, tool integration (dbg, fprof, redbug, recon to mention some) and probably more. > > To start the discussion, here are a few of my ideas in no particular order: > * Allow multiple tracers. Today only one port/process can be the receiver of trace data. > * Create a couple of scalable high throughput tracing output backends with different overflow mechanics. Today all tracing is funneled through one bottleneck and has no overflow handling at all. > * Expose vm probes (today dtrace probes) to the erlang tracer. > * Better integration of dtrace/lttngt/systemtap into the erlang trace. > * Allow the erlang tracer to be an Erlang callback module. Today only ports/processes are allowed. > * Optimize trace output to file/ip. Maybe use something like the Common Trace Format (http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.md;hb=master ), instead of the term_to_binary that we have today. > * Write much much better documentation for dbg :) > > We are looking for feedback from both beginners as well as seasoned veterans to make erlang tracing the best it can be. So if you have any thoughts or ideas, join the discussion to make Erlang tracing better for you and everyone else. > > Thanks in advance, > Lukas > Erlang/OTP team > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Simon Thompson | Professor of Logic and Computation School of Computing | University of Kent | Canterbury, CT2 7NF, UK s.j.thompson@REDACTED | M +44 7986 085754 | W www.cs.kent.ac.uk/~sjt -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Mon Sep 21 23:15:04 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 21 Sep 2015 23:15:04 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: <56005357.5020309@gmail.com> References: <56005357.5020309@gmail.com> Message-ID: On Mon, Sep 21, 2015 at 8:58 PM, Michael L Martin wrote: > If that's already possible, I couldn't find a way to do it. > [I do have comments, but this question warrants an answer] You can do this via 'dbg', because it is the same calls you have to make. The problem is you need to set up call tracing on the functions for which you are interested. And you also need to set up which pids it affects. The intersection, i.e., the marked pids which calls the marked functions, will output trace patterns: -module(z). -export([f/0]). g() -> 37. f() -> dbg:tracer(), %% Construct a tracer dbg:p(self(), call), %% Call tracing for ourselves enabled dbg:tpl(z, g, 0, cx), %% Set a tracepoint for a local function, enable caller/exception trace flags for it g(), dbg:stop_clear(). %% Get rid of the tracer again ; erl Eshell V7.0.3 (abort with ^G) 1> c(z). {ok,z} 2> z:f(). (<0.33.0>) call z:g() ({z,f,0}) (<0.33.0>) returned from z:g/0 -> 37 ok 3> though the dbg interface is unwieldy, it does provide a lot of power. A simpler and safer solution is Fred Hebert's recon library: f() -> recon_trace:calls({z, g, fun(_) -> return_trace() end}, 10, [{scope, local}]), g(), recon_trace:clear(). Simpler interface, but also less powerful. I tend to use both. The power is the dynamic tracing capabilities. You can enable them on production systems as you inspect them. If tracing is enabled through recompilation, you have to mess with the injected modules on the running system. Very good for debugging, but your operations department will hate you for the fact that they don't know which code is currently executing on their system. That said, however, a simpler debug-tracing interface is not a bad idea I think. Anything that can help programmers build robust code. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Tue Sep 22 01:43:29 2015 From: g@REDACTED (Garrett Smith) Date: Mon, 21 Sep 2015 18:43:29 -0500 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: <56005357.5020309@gmail.com> Message-ID: On Mon, Sep 21, 2015 at 4:15 PM, Jesper Louis Andersen wrote: > That said, however, a simpler debug-tracing interface is not a bad idea I > think. Anything that can help programmers build robust code. Once I started using tracing (again, using the super simple interface provided by e2_debug) I just can't go back to printfs. It's irritating that I can't trace *any* program now. This now feels like something as essential as a repl. From zxq9@REDACTED Tue Sep 22 02:18:49 2015 From: zxq9@REDACTED (zxq9) Date: Tue, 22 Sep 2015 09:18:49 +0900 Subject: [erlang-questions] -include with define macro In-Reply-To: References: <55FFE96B.5000303@gmail.com> <4760812.XdNxuBg2Ug@changa> Message-ID: <14443451.urRhYvTxFk@changa> On 2015?9?21? ??? 17:36:18 you wrote: > On Mon, Sep 21, 2015 at 1:56 PM, zxq9 wrote: > > > On 2015?9?21? ??? 14:26:35 Martin Koroudjiev wrote: > > > -ifdef(CONTAINING_APP_INCLUDE). > > > -include(?CONTAINING_APP_INCLUDE). > > > -endif. > > > > > > and to compile the source files as: > > > > > > erlc -Werror -Wall -DCONTAINING_APP_INCLUDE="\"/.hrl\"" > > > .erl > > > > > > but I get the following error: > > > > > > badly formed 'include' > > > > I think includes run before macros, so your include is trying to include a > > very odd-looking thing that starts with a question-mark instead of a > > filename. > > > > Not quite. The -include and -include_lib directives require that the thing > between the parentheses is a string literal, so the tokens '?' > 'CONTAINING_APP_INCLUDE' > are not even subject to macro expansion in this case. (This could probably > be easily fixed, if it is important to anyone.) This would certainly make macros more flexible for Martin's purpose and be more in the spirit of how one expects macros to work (coming from C, anyway). I think it wouldn't just be an issue of allowing macros in addition to string literals, though. It would require an extra pass to determine what includes are macro expansions, expand them, then perform the includes. After that continue on with normal macro expansion over the assembled source (included files often contain macros themselves). Not that this is difficult to do, but it is slightly more work than just allowing other tokens to appear in -include and -include_lib. -Craig From ok@REDACTED Tue Sep 22 04:05:29 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 22 Sep 2015 14:05:29 +1200 Subject: [erlang-questions] Erlang Cost Model In-Reply-To: <1442565243-sup-6508@arch.local> References: <20150917155645.GC9409@fhebert-ltm1> <2920189.nkzdrQtalV@changa> <1442565243-sup-6508@arch.local> Message-ID: On 18/09/2015, at 9:59 pm, Adam Krupicka wrote: >> Note that O(1) does not mean fast - it just means "constant" time >> without knowing *what* the constant is this information is useless. > > It's not completely useless. More than anything it tells you that this > function scales perfectly, independent on the amount of data you shovel > into the function. Of course, the definition of O is such that O(9999) = O(1). Joe has already hinted at this, but let me be explicit. Most of the data-structures-and-algorithms books give you cost formulas for data structures that are perfectly true for a PDP-6 or a Univac 1108 but not terribly realistic on today's machines. Let me give you an example from about 30 years ago. I was working at Quintus. The atom table for Quintus Prolog could hold 2^20 atoms. I wondered how much of a limit that was. So I wrote a little program like this: :- between(1, 1000000, N), (N mod 1000 =:= 0 -> write(N), nl ; true), name(N, Digits), name(_, [0'x|Digits]), fail. It ticked along at something like a thousand atoms per second, and then suddenly, the speed dropped to 6 atoms per second. Hey, it's a hash table! They are supposed to be O(1)! Well yes, it sort of was O(1), but the constant factor depended on whether the entire table fitted into your share of the 4 MiB memory or whether you were paging (over the network...). In *theoretical* terms, creating a new atom really was an O(1) constants, but in *practical* terms performance fell off such a steep cliff that it wasn't even a fast vs slow question but a fine vs don't even think about doing that question. Here's another example from a couple of years ago. I wanted to be able to compute the Singular Value Decompositions of some matrices. The department had bought me a thick book of numerical algorithms in Java and an accompanying CD-ROM. I decided to benchmark the code before committing to writing any serious amount of Java. The code was claimed to be "high quality". The performance was pretty bad. (1) Look at the code. Notice that it uses structures like for (col = 0; col < nCols; col++) { for (row = 0; row < nRows; row++) { use array[row][col]; } } Whoops! That's a great way to access an array that's stored in column-major (Fortran) order, but a lousy way to access an array stored in row-major (Pascal) order. Guess what Java uses? Switching things around to access arrays in the right order speeded the code up by a factor of TWO. (2) Rewrite the code in C. Another factor of TWO. (3) Rewrite the C code to use the level-1 BLAS, using the hardware vendor's library. Another factor of TWO. Now an enormous amount of labour had gone into transcribing an *old* library developed for old machines into a new language for new machines. But the *cost model* of the old language -- namely that all array indexing is O(1) -- wasn't even close to reality on new machines. Bearing in mind things like cache conflicts, lock contention, inter-core communication costs, I don't think our dear old data-structures-and-algorithms books are going to be much of a guide for Erlang programs. From jerome.debretagne@REDACTED Mon Sep 21 23:32:31 2015 From: jerome.debretagne@REDACTED (=?UTF-8?B?SsOpcsO0bWUgZGUgQnJldGFnbmU=?=) Date: Mon, 21 Sep 2015 23:32:31 +0200 Subject: [erlang-questions] Patch package OTP 18.0.3 released Message-ID: Hello, I guess this is a question for you, Henrik, but maybe someone else will have an idea as well. The maint-18 branch on GitHub contains the commits up to the OTP-18.0.2 tag, but not the more recent ones up to OTP-18.0.3. Is it a simple oversight on GitHub or am I making a wrong assumption about this branch? I am asking the question as the doc at https://github.com/erlang/otp/wiki/Branches explains the usage of master and maint, but there is no mention of maint-17 or maint-18. Thanks, J?r?me -------------- next part -------------- An HTML attachment was scrubbed... URL: From askjuise@REDACTED Tue Sep 22 10:29:25 2015 From: askjuise@REDACTED (Alexander Petrovsky) Date: Tue, 22 Sep 2015 11:29:25 +0300 Subject: [erlang-questions] Binary constantly growing on erlang 18 In-Reply-To: References: Message-ID: Hello, Does this commit fix the problem ? https://github.com/erlang/otp/commit/d2bb1ca56dcd0ab8e5b43c4368409317c3ddd4db ? 2015-08-18 17:14 GMT+03:00 Lukas Larsson : > Hello, > > Erlang 18.0 seems to have introduced a memory leak in the tty driver for > unix platforms. This means that when printing to stdout the binary usage > will grow. I believe I have found a solution to the problem. > > If you have problems with this leak, using -oldshell removes the leak. > > Thanks for reporting this issue. > > Lukas > > On Tue, Aug 18, 2015 at 2:18 PM, Alexander Petrovsky > wrote: > >> Hi! >> >> A few days ago a noticed that erlang vm eat too many memory. I've checked >> system and binary memory and found that it is constantly growing. I'm start >> debugging with recon but without success. After disabling all application >> the memory sill growing. >> >> I've checken on clean vm: >> >> 1> application:which_applications(). >> [{stdlib,"ERTS CXC 138 10","2.5"}, >> {kernel,"ERTS CXC 138 10","4.0"}] >> >> 2> [begin io:format("~p~n", [erlang:memory(binary)]), timer:sleep(1000) >> end || _ <- lists:seq(1, 5)]. >> 2105616 >> 2113976 >> 2122144 >> 2130504 >> 2138864 >> >> 3> io:format("~p~n", [erlang:memory(binary)]), timer:sleep(5000), >> io:format("~p~n", [erlang:memory(binary)]). >> 3124360 >> 3132720 >> >> 4> 2138864 - 2105616. >> 33248 >> >> 5> 3132720 - 3124360. >> 8360 >> >> As you can see, the binary grow non linear and only when I call >> erlang:memory(binary). >> >> System: Debian 7 >> Erlang: 18.0 esl-erlang >> >> Can anyone help me? >> >> -- >> ?????????? ????????? / Alexander Petrovsky, >> >> Skype: askjuise >> Phone: +7 914 8 820 815 >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Tue Sep 22 10:41:17 2015 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 22 Sep 2015 10:41:17 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: <6DC236FB-01AF-49C7-8432-39B3199AEDED@kent.ac.uk> References: <6DC236FB-01AF-49C7-8432-39B3199AEDED@kent.ac.uk> Message-ID: On Mon, Sep 21, 2015 at 10:08 PM, Simon Thompson wrote: > > I see that discussions have already aired filtering, and I think that the > main lesson for me from RELEASE was it?s crucial to find ways to deal with > torrents of potential data. It?s no good to generate trace info which is > filtered out downstream: if it can be filtered at source, so only generated > when required, then that would be great. > Filtering close to the source is indeed important. My hope is that if you can implement the tracer as a nif, you can in the backend inspect the data that is to be outputted without having to copy it anywhere and from there make smarter decisions on what you actually want to trace on. We still need match specs to do the initial filtering from erlang, but for those of us having to deal with very specific tracing scenarios we could write a specialized backend that takes care of the filtering. Unfortunately I have not yet come up with a way to make the tracer backend be a pure erlang module and still be fast. > Another facility which may be problematic to implement, but which could be > really helpful, would be mechanisms to aggregate information as part of the > BEAM: e.g. to be able to emit a trace message every time 1000 messages have > passed from process A to process B, or node A to node B. Again this would > push into the infrastructure something that currently needs to be done in > trace processing. > My hope is again that the more flexible nif tracer backends will help in creating sampling profilers. If for instance you want to only trace the stack of every 1000th call, you would just have a counter in the nif backend and extract the stack only if you need it. There will of course be some overhead, but it will be minuscule compared to what there is today. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Tue Sep 22 10:46:08 2015 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 22 Sep 2015 10:46:08 +0200 Subject: [erlang-questions] Binary constantly growing on erlang 18 In-Reply-To: References: Message-ID: yes it does. The fix was released in 18.0.3 I believe. On Tue, Sep 22, 2015 at 10:29 AM, Alexander Petrovsky wrote: > Hello, > > Does this commit fix the problem ? > https://github.com/erlang/otp/commit/d2bb1ca56dcd0ab8e5b43c4368409317c3ddd4db > ? > > 2015-08-18 17:14 GMT+03:00 Lukas Larsson : > >> Hello, >> >> Erlang 18.0 seems to have introduced a memory leak in the tty driver for >> unix platforms. This means that when printing to stdout the binary usage >> will grow. I believe I have found a solution to the problem. >> >> If you have problems with this leak, using -oldshell removes the leak. >> >> Thanks for reporting this issue. >> >> Lukas >> >> On Tue, Aug 18, 2015 at 2:18 PM, Alexander Petrovsky >> wrote: >> >>> Hi! >>> >>> A few days ago a noticed that erlang vm eat too many memory. I've >>> checked system and binary memory and found that it is constantly growing. >>> I'm start debugging with recon but without success. After disabling all >>> application the memory sill growing. >>> >>> I've checken on clean vm: >>> >>> 1> application:which_applications(). >>> [{stdlib,"ERTS CXC 138 10","2.5"}, >>> {kernel,"ERTS CXC 138 10","4.0"}] >>> >>> 2> [begin io:format("~p~n", [erlang:memory(binary)]), timer:sleep(1000) >>> end || _ <- lists:seq(1, 5)]. >>> 2105616 >>> 2113976 >>> 2122144 >>> 2130504 >>> 2138864 >>> >>> 3> io:format("~p~n", [erlang:memory(binary)]), timer:sleep(5000), >>> io:format("~p~n", [erlang:memory(binary)]). >>> 3124360 >>> 3132720 >>> >>> 4> 2138864 - 2105616. >>> 33248 >>> >>> 5> 3132720 - 3124360. >>> 8360 >>> >>> As you can see, the binary grow non linear and only when I call >>> erlang:memory(binary). >>> >>> System: Debian 7 >>> Erlang: 18.0 esl-erlang >>> >>> Can anyone help me? >>> >>> -- >>> ?????????? ????????? / Alexander Petrovsky, >>> >>> Skype: askjuise >>> Phone: +7 914 8 820 815 >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > > > -- > ?????????? ????????? / Alexander Petrovsky, > > Skype: askjuise > Phone: +7 914 8 820 815 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From askjuise@REDACTED Tue Sep 22 10:47:04 2015 From: askjuise@REDACTED (Alexander Petrovsky) Date: Tue, 22 Sep 2015 11:47:04 +0300 Subject: [erlang-questions] Binary constantly growing on erlang 18 In-Reply-To: References: Message-ID: Thanks! 2015-09-22 11:46 GMT+03:00 Lukas Larsson : > yes it does. The fix was released in 18.0.3 I believe. > > On Tue, Sep 22, 2015 at 10:29 AM, Alexander Petrovsky > wrote: > >> Hello, >> >> Does this commit fix the problem ? >> https://github.com/erlang/otp/commit/d2bb1ca56dcd0ab8e5b43c4368409317c3ddd4db >> ? >> >> 2015-08-18 17:14 GMT+03:00 Lukas Larsson : >> >>> Hello, >>> >>> Erlang 18.0 seems to have introduced a memory leak in the tty driver for >>> unix platforms. This means that when printing to stdout the binary usage >>> will grow. I believe I have found a solution to the problem. >>> >>> If you have problems with this leak, using -oldshell removes the leak. >>> >>> Thanks for reporting this issue. >>> >>> Lukas >>> >>> On Tue, Aug 18, 2015 at 2:18 PM, Alexander Petrovsky >> > wrote: >>> >>>> Hi! >>>> >>>> A few days ago a noticed that erlang vm eat too many memory. I've >>>> checked system and binary memory and found that it is constantly growing. >>>> I'm start debugging with recon but without success. After disabling all >>>> application the memory sill growing. >>>> >>>> I've checken on clean vm: >>>> >>>> 1> application:which_applications(). >>>> [{stdlib,"ERTS CXC 138 10","2.5"}, >>>> {kernel,"ERTS CXC 138 10","4.0"}] >>>> >>>> 2> [begin io:format("~p~n", [erlang:memory(binary)]), timer:sleep(1000) >>>> end || _ <- lists:seq(1, 5)]. >>>> 2105616 >>>> 2113976 >>>> 2122144 >>>> 2130504 >>>> 2138864 >>>> >>>> 3> io:format("~p~n", [erlang:memory(binary)]), timer:sleep(5000), >>>> io:format("~p~n", [erlang:memory(binary)]). >>>> 3124360 >>>> 3132720 >>>> >>>> 4> 2138864 - 2105616. >>>> 33248 >>>> >>>> 5> 3132720 - 3124360. >>>> 8360 >>>> >>>> As you can see, the binary grow non linear and only when I call >>>> erlang:memory(binary). >>>> >>>> System: Debian 7 >>>> Erlang: 18.0 esl-erlang >>>> >>>> Can anyone help me? >>>> >>>> -- >>>> ?????????? ????????? / Alexander Petrovsky, >>>> >>>> Skype: askjuise >>>> Phone: +7 914 8 820 815 >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> >> >> -- >> ?????????? ????????? / Alexander Petrovsky, >> >> Skype: askjuise >> Phone: +7 914 8 820 815 >> >> > -- ?????????? ????????? / Alexander Petrovsky, Skype: askjuise Phone: +7 914 8 820 815 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Sep 22 12:47:38 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 22 Sep 2015 12:47:38 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: On Mon, Sep 21, 2015 at 10:52 AM, Lukas Larsson wrote: > * Allow multiple tracers. Today only one port/process can be the receiver > of trace data. > This would be especially welcome. You can run profiling next to debug-tracing then, which is a nice thing. > * Better integration of dtrace/lttngt/systemtap into the erlang trace. > My advice would be: Study the DTrace model. Steal every good idea. In DTrace, you enable probes, then define a filter set on those probes and then you define what action (i.e., program) to run if the filter is passed. Example: People have requested a "who calls me" query. Suppose we had a probe in DTrace for when a function calls another function, in module 'm' erlang*:::funcall / copyinstr(arg1) = "m" / { caller = current_function() } erlang*:::global-function-enry { @callers[caller] = count() } This would in principle be enough to satisfy that query, if only we extended DTrace probes a little bit. It would output a list of pairs [{Caller, Count}]. Second observation: DTrace has something called a ustack(), which is the backtrace stack in Userland. We can get that for the C layer in BEAM, but we can't look into it for the Erlang world. The ustack() helpers are D-scripts (they run in the kernel, and can only have finite loops among other things). So we would probably need an easy way, in a struct, where we can grab information about the frames on the emulator stack from the kernel. Any tracing facility outside the Erlang world needs this. Third observation: Simon's request is also easily satisfiable. Set a tick timer at 997Hz from the kernel, then look up where we are once the timer ticks. Preferably once per scheduler thread. Key takeaway: Provide tools, not solutions. Once you can enable probes and filter probes, a process can tie actions to probes like the above. We can always push specialized optimizations down into the tracing layer for efficiency, but we should go for a variant where we provide enabling tools for programmers, rather than baked solutions. I envision that things such as e2_debug, recon_trace, redbug and so on are adaptable to a new model, easily. Finally, we should really write a good dbg tutorial together with better documentation. I'm willing to throw some hours into this if needed :) -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Sep 22 13:33:38 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 22 Sep 2015 13:33:38 +0200 Subject: [erlang-questions] Patch package OTP 18.0.3 released In-Reply-To: References: Message-ID: On Mon, Sep 21, 2015 at 11:32 PM, J?r?me de Bretagne < jerome.debretagne@REDACTED> wrote: > The maint-18 branch on GitHub contains the commits up to the OTP-18.0.2 > tag, but not the more recent ones up to OTP-18.0.3. Is it a simple > oversight on GitHub or am I making a wrong assumption about this branch? It just hasn't been moved yet, and I think it is an oversight. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From henrik.x.nord@REDACTED Tue Sep 22 14:13:03 2015 From: henrik.x.nord@REDACTED (Henrik Nord X) Date: Tue, 22 Sep 2015 14:13:03 +0200 Subject: [erlang-questions] Patch package OTP 18.0.3 released In-Reply-To: References: Message-ID: <560145CF.5060901@ericsson.com> Hi Actually, we decided to not push the maint-XX branches at all. But we have sense re-evaluated that decision and should push them again as there was some usage on the branches after all. The missing tag and the new tag OTP-18.1 will be there tomorrow. On 09/22/2015 01:33 PM, Jesper Louis Andersen wrote: > > On Mon, Sep 21, 2015 at 11:32 PM, J?r?me de Bretagne > > wrote: > > The maint-18 branch on GitHub contains the commits up to the > OTP-18.0.2 tag, but not the more recent ones up to OTP-18.0.3. Is > it a simple oversight on GitHub or am I making a wrong assumption > about this branch? > > > It just hasn't been moved yet, and I think it is an oversight. > > > -- > J. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From samuelrivas@REDACTED Tue Sep 22 15:34:19 2015 From: samuelrivas@REDACTED (Samuel) Date: Tue, 22 Sep 2015 15:34:19 +0200 Subject: [erlang-questions] user process suspended Message-ID: Hi, We are experiencing problems were an erlang service stops receiving HTTP requests while still alive and responding to remote_shell. When this happens cpu usage drops and memory grows slowly and steadily, I suspect cpu just goes down as the node stops getting requests and memory goes up because some mailboxes keep growing. We have been collecting crashdumps for all thos instances and all of them show the same process pattern: the user process is suspended while all others are waiting. Again I suspect that all of them are waiting simply because there is not a lot of activity in the node so I am inclined to think that the suspended user process is probably the actual issue. We can remote shell into those nodes and we have tested that we can create and write files, so IO is not completely blocked. This is under R16B03. Any hints on what could be the cause and/or how to debug this problem? Cheers -- Samuel From garry@REDACTED Tue Sep 22 16:03:21 2015 From: garry@REDACTED (Garry Hodgson) Date: Tue, 22 Sep 2015 10:03:21 -0400 Subject: [erlang-questions] "calling" a gen_fsm Message-ID: <56015FA9.7040701@research.att.com> i'm using a gen_fsm to implement the client side of a binary protocol. clients connect to a controller, do a login thing, then settle into connected state. so far, so good. when in connected state, i need to be able to query the fsm for information. some of these queries require sending a message to a controller and waiting for a response to come back. herein lies the problem, because the query relies on the sync event to ask the question, but must wait for an { ssl, Socket, Data } event to get the answer. i've got it sort of working, but only by intercepting the ssl message inline, vs. the normal handle_info() method. this feels like cheating, and seems fragile, as it relies on there being no intervening messages. so i'm wondering, is there a good/standard way of accomplishing this? or is gen_fsm not the right approach? any advice would be appreciated. thanks code snippets follow: who_provides( Service ) -> gen_fsm:sync_send_event( ?MODULE, { request_services, Service } ). ... connected( { request_services, Service }, _From, #state{ socket=Socket, transport=Transport } = State ) -> Transport:send( Socket, service_request( Service ) ), case wait_for_services( Socket ) of { ok, Services } -> { reply, { ok, Services }, connected, State }; Other -> { reply, { error, Other }, connected, State } end. wait_for_services( Socket ) -> receive { ssl, Socket, Data } -> ssl:setopts( Socket, [ { active, once } ] ), { ok, parse( Data ) } after 30000 -> timeout end. -- Garry Hodgson Lead Member of Technical Staff AT&T Chief Security Office (CSO) "This e-mail and any files transmitted with it are AT&T property, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited." From vasdeveloper@REDACTED Tue Sep 22 16:07:41 2015 From: vasdeveloper@REDACTED (Theepan) Date: Tue, 22 Sep 2015 19:37:41 +0530 Subject: [erlang-questions] user process suspended In-Reply-To: References: Message-ID: The first step should be to check if the server is still listening on the assigned port (may be 80). You can check the mailbox length of relevant processes to confirm if the mailbox sizes are increasing, using the process information API. If you are handling gen_tcp your self instead of using a http library, check how you have configured the 'active' option. {active, once} can sometimes be a culprit, if you don't do it right. Theepan On Tue, Sep 22, 2015 at 7:04 PM, Samuel wrote: > Hi, > > We are experiencing problems were an erlang service stops receiving > HTTP requests while still alive and responding to remote_shell. When > this happens cpu usage drops and memory grows slowly and steadily, I > suspect cpu just goes down as the node stops getting requests and > memory goes up because some mailboxes keep growing. > > We have been collecting crashdumps for all thos instances and all of > them show the same process pattern: the user process is suspended > while all others are waiting. Again I suspect that all of them are > waiting simply because there is not a lot of activity in the node so I > am inclined to think that the suspended user process is probably the > actual issue. > > We can remote shell into those nodes and we have tested that we can > create and write files, so IO is not completely blocked. > > This is under R16B03. > > Any hints on what could be the cause and/or how to debug this problem? > > Cheers > > -- > Samuel > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Tue Sep 22 16:18:32 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 22 Sep 2015 10:18:32 -0400 Subject: [erlang-questions] user process suspended In-Reply-To: References: Message-ID: <20150922141831.GJ9409@fhebert-ltm1> On 09/22, Samuel wrote: >We have been collecting crashdumps for all thos instances and all of >them show the same process pattern: the user process is suspended >while all others are waiting. Again I suspect that all of them are >waiting simply because there is not a lot of activity in the node so I >am inclined to think that the suspended user process is probably the >actual issue. > >We can remote shell into those nodes and we have tested that we can >create and write files, so IO is not completely blocked. > I've seen similar issues to this on EC2 nodes on AWS that were outputting content whenever the host instance had major issues (randomly going bad on disk or whatever), which we more or less chalked up to be issues with "the cloud being the cloud". We would be logging a lot of content out and it could suddenly stall entire nodes. The workaround we had for it was to move all of our logging output to an asynchronous mechanism with buffering: https://github.com/ferd/batchio The thing is marked experimental, but has been at the core of logging for logplex (https://github.com/heroku/logplex) for a couple of years now without a problem. The big downside you could see from it would be that when the user process can't deal with content anymore, it will drop output messages on the floor and just shed load. Mind you that this may only help if you can notice significant amount of outputs that could explain the IO stuff stalling. If not, the problem could very well be elsewhere. Regards, Fred. From vances@REDACTED Tue Sep 22 16:21:31 2015 From: vances@REDACTED (Vance Shipley) Date: Tue, 22 Sep 2015 19:51:31 +0530 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: <56015FA9.7040701@research.att.com> References: <56015FA9.7040701@research.att.com> Message-ID: Send your request, stuff From into StateData and return next_state. Then you may receive the response in handle_info/3 and send it to the client with reply/2. On Sep 22, 2015 7:33 PM, "Garry Hodgson" wrote: > > i'm using a gen_fsm to implement the client side of > a binary protocol. clients connect to a controller, do > a login thing, then settle into connected state. so far, > so good. > > when in connected state, i need to be able to query > the fsm for information. some of these queries require > sending a message to a controller and waiting for a > response to come back. herein lies the problem, because > the query relies on the sync event to ask the question, > but must wait for an { ssl, Socket, Data } event to get > the answer. > > i've got it sort of working, but only by intercepting the > ssl message inline, vs. the normal handle_info() method. > this feels like cheating, and seems fragile, as it relies on > there being no intervening messages. > > so i'm wondering, is there a good/standard way of > accomplishing this? or is gen_fsm not the right approach? > any advice would be appreciated. > > thanks > > code snippets follow: > > who_provides( Service ) -> gen_fsm:sync_send_event( ?MODULE, { request_services, Service } ). > ... > connected( { request_services, Service }, _From, #state{ socket=Socket, transport=Transport } = State ) -> > Transport:send( Socket, service_request( Service ) ), > case wait_for_services( Socket ) of > { ok, Services } -> > { reply, { ok, Services }, connected, State }; > Other -> > { reply, { error, Other }, connected, State } > end. > > wait_for_services( Socket ) -> > receive > { ssl, Socket, Data } -> > ssl:setopts( Socket, [ { active, once } ] ), > { ok, parse( Data ) } > after 30000 -> > timeout > end. > > > -- > Garry Hodgson > Lead Member of Technical Staff > AT&T Chief Security Office (CSO) > > "This e-mail and any files transmitted with it are AT&T property, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited." > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Tue Sep 22 16:22:59 2015 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Tue, 22 Sep 2015 10:22:59 -0400 Subject: [erlang-questions] Add a field to a record In-Reply-To: References: <1442528164.751430151@apps.rackspace.com> Message-ID: Many thanks, Michael and Marc! Another useful brick in my ever so slowly growing edifice of Erlang lore. All the best, Lloyd Sent from my iPad > On Sep 18, 2015, at 4:20 AM, Marc Worrell wrote: > > Hi Lloyd, > > That is a very common situation. > > We have two tactics to handle that: > > 1) Just let it crash and then restart with the new definition > 2) If (1) is not acceptable, use module/function that can ?upgrade? your record. > > Example of #2: > > upgrade(#anything{} = Rec) -> Rec; > upgrade({anything, Field1, Field2}) -> #anything{field1=Field1, field2=Field2}. > > And then, if you receive a record from old(er) code/nodes or a backing store, first ?upgrade? your record before you do anything with it. > > do_something(Rec0) -> > Rec = upgrade(Rec0), > ? your code with Rec ? > > - Marc > > >> On 18 sep. 2015, at 00:16, lloyd@REDACTED wrote: >> >> Hello, >> >> I understand that it would be better to use maps if you anticipate adding fields to a record. >> >> But suppose you didn't anticipate and you've been merrily adding data to your db when you discover--- oops! I need another field. >> >> e.g. You have: >> -record anything, { >> field1, >> field2 >> } >> >> and you discover that you really need: >> -record anything, { >> field1, >> new_field, >> field2 >> } >> >> Is there a cunning trick or technique to make the change without losing data? >> >> Thanks, >> >> LRP >> >> >> ********************************************* >> My books: >> >> THE GOSPEL OF ASHES >> http://thegospelofashes.com >> >> Strength is not enough. Do they have the courage >> and the cunning? Can they survive long enough to >> save the lives of millions? >> >> FREEIN' PANCHO >> http://freeinpancho.com >> >> A community of misfits help a troubled boy find his way >> >> AYA TAKEO >> http://ayatakeo.com >> >> Star-crossed love, war and power in an alternative >> universe >> >> Available through Amazon or by request from your >> favorite bookstore >> >> >> ********************************************** >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From samuelrivas@REDACTED Tue Sep 22 16:24:26 2015 From: samuelrivas@REDACTED (Samuel) Date: Tue, 22 Sep 2015 16:24:26 +0200 Subject: [erlang-questions] user process suspended In-Reply-To: References: Message-ID: > The first step should be to check if the server is still listening on the > assigned port (may be 80). It is, but the connections timeout. I haven't traced that yet, but I suspect the processes don't get any message from the outside, as they don't have them in the inbox when dropping the crashdump. > You can check the mailbox length of relevant processes to confirm if the > mailbox sizes are increasing, using the process information API. The mailboxes increasing are lager_event (faster) and error_logger (slower). Most processes have empty inboxes. > If you are handling gen_tcp your self instead of using a http library, check > how you have configured the 'active' option. {active, once} can sometimes be > a culprit, if you don't do it right. We use webmachine/mochiweb. I haven't traced that part yet as it seems to be a more generic problem since other processes stop writing to files too. Best -- Samuel From samuelrivas@REDACTED Tue Sep 22 16:33:12 2015 From: samuelrivas@REDACTED (Samuel) Date: Tue, 22 Sep 2015 16:33:12 +0200 Subject: [erlang-questions] user process suspended In-Reply-To: <20150922141831.GJ9409@fhebert-ltm1> References: <20150922141831.GJ9409@fhebert-ltm1> Message-ID: >> We have been collecting crashdumps for all thos instances and all of >> them show the same process pattern: the user process is suspended >> while all others are waiting. Again I suspect that all of them are >> waiting simply because there is not a lot of activity in the node so I >> am inclined to think that the suspended user process is probably the >> actual issue. >> >> We can remote shell into those nodes and we have tested that we can >> create and write files, so IO is not completely blocked. >> > > I've seen similar issues to this on EC2 nodes on AWS that were outputting > content whenever the host instance had major issues (randomly going bad on > disk or whatever), which we more or less chalked up to be issues with "the > cloud being the cloud". These nodes are running in virtual machines too, but in our own Vmware servers. It could be related though. > We would be logging a lot of content out and it could suddenly stall entire > nodes. The workaround we had for it was to move all of our logging output to > an asynchronous mechanism with buffering: We do log a lot too. However most of the logging goes through an internal queue implemented on top of disk_log to prevent transient io issues to backpressure all the chain back. We are starting to suspect that it is the lager console_backed that hangs in a send to the io port. We have just disabled it and we'll see if it helps. > https://github.com/ferd/batchio > > The thing is marked experimental, but has been at the core of logging for > logplex (https://github.com/heroku/logplex) for a couple of years now > without a problem. > > The big downside you could see from it would be that when the user process > can't deal with content anymore, it will drop output messages on the floor > and just shed load. Thanks. I believe we have code to do similar things in our system, but I'll take a closer look > Mind you that this may only help if you can notice significant amount of > outputs that could explain the IO stuff stalling. If not, the problem could > very well be elsewhere. What do you mean with IO stuff stalling? Temporarily or permanently? We do send quite some io, but the systems stops to io when this happens and never recovers (we need to restart the node). -- Samuel From mononcqc@REDACTED Tue Sep 22 16:42:58 2015 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 22 Sep 2015 10:42:58 -0400 Subject: [erlang-questions] user process suspended In-Reply-To: References: <20150922141831.GJ9409@fhebert-ltm1> Message-ID: <20150922144257.GK9409@fhebert-ltm1> On 09/22, Samuel wrote: > >What do you mean with IO stuff stalling? Temporarily or permanently? >We do send quite some io, but the systems stops to io when this >happens and never recovers (we need to restart the node). > It's been a while, but I do believe it was temporary for us. Like the EC2 instances output would drop from say 10MB/s capacity down to 150kb/s for blips of anywhere between 2 and 10 minutes. The degradation in performance all but stalled everything we had that kept expecting multiple MB/s of output to be possible. From vasdeveloper@REDACTED Tue Sep 22 16:47:49 2015 From: vasdeveloper@REDACTED (Theepan) Date: Tue, 22 Sep 2015 20:17:49 +0530 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: <56015FA9.7040701@research.att.com> References: <56015FA9.7040701@research.att.com> Message-ID: Just use {next_state,NextStateName,NewStateData} instead of {reply,Reply,NextStateName,NewStateData} and keep the PID of the calling process in the NewStateData. This helps you in deferring the response. When the data comes form the service, you can send the response back with gen_fsm:reply(PID, Reply). Theepan On Tue, Sep 22, 2015 at 7:33 PM, Garry Hodgson wrote: > i'm using a gen_fsm to implement the client side of > a binary protocol. clients connect to a controller, do > a login thing, then settle into connected state. so far, > so good. > > when in connected state, i need to be able to query > the fsm for information. some of these queries require > sending a message to a controller and waiting for a > response to come back. herein lies the problem, because > the query relies on the sync event to ask the question, > but must wait for an { ssl, Socket, Data } event to get > the answer. > > i've got it sort of working, but only by intercepting the > ssl message inline, vs. the normal handle_info() method. > this feels like cheating, and seems fragile, as it relies on > there being no intervening messages. > > so i'm wondering, is there a good/standard way of > accomplishing this? or is gen_fsm not the right approach? > any advice would be appreciated. > > thanks > > code snippets follow: > > who_provides( Service ) -> gen_fsm:sync_send_event( ?MODULE, { > request_services, Service } ). > ... > connected( { request_services, Service }, _From, #state{ socket=Socket, > transport=Transport } = State ) -> > Transport:send( Socket, service_request( Service ) ), > case wait_for_services( Socket ) of > { ok, Services } -> > { reply, { ok, Services }, connected, State }; > Other -> > { reply, { error, Other }, connected, State } > end. > > wait_for_services( Socket ) -> > receive > { ssl, Socket, Data } -> > ssl:setopts( Socket, [ { active, once } ] ), > { ok, parse( Data ) } > after 30000 -> > timeout > end. > > > -- > Garry Hodgson > Lead Member of Technical Staff > AT&T Chief Security Office (CSO) > > "This e-mail and any files transmitted with it are AT&T property, are > confidential, and are intended solely for the use of the individual or > entity to whom this e-mail is addressed. If you are not one of the named > recipient(s) or otherwise have reason to believe that you have received > this message in error, please notify the sender and delete this message > immediately from your computer. Any other use, retention, dissemination, > forwarding, printing, or copying of this e-mail is strictly prohibited." > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhuo.quan@REDACTED Tue Sep 22 15:34:58 2015 From: zhuo.quan@REDACTED (=?utf-8?B?5YWo5Y2T?=) Date: Tue, 22 Sep 2015 21:34:58 +0800 Subject: [erlang-questions] =?utf-8?b?562U5aSNOiAgbXVsdGkgYWRhcHRvciBwcm9i?= =?utf-8?q?lem=2E?= Message-ID: It fixed the problem for me too. Thanks! Quan Zhuo. ???: Lo?c Hoguin ????: 2015?9?21? 14:54 ???: ??;Erlang???? ??: Re: [erlang-questions] multi adaptor problem. Check if the name of your computer is listed in your hosts file. I experienced the same as you and adding 'alice' (my computer name) on the 127.0.0.1 line in /etc/hosts fixed it for me. On 09/21/2015 03:46 AM, ?? wrote: > Hi,list: > >??? My computer has 2 adaptor, one local adaptor(ip:10.206.25.230) and > another USB Wireless LAN Card(ip:192.168.1.100). Once I turned the USB > Wireless LAN Card on, it will take a long time(about 5 seconds) to start > the erlang vm when I type ?erl? in the terminal. However, it will behave > normal once I turn the wireless card off. > >??? Anyone help? > >??? The OS is fedora 22,and OTP version 17, erts 6.3. > > Thanks > > Quan Zhuo > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin http://ninenines.eu Author of The Erlanger Playbook, A book about software development using Erlang -------------- next part -------------- An HTML attachment was scrubbed... URL: From garry@REDACTED Tue Sep 22 19:37:25 2015 From: garry@REDACTED (Garry Hodgson) Date: Tue, 22 Sep 2015 13:37:25 -0400 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: References: <56015FA9.7040701@research.att.com> Message-ID: <560191D5.6050905@research.att.com> On 9/22/15 10:21 AM, Vance Shipley wrote: > > Send your request, stuff From into StateData and return next_state. > Then you may receive the response in handle_info/3 and send it to the > client with reply/2. > On 9/22/15 10:47 AM, Theepan wrote: > Just use {next_state,NextStateName,NewStateData} instead of > {reply,Reply,NextStateName,NewStateData} and keep the PID of the > calling process in the NewStateData. This helps you in deferring the > response. When the data comes form the service, you can send the > response back with gen_fsm:reply(PID, Reply). > thanks to both of you for your quick and helpful answers. gotta love this list. From roger@REDACTED Tue Sep 22 21:02:43 2015 From: roger@REDACTED (Roger Lipscombe) Date: Tue, 22 Sep 2015 20:02:43 +0100 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: <560191D5.6050905@research.att.com> References: <56015FA9.7040701@research.att.com> <560191D5.6050905@research.att.com> Message-ID: On 22 September 2015 at 18:37, Garry Hodgson wrote: > thanks to both of you for your quick and helpful answers. > gotta love this list. Both Theepan and Vance are correct in that this is a useful way to build this kind of thing. In fact, there's a good example (using a gen_server, rather than a gen_fsm) here: https://erlangcentral.org/wiki/index.php?title=Building_Non_Blocking_Erlang_apps I'd just add one point, though: I recently built something similar, and I found it simpler to separate the client into two processes: one gen_fsm to handle the state machine, and a separate gen_server to handle the decoding and framing. From federico.carrone@REDACTED Tue Sep 22 21:53:53 2015 From: federico.carrone@REDACTED (Federico Carrone) Date: Tue, 22 Sep 2015 19:53:53 +0000 Subject: [erlang-questions] [ANN] Spawned Shelter In-Reply-To: References: Message-ID: Thanks for the comments! The wesbsite got a few thousands visits in only a few days. Don't hesitate to create an issue or send a PR if you want to add or remove anything: https://github.com/unbalancedparentheses/spawnedshelter Thanks, Federico Carrone. On Fri, Sep 18, 2015 at 4:02 PM Federico Carrone wrote: > > I have created a simple website called Spawned Shelter ( > http://spawnedshelter.com/). > > The website tries to collect the best articles, videos and books for > learning Erlang, the internals of its virtual machine and other languages > that run on top of it. I think that it can be quite useful for the > community to have a simple website like this for sending out to people who > want to learn Erlang or are learning it. > > If you want to add or remove any link you only have to send a pull request > that changes this markdown file: > > https://github.com/unbalancedparentheses/spawnedshelter/blob/master/src/content.md > > Also if you want to help in sharing it to the rest of the wolrd, you can > vote it in reddit ( > https://www.reddit.com/r/programming/comments/3lgg6v/spawned_shelter_the_best_articles_videos_and/) > or hacker news (https://news.ycombinator.com/item?id=10240572). > > Let me know if you have any recommadation or comment. > > Regards, > Federico. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Sep 23 04:13:51 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 23 Sep 2015 14:13:51 +1200 Subject: [erlang-questions] distinguish proplists vs. lists of proplists In-Reply-To: References: <429653016.190925.1442336313342.JavaMail.yahoo@mail.yahoo.com> <1442339510-sup-9669@arch.local> <7FB5964B-A263-43D0-A656-2D483CF3ED5E@redops.org> <2420C084-E494-4234-8D11-B7580542738C@cs.otago.ac.nz> Message-ID: On 18/09/2015, at 8:57 pm, Richard Carlsson wrote: [some very helpful background about proplists] I for one would find it useful if information like that were present in the module manuals. For example, it had seemed perfectly obvious to me that if I was looking for 'x' and a proplist contained {'x',1,"too"} I should get the answer 1. My reaction to _not_ getting that was unwillingness to use proplists because the interface was too confusing. With the background in the manual, it would not have been confusing any more. From jerome.debretagne@REDACTED Tue Sep 22 23:41:57 2015 From: jerome.debretagne@REDACTED (=?UTF-8?B?SsOpcsO0bWUgZGUgQnJldGFnbmU=?=) Date: Tue, 22 Sep 2015 23:41:57 +0200 Subject: [erlang-questions] Patch package OTP 18.0.3 released In-Reply-To: <560145CF.5060901@ericsson.com> References: <560145CF.5060901@ericsson.com> Message-ID: Thanks to both of you for your quick feedback, Jesper and Henrik! J?r?me Le 22 sept. 2015 2:13 PM, "Henrik Nord X" a ?crit : > Hi > > Actually, we decided to not push the maint-XX branches at all. > But we have sense re-evaluated that decision and should push them again as > there was some usage on the branches after all. > > The missing tag and the new tag OTP-18.1 will be there tomorrow. > > > > On 09/22/2015 01:33 PM, Jesper Louis Andersen wrote: > > > On Mon, Sep 21, 2015 at 11:32 PM, J?r?me de Bretagne < > jerome.debretagne@REDACTED> wrote: > >> The maint-18 branch on GitHub contains the commits up to the OTP-18.0.2 >> tag, but not the more recent ones up to OTP-18.0.3. Is it a simple >> oversight on GitHub or am I making a wrong assumption about this branch? > > > It just hasn't been moved yet, and I think it is an oversight. > > > -- > J. > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From garry@REDACTED Wed Sep 23 06:01:07 2015 From: garry@REDACTED (Garry Hodgson) Date: Wed, 23 Sep 2015 00:01:07 -0400 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: References: <56015FA9.7040701@research.att.com> <560191D5.6050905@research.att.com> Message-ID: <56022403.9020702@research.att.com> On 9/22/15 3:02 PM, Roger Lipscombe wrote: > On 22 September 2015 at 18:37, Garry Hodgson wrote: >> thanks to both of you for your quick and helpful answers. >> gotta love this list. > Both Theepan and Vance are correct in that this is a useful way to > build this kind of thing. In fact, there's a good example (using a > gen_server, rather than a gen_fsm) here: > https://erlangcentral.org/wiki/index.php?title=Building_Non_Blocking_Erlang_apps very nice. > I'd just add one point, though: I recently built something similar, > and I found it simpler to separate the client into two processes: one > gen_fsm to handle the state machine, and a separate gen_server to > handle the decoding and framing. i briefly considered that, but it wasn't obvious how to divide the work between the two. i'd be interested in how you did, if you'd care to elaborate. thanks From vances@REDACTED Wed Sep 23 06:30:43 2015 From: vances@REDACTED (Vance Shipley) Date: Wed, 23 Sep 2015 10:00:43 +0530 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: References: <56015FA9.7040701@research.att.com> <560191D5.6050905@research.att.com> Message-ID: On Wed, Sep 23, 2015 at 12:32 AM, Roger Lipscombe wrote: > Both Theepan and Vance are correct in that this is a useful way to > build this kind of thing. In fact, there's a good example (using a > gen_server, rather than a gen_fsm) here: > https://erlangcentral.org/wiki/index.php?title=Building_Non_Blocking_Erlang_apps What I don't like about that example is that it's not purely functional. I would prefer: handle_call({request, Tr_id, Req}, From, #state{tr = Tr, socket = Socket} = State) -> Data = encode(Tr_id, Req), gen_tcp:send(Socket, Data), NewState = State#state{tr = dict:store(Tr_id, From, Tr)}, {noreply, NewState}. handle_info({tcp, _Socket, Data}, #state{tr = Tr} = State) -> {Tr_id, Result} = decode(Data), From = dict:fetch(Tr_id, Tr), gen_server:reply(From, Result), NewState = State#state{tr = dict:erase(From, Tr)}, {noreply, NewState}. -- -Vance From zxq9@REDACTED Wed Sep 23 06:31:24 2015 From: zxq9@REDACTED (zxq9) Date: Wed, 23 Sep 2015 13:31:24 +0900 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: <56022403.9020702@research.att.com> References: <56015FA9.7040701@research.att.com> <56022403.9020702@research.att.com> Message-ID: <4135987.0G3UQNkJij@changa> On 2015?9?23? ??? 00:01:07 Garry Hodgson wrote: > On 9/22/15 3:02 PM, Roger Lipscombe wrote: > > On 22 September 2015 at 18:37, Garry Hodgson wrote: > >> thanks to both of you for your quick and helpful answers. > >> gotta love this list. > > Both Theepan and Vance are correct in that this is a useful way to > > build this kind of thing. In fact, there's a good example (using a > > gen_server, rather than a gen_fsm) here: > > https://erlangcentral.org/wiki/index.php?title=Building_Non_Blocking_Erlang_apps > very nice. > > I'd just add one point, though: I recently built something similar, > > and I found it simpler to separate the client into two processes: one > > gen_fsm to handle the state machine, and a separate gen_server to > > handle the decoding and framing. > i briefly considered that, but it wasn't obvious how to divide the > work between the two. i'd be interested in how you did, if you'd > care to elaborate. When I have two identical processes that need to speak synchronously or when I have something like a gen_fsm that must remain responsive while something other activity it initiates is carried out I tend to spawn_link or spawn_monitor a one-off process to do the work and let it exit(normal) when the task is done. This sidesteps some obvious deadlocks in synch communications and lets an fsm control flow without getting tied down in execution. To use the fsm movie ticket counter example, the ticket counter accepts tickets or not based on its state, but it isn't concerned with the verification of the tickets or what the newly authorized folks do once they enter. Those extended tasks can be independent processes. If they are their own conceptual tasks (people entering the theater, for example) they should be under their own supervisor; if its non-blocking verification you need then it may make sense to just spawn_link or spawn_monitor a tiny task that knows where to send an {auth_message, #ref} when it is complete -- but this could very well be rolled into session management and the fsm spawns a new session which then makes sense to be under its own supervisor. The choice is up to you. I think the trend is to build bigger processes and try to make everything a gen_* -- even when that's an awkward fit (like socket handling, imo, is a better fit for a pure Erlang process started via proc_lib). >From the perspective of the supervisor tree a one-off process is part of the gen_fsm; the choice to link or monitor is based on how task failure fits in with your concept of state -- with an fsm guiding things this choice is usually pretty obvious (hence the choice of an fsm...). Don't make the mistake of making the gen_fsm become a supervisor, though -- that is the path to madness; when you need supervision do it for real. This idea of spawning short-lived processes to handle one-off tasks is occasionally viewed as heresy. YMMV. -Craig From touch.sereysethy@REDACTED Wed Sep 23 07:06:12 2015 From: touch.sereysethy@REDACTED (Sereysethy TOUCH) Date: Wed, 23 Sep 2015 12:06:12 +0700 Subject: [erlang-questions] Patch package OTP 18.0.3 released In-Reply-To: References: <560145CF.5060901@ericsson.com> Message-ID: Hi, Will it be available for all repositories like Ubuntu & Debian repository? I have just recently upgraded to Erlang 18, we have a lot of problem of memory leak: binary keeps increasing over time. Sethy On Wed, Sep 23, 2015 at 4:41 AM, J?r?me de Bretagne < jerome.debretagne@REDACTED> wrote: > Thanks to both of you for your quick feedback, Jesper and Henrik! > > J?r?me > Le 22 sept. 2015 2:13 PM, "Henrik Nord X" a > ?crit : > >> Hi >> >> Actually, we decided to not push the maint-XX branches at all. >> But we have sense re-evaluated that decision and should push them again >> as there was some usage on the branches after all. >> >> The missing tag and the new tag OTP-18.1 will be there tomorrow. >> >> >> >> On 09/22/2015 01:33 PM, Jesper Louis Andersen wrote: >> >> >> On Mon, Sep 21, 2015 at 11:32 PM, J?r?me de Bretagne < >> jerome.debretagne@REDACTED> wrote: >> >>> The maint-18 branch on GitHub contains the commits up to the OTP-18.0.2 >>> tag, but not the more recent ones up to OTP-18.0.3. Is it a simple >>> oversight on GitHub or am I making a wrong assumption about this branch? >> >> >> It just hasn't been moved yet, and I think it is an oversight. >> >> >> -- >> J. >> >> >> _______________________________________________ >> erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions >> >> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jean-sebastien.pedron@REDACTED Wed Sep 23 09:36:42 2015 From: jean-sebastien.pedron@REDACTED (=?UTF-8?Q?Jean-S=c3=a9bastien_P=c3=a9dron?=) Date: Wed, 23 Sep 2015 09:36:42 +0200 Subject: [erlang-questions] TLS cipher suite with Galois Counter Mode (GCM) In-Reply-To: References: <1468286756.3609392.1441790617301.JavaMail.zimbra@tpip.net> Message-ID: <5602568A.3090406@dumbbell.fr> On 09.09.2015 11:38, John Foldager wrote: > We'll probably upgrade directly from Erlang 16.x to 18.x, but we need > to know the following: > > - Will RabbitMQ team be supportive if issues arise with the > combination of Erlang 18.x and RabbitMQ 3.5.4+ ? Yes. One thing we already did was to replace the deprecated erlang:now/0 with the new Time API; it will be available in RabbitMQ 3.6.0. > - Will RabbitMQ be able to handle the GCM ciphers? My guess is yes > because I would believe that RabbitMQ is only using the Erlang SSL/TLS > layer and not restricting anything in it if it is supported. Yes, as you said, RabbitMQ does not do anything special with the TLS parameters. -- Jean-S?bastien P?dron -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 949 bytes Desc: OpenPGP digital signature URL: From samuelrivas@REDACTED Wed Sep 23 09:49:48 2015 From: samuelrivas@REDACTED (Samuel) Date: Wed, 23 Sep 2015 09:49:48 +0200 Subject: [erlang-questions] user process suspended In-Reply-To: <20150922144257.GK9409@fhebert-ltm1> References: <20150922141831.GJ9409@fhebert-ltm1> <20150922144257.GK9409@fhebert-ltm1> Message-ID: >> What do you mean with IO stuff stalling? Temporarily or permanently? >> We do send quite some io, but the systems stops to io when this >> happens and never recovers (we need to restart the node). >> > > It's been a while, but I do believe it was temporary for us. Like the EC2 > instances output would drop from say 10MB/s capacity down to 150kb/s for > blips of anywhere between 2 and 10 minutes. The degradation in performance > all but stalled everything we had that kept expecting multiple MB/s of > output to be possible. Ok, I think that is different. Once the problem starts to manifest the nodes are taken out of traffic by the load balancer, but they never recover. And processes are not busy doing anything. -- Samuel From roger@REDACTED Wed Sep 23 10:43:24 2015 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 23 Sep 2015 09:43:24 +0100 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: References: <56015FA9.7040701@research.att.com> <560191D5.6050905@research.att.com> Message-ID: On 23 September 2015 at 05:30, Vance Shipley wrote: > On Wed, Sep 23, 2015 at 12:32 AM, Roger Lipscombe > wrote: >> Both Theepan and Vance are correct in that this is a useful way to >> build this kind of thing. In fact, there's a good example (using a >> gen_server, rather than a gen_fsm) here: >> https://erlangcentral.org/wiki/index.php?title=Building_Non_Blocking_Erlang_apps > > What I don't like about that example is that it's not purely > functional. In that it uses ETS, rather than State, to store the {Tr_Id, From} associations? Yeah, that did seem a little odd. From roger@REDACTED Wed Sep 23 10:58:57 2015 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 23 Sep 2015 09:58:57 +0100 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: <56022403.9020702@research.att.com> References: <56015FA9.7040701@research.att.com> <560191D5.6050905@research.att.com> <56022403.9020702@research.att.com> Message-ID: On 23 September 2015 at 05:01, Garry Hodgson wrote: > On 9/22/15 3:02 PM, Roger Lipscombe wrote: >> I'd just add one point, though: I recently built something similar, >> and I found it simpler to separate the client into two processes: one >> gen_fsm to handle the state machine, and a separate gen_server to >> handle the decoding and framing. > > i briefly considered that, but it wasn't obvious how to divide the > work between the two. i'd be interested in how you did, if you'd > care to elaborate. Hmmm. I'll have a go. This is an approximation: % client.erl -- lots of details ellided -module(client). -behaviour(gen_fsm). init([Host, Port, User, Password]) -> {ok, C} = client_connection:start_link(self(), Host, Port), client_connection:send(C, {connect, User, Password}), {ok, connecting, #state{ conn = C }}. handle_info({connected, Token}, connecting, StateData) -> {next_state, connected, StateData#state{ token = Token }}; handle_info({friends, Txn, Friends}, connected, StateData) -> {From, NewStateData} = pop_txn(Txn, StateData), gen_fsm:reply(From, {ok, Friends}), {next_state, connected, NewStateData}; % etc. handle_sync_event(get_friends, _From, connecting, State) -> {reply, {error, not_connected}, connecting, State}; handle_sync_event(get_friends, From, connected, State) -> Txn = new_txn_id(), client_connection:send(C, {get_friends, Txn, State#state.token}), {next_state, connected, add_txn_to_state({Txn, From}, State)}; % etc. % client_connection.erl -module(client_connection). -behaviour(gen_server). start_link(Parent, Host, Port) -> % etc. {ok, #state{ parent = Parent, %etc }}. send(C, Msg) -> Bin = client_codec:encode(Msg), gen_server:cast(C, {send, Bin}). handle_info({tcp, Bin}, State = #state{ buffer = Buffer, parent = Parent }) -> case client_codec:decode(Buffer, Bin) of {need_more, NewBuffer} -> {noreply, State#state{ buffer = NewBuffer }}; {ok, Msg, NewBuffer} -> Parent ! Msg, {noreply, State#state{ buffer = NewBuffer }} end ...something like that, anyway. From jesper.louis.andersen@REDACTED Wed Sep 23 12:17:40 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 23 Sep 2015 12:17:40 +0200 Subject: [erlang-questions] Patch package OTP 18.0.3 released In-Reply-To: References: <560145CF.5060901@ericsson.com> Message-ID: On Wed, Sep 23, 2015 at 7:06 AM, Sereysethy TOUCH < touch.sereysethy@REDACTED> wrote: > Will it be available for all repositories like Ubuntu & Debian repository? > > I have just recently upgraded to Erlang 18, we have a lot of problem of > memory leak: binary keeps increasing over time. > Usually the Ubuntu and Debian repositories are behind by at least 1-2 releases, so it is often better to roll your own release. 18.1 is probably going out this week (maybe today), and it will contain every fix for the point releases 18.0.x. Your problem may be another one. If the binary space keeps growing, you need to use a tool like recon to hunt for the processes to which the binaries are leaking. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Wed Sep 23 12:56:21 2015 From: vasdeveloper@REDACTED (Theepan) Date: Wed, 23 Sep 2015 16:26:21 +0530 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: <56022403.9020702@research.att.com> References: <56015FA9.7040701@research.att.com> <560191D5.6050905@research.att.com> <56022403.9020702@research.att.com> Message-ID: I don't think it is a good idea to separate the state maintenance from the real work that is happening, unless you have a strong reason for it. What happens if you separate them is that you are keeping a process just for state maintenance, and most of the time the gen_fsm process is going to be idle. Other drawback is that the worker process will have to constantly check the state machine to ensure that it doing work at the correct state. There is a chance of off state processing and it is hardware hungry. There are places where you will separate the state maintenance, for instance many workers depending on a single state. At the low level gen_fsm and gen_tcp are just Erlang process, and what you are trying to do right now is good for what you are doing. Theepan On Wed, Sep 23, 2015 at 9:31 AM, Garry Hodgson wrote: > On 9/22/15 3:02 PM, Roger Lipscombe wrote: > >> On 22 September 2015 at 18:37, Garry Hodgson >> wrote: >> >>> thanks to both of you for your quick and helpful answers. >>> gotta love this list. >>> >> Both Theepan and Vance are correct in that this is a useful way to >> build this kind of thing. In fact, there's a good example (using a >> gen_server, rather than a gen_fsm) here: >> >> https://erlangcentral.org/wiki/index.php?title=Building_Non_Blocking_Erlang_apps >> > very nice. > >> I'd just add one point, though: I recently built something similar, >> and I found it simpler to separate the client into two processes: one >> gen_fsm to handle the state machine, and a separate gen_server to >> handle the decoding and framing. >> > i briefly considered that, but it wasn't obvious how to divide the > work between the two. i'd be interested in how you did, if you'd > care to elaborate. > > thanks > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From henrik.x.nord@REDACTED Wed Sep 23 13:15:25 2015 From: henrik.x.nord@REDACTED (Henrik Nord X) Date: Wed, 23 Sep 2015 13:15:25 +0200 Subject: [erlang-questions] Erlang/OTP 18.1 has been released! Message-ID: <560289CD.4090703@ericsson.com> Erlang/OTP 18.1 is a service release on the 18 track with mostly bug fixes, but is does contain a number of new features and characteristics improvements as well. Some highlights of the release are: * ssl: Add possibility to downgrade an SSL/TLS connection to a tcp connection, and give back the socket control to a user process. * ssh: The following new key exchange algorithms are implemented:'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521','diffie-hellman-group14-sha1', 'diffie-hellman-group-exchange-sha1' and 'diffie-hellman-group-exchange-sha256'. This raises the security level considerably. * kernel,stdlib,sasl: A mechanism for limiting the amount of text that the built-in error logger events will produce has been introduced. It is useful for limiting both the size of log files and the CPU time used to produce them. This mechanism is experimental in the sense that it may be changed based on feedback. See config parameter error_logger_format_depth in the Kernel application. You can find the Release Notes with more detailed info at http://www.erlang.org/download/ otp_src_18.1.readme You can download the full source distribution from http://www.erlang.org/download/otp_src_18.1.tar.gz Note: To unpack the TAR archive you need a GNU TAR compatible program. For installation instructions please read the README that is part of the distribution. You can also find the source code at github.com in the official Erlang repository. Git tag OTP-18.1 https://github.com/erlang/otp/tree/OTP-18.1 The Windows binary distributions can be downloaded from http://www.erlang.org/download/otp_win32_18.1.exe http://www.erlang.org/download/otp_win64_18.1.exe You can also download the complete HTML documentation or the Unix manual files http://www.erlang.org/download/otp_doc_html_18.1.tar.gz http://www.erlang.org/download/otp_doc_man_18.1.tar.gz You can also read the documentation on-line here: (see the Release Notes mentioned above for release notes which are not updated in the doc, but the new functionality is) http://www.erlang.org/doc/ We also want to thank those that sent us patches, suggestions and bug reports. The Erlang/OTP Team at Ericsson -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Sep 23 15:25:30 2015 From: zxq9@REDACTED (zxq9) Date: Wed, 23 Sep 2015 22:25:30 +0900 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: References: <56015FA9.7040701@research.att.com> <56022403.9020702@research.att.com> Message-ID: <2525818.jpc6u7RUbI@changa> An opposing view... On 2015?9?23? ??? 16:26:21 Theepan wrote: > I don't think it is a good idea to separate the state maintenance from the > real work that is happening, unless you have a strong reason for it. What > happens if you separate them is that you are keeping a process just for > state maintenance, and most of the time the gen_fsm process is going to be > idle. Other drawback is that the worker process will have to constantly > check the state machine to ensure that it doing work at the correct state. > There is a chance of off state processing and it is hardware hungry. I almost completely disagree with this view. I find gen_fsm to be useful specifically so that I can use an OTP process "just for state maintenance" -- sometimes this means a system composed mostly of tiny fsms, but more often it just means fsms are acting as logical gateways to control processing flow throughout the system (the valves and gateways of my digital kingdom). I *like* this separation because it makes the overall system much easier to reason about and modify. Some really huge system-wide behaviors can be modified very simply this way. As for being "hardware hungry"... I've never run into this problem. My gen_fsms tend to be really tiny, take up very little memory (generally ending with very close to the same number of bits as they started), and hardly running at all (sometimes good candidates for hiberation -- but usually there isn't any call for this). I really have a hard time understanding performance arguments (in terms of memory or processing time) against spawning temporary processes to get work done and terminating them when they are complete, as I have *never* encountered a case where this was actually a performance problem (at least not a noticeable one). Perhaps I'm just doing very different things with Erlang than everyone else -- but in any case without knowing the context *any* effort to deliberately increase instead of decrease the number of responsibilities a process has for performance reasons is headed in the wrong direction. At the very least when hacking around a problem I find it most useful to segregate as many responsibilites as possible early on, and that means gen_fsms are *only* finite-state machines, and they control processing flow based on their state. Any points of extended work are things I *want* to put in other processes, if for no other reason than an "idle" gen_fsm is a *highly responsive* gen_fsm. This helps me know more about the way a system responds under load (if this is even a concern) because I can just see what work is being spawned and how long bits of it is sticking around instead of trying to figure out how busy a given pack of fsms is. Sometimes (usually?) after some initial development I wind up moving things into a very different configuration because I understand the problem better, the needs of the system solidify a bit, any bottlenecks become obvious (and finding ways to avoid having them to begin with is usually pretty easy once the system is better understood), etc. But to start out with, ghah, putting more duties into a process than less has just never worked out very well for me. > There are places where you will separate the state maintenance, for > instance many workers depending on a single state. At the low level gen_fsm > and gen_tcp are just Erlang process, and what you are trying to do right > now is good for what you are doing. "Many workers depending on a single state." I'm trying to see where this is meaningful without an elaboration. -Craig From ilya.khlopotov@REDACTED Wed Sep 23 17:42:34 2015 From: ilya.khlopotov@REDACTED (ILYA Khlopotov) Date: Wed, 23 Sep 2015 08:42:34 -0700 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: +1 for multiple independent tracers. It would be also very helpful to support following: 1. Cluster avare seq_trace. Currently trace token is not send over erlang distribution protocol. It could be solved if new control message is added to distribution protocol for example. 2. There are a lot of different formats for trace messages and any tracing library need to be able to understand all of them. Almost any tracing librbary has to do something like: https://github.com/erlang/otp/blob/a2670f0822fc6729df956c8ec8c381340ff0a5fb/lib/et/src/et_selector.erl#L218 https://github.com/erlang/otp/blob/a2670f0822fc6729df956c8ec8c381340ff0a5fb/lib/et/src/et_selector.erl#L268 https://github.com/erlang/otp/blob/a2670f0822fc6729df956c8ec8c381340ff0a5fb/lib/runtime_tools/src/dbg.erl#L942 https://github.com/ferd/recon/blob/master/src/recon_trace.erl#L482 It is problematic though since tuples are of different size. Would it be better to send this info as an erlang record which anyone could import and use? It could be two records as well. Since every trace message has a meta info which describes where and what happend and list of arguments of the message. I.e. something like meta: { pid, ts, type, serial, ctx } event: { meta :: #meta{}, args :: [], sequence = 0 } 3. For seq_trace in particular it could be useful to be able to inject some context into the tracing process. Consider following use case. Let's say we have a notion of sessions. First user logins in order to start session, then he issues some commands using session_cookie as a mean to prove his identity. Since calls to login and command are independent http requests there is no way to link them together. We could have following: maybe_trace_login(_User, _Cookie) -> ok. %% function we use for dbg:fun2ms login(User, Password) -> %% check password raise if it is invalid Cookie = ..., maybe_trace_login(User, Cookie), ... command(Cookie, Command, Args) -> %% check Cookie raise if unknown do_command(Command, Args), %% we want to get trace of this ... However if we could store {User, Cookie} in the context of a tracer process from match spec. We could produce nice trace messages for calls to command which would include user name. This is the reason why ctx field is added in #meta{}. We also would need to be able to initiate tracing of command when cookie match somehow. BR, ILYA On Mon, Sep 21, 2015 at 1:52 AM, Lukas Larsson wrote: > Hello everyone. > > As you may know, one of the OTP teams focus areas for the coming year is > make tracing better. At the moment we are gathering ideas and attempting to > put together a vision of what we would like to have, before deciding what > we can make. > > I'm pretty sure that many of you have much more experience with using > Erlang tracing while developing and in production than I do, which is we > would love to have your input as to what you would like to change about > tracing. > > To set the scope of the discussion, when I say tracing I include; erlang > tracing, dtrace/systemtap, trace outputs (stdout/file/IP), filtering > through match specs, sequence tracing, tool integration (dbg, fprof, > redbug, recon to mention some) and probably more. > > To start the discussion, here are a few of my ideas in no particular order: > * Allow multiple tracers. Today only one port/process can be the > receiver of trace data. > * Create a couple of scalable high throughput tracing output backends > with different overflow mechanics. Today all tracing is funneled through > one bottleneck and has no overflow handling at all. > * Expose vm probes (today dtrace probes) to the erlang tracer. > * Better integration of dtrace/lttngt/systemtap into the erlang trace. > * Allow the erlang tracer to be an Erlang callback module. Today only > ports/processes are allowed. > * Optimize trace output to file/ip. Maybe use something like the Common > Trace Format ( > http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.md;hb=master), > instead of the term_to_binary that we have today. > * Write much much better documentation for dbg :) > > We are looking for feedback from both beginners as well as seasoned > veterans to make erlang tracing the best it can be. So if you have any > thoughts or ideas, join the discussion to make Erlang tracing better for > you and everyone else. > > Thanks in advance, > Lukas > Erlang/OTP team > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Wed Sep 23 17:51:45 2015 From: rvirding@REDACTED (Robert Virding) Date: Wed, 23 Sep 2015 17:51:45 +0200 Subject: [erlang-questions] Abstract code in .beam files In-Reply-To: References: Message-ID: LFE compiles to Core not to Erlang AST. I have a hack for dialyzer which allows it to get the Core erlang it uses from an LFE source file, This works but it means that the the LFE compiler can't add the Erlang AST which dialyzer uses to the .beam file. It is a bit hacky but it works. The dialyzer interface code is not very cleanly written but IIRC it assumes that one source file results in one module. I have now extended the LFE compiler so you can compile multiple modules in one source file. (interesting for flavors and structs) My thought was that if I can hack the compiler so it puts the Core erlang AST in the .beam file instead of the Erlang AST then I could extend my dialyzer hack to even work with the .beam file. However that would mean hacking the Erlang compiler to handle a new option which I don't really want to do. It's a pretty easy hack but there are limits. Anyway that was my idea. I see I will have to wait until the Erlang compiler is fixed to do this. And dialyzer. Robert On 21 September 2015 at 14:46, Jos? Valim wrote: > Robert, the .beam file is made of a bunch of chunks. The abstract code is > stored in a chunk named "Abst", where it is versioned and > term_to_binary'ed. Literally: > > erlang:term_to_binary({raw_abstract_v1, AST}) > > You can't/shouldn't store the core format here but you can create as many > chunks as you want in the BEAM file. We actually do this in Elixir to store > the documentation. At EUC, I talked to Bj?rn Gustavsson about officially > supporting a core chunk. Here is what we agreed on to the best of my memory: > > 1. We will have a new chunk named "Core" (they are all four letters) > > 2. The core AST will be versioned as well (raw_core_v1) and > term_to_binary'ed, like this: erlang:term_to_binary({raw_core_v1, CoreAST}) > > I also talked to Kostis so dialyzer starts reading the core information > from it too. Let me know if you go ahead with this so we can sync before I > eventually send a PR to OTP and dialyzer. > > > > > *Jos? Valim* > www.plataformatec.com.br > Skype: jv.ptec > Founder and Director of R&D > > On Mon, Sep 21, 2015 at 2:14 PM, Robert Virding > wrote: > >> When you compile a file with debug_info then the Erlang AST of the file >> is included in the .beam file. Does anyone know if the actual structure >> which is put in the file is checked for format? Can I put anything in >> there, for example the core code? >> >> Robert >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Wed Sep 23 18:01:37 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 23 Sep 2015 18:01:37 +0200 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: <2525818.jpc6u7RUbI@changa> References: <56015FA9.7040701@research.att.com> <56022403.9020702@research.att.com> <2525818.jpc6u7RUbI@changa> Message-ID: On Wed, Sep 23, 2015 at 3:25 PM, zxq9 wrote: > I almost completely disagree with this view. Process isolation is another reason as to why you want to take a look at splitting up stuff. If you runs the processing loop in a separate process, it can be easier to handle the case where that part of the system crashes. In other situations, there are advantages to writing a single gen_fsm with a separate reply back to the caller. It really depends on the situation what is the best course of action :) -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Wed Sep 23 18:25:37 2015 From: roger@REDACTED (Roger Lipscombe) Date: Wed, 23 Sep 2015 17:25:37 +0100 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: References: <56015FA9.7040701@research.att.com> <56022403.9020702@research.att.com> <2525818.jpc6u7RUbI@changa> Message-ID: On 23 September 2015 at 17:01, Jesper Louis Andersen wrote: > Process isolation is another reason as to why you want to take a look at > splitting up stuff. If you runs the processing loop in a separate process, > it can be easier to handle the case where that part of the system crashes. > In other situations, there are advantages to writing a single gen_fsm with a > separate reply back to the caller. It really depends on the situation what > is the best course of action :) It's also easier to test if you don't have to do it by poking your FSM with binary. From vasdeveloper@REDACTED Wed Sep 23 18:50:41 2015 From: vasdeveloper@REDACTED (Theepan) Date: Wed, 23 Sep 2015 22:20:41 +0530 Subject: [erlang-questions] "calling" a gen_fsm In-Reply-To: <2525818.jpc6u7RUbI@changa> References: <56015FA9.7040701@research.att.com> <56022403.9020702@research.att.com> <2525818.jpc6u7RUbI@changa> Message-ID: Hello Craig, I respect your views, although I have different views. I have used both the methods in the system that I had developed, of course with a reason. One of the systems that I developed had to handle more than half a million transactions per second from a single box, and thus performance and hardware efficiency was paramount for us. I was very conscious about the design and each line I wrote. May be for a couple of thousands transactions per second system, you don't have to worry about these much. >> "Many workers depending on a single state." I'm trying to see where this is meaningful without an elaboration. There are plenty of example for this. How would you model the states of a stock market (singleton), on a trading system? Theepan On Wed, Sep 23, 2015 at 6:55 PM, zxq9 wrote: > An opposing view... > > On 2015?9?23? ??? 16:26:21 Theepan wrote: > > I don't think it is a good idea to separate the state maintenance from > the > > real work that is happening, unless you have a strong reason for it. What > > happens if you separate them is that you are keeping a process just for > > state maintenance, and most of the time the gen_fsm process is going to > be > > idle. Other drawback is that the worker process will have to constantly > > check the state machine to ensure that it doing work at the correct > state. > > There is a chance of off state processing and it is hardware hungry. > > I almost completely disagree with this view. I find gen_fsm to be useful > specifically so that I can use an OTP process "just for state maintenance" > -- sometimes this means a system composed mostly of tiny fsms, but more > often it just means fsms are acting as logical gateways to control > processing flow throughout the system (the valves and gateways of my > digital kingdom). > > I *like* this separation because it makes the overall system much easier > to reason about and modify. Some really huge system-wide behaviors can be > modified very simply this way. > > As for being "hardware hungry"... I've never run into this problem. My > gen_fsms tend to be really tiny, take up very little memory (generally > ending with very close to the same number of bits as they started), and > hardly running at all (sometimes good candidates for hiberation -- but > usually there isn't any call for this). I really have a hard time > understanding performance arguments (in terms of memory or processing time) > against spawning temporary processes to get work done and terminating them > when they are complete, as I have *never* encountered a case where this was > actually a performance problem (at least not a noticeable one). Perhaps I'm > just doing very different things with Erlang than everyone else -- but in > any case without knowing the context *any* effort to deliberately increase > instead of decrease the number of responsibilities a process has for > performance reasons is headed in the wrong direction. > > At the very least when hacking around a problem I find it most useful to > segregate as many responsibilites as possible early on, and that means > gen_fsms are *only* finite-state machines, and they control processing flow > based on their state. Any points of extended work are things I *want* to > put in other processes, if for no other reason than an "idle" gen_fsm is a > *highly responsive* gen_fsm. This helps me know more about the way a system > responds under load (if this is even a concern) because I can just see what > work is being spawned and how long bits of it is sticking around instead of > trying to figure out how busy a given pack of fsms is. Sometimes (usually?) > after some initial development I wind up moving things into a very > different configuration because I understand the problem better, the needs > of the system solidify a bit, any bottlenecks become obvious (and finding > ways to avoid having them to begin with is usually pretty easy once the > system is better understood), etc. But to start out with, ghah, putting > more duties into a process than less has just never worked out very well > for me. > > > There are places where you will separate the state maintenance, for > > instance many workers depending on a single state. At the low level > gen_fsm > > and gen_tcp are just Erlang process, and what you are trying to do right > > now is good for what you are doing. > > "Many workers depending on a single state." I'm trying to see where this > is meaningful without an elaboration. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexvazquezfente@REDACTED Wed Sep 23 19:55:54 2015 From: alexvazquezfente@REDACTED (Alex Vazquez) Date: Wed, 23 Sep 2015 19:55:54 +0200 Subject: [erlang-questions] inets httpc pipelining doubt Message-ID: Hi list, I'm debugging an issue with concurrent downloads and, first of all, i need help understanding how httpc pipelining works. If i execute the following snippet in Erlang 17.5, this is what i get: --- SNIP --- SNIP --- SNIP --- Erlang/OTP 17 [erts-6.4] [source] [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false] Eshell V6.4 (abort with ^G) 1> inets:start(). ok 2> httpc:set_options([ 2> {max_keep_alive_length, 2} 2> ,{keep_alive_timeout, 120000} 2> ,{max_pipeline_length, 2} 2> ,{pipeline_timeout, 120000} 2> ,{max_sessions, 2} 2> ]). ok 3> DownloadFun = fun(Id, Url) -> 3> fun() -> 3> Res = httpc:request( 3> get, {Url, []}, 3> [{version, "HTTP/1.1"}, {connect_timeout, 5000}, {timeout, 300000}], 3> [{sync, true}, 3> {full_result, false}, 3> {stream, "/tmp/test_file" ++ integer_to_list(Id)}]), 3> io:format(user, "Process ~p (~p) finished @~p with result ~p, sessions ~p~n", [Id, self(), erlang:now(), Res, httpc:which_sessions()]) 3> end 3> end. #Fun 4> erlang:now(). {1443,30436,547744} 5> [spawn(DownloadFun(Id, " http://www.erlang.org/download/otp_src_18.1.tar.gz")) || Id <- lists:seq(1,3)]. [<0.50.0>,<0.51.0>,<0.52.0>] 6> Process 2 (<0.51.0>) finished @{1443,30519,307502} with result {ok, saved_to_file}, sessions {[{session, {{"www.erlang.org", 80}, <0.55.0>}, false, http, #Port<0.725>, ip_comm, 1, pipeline, false}], [], []} Process 1 (<0.50.0>) finished @{1443,30526,429106} with result {ok, saved_to_file}, sessions {[{session, {{"www.erlang.org", 80}, <0.54.0>}, false, http, #Port<0.723>, ip_comm, 0, pipeline, true}, {session, {{"www.erlang.org", 80}, <0.55.0>}, false, http, #Port<0.725>, ip_comm, 0, pipeline, true}], [], []} Process 3 (<0.52.0>) finished @{1443,30526,490409} with result {ok, saved_to_file}, sessions {[{session, {{"www.erlang.org", 80}, <0.54.0>}, false, http, #Port<0.723>, ip_comm, 0, pipeline, true}, {session, {{"www.erlang.org", 80}, <0.55.0>}, false, http, #Port<0.725>, ip_comm, 0, pipeline, true}, {session, {{"www.erlang.org", 80}, <0.56.0>}, false, http, #Port<0.724>, ip_comm, 0, pipeline, true}], [], []} 6> --- SNIP --- SNIP --- SNIP --- Given the options i'm setting with httpc:set_options/1, i would expect debugging function httpc:which_sessions/0 to never return 3 open connections to the same host. I would expect 2 connections no matter if the server supports pipelining or not (if not, i guess that one of the three downloads would take longer). Another weird thing is that httpc:which_sessions/0 only adds the connection to the list after the resource is downloaded but maybe it is just its expected behaviour. I'd appreciate any comment that helps me understanding how all this stuff works. Maybe then, i could properly debug my issue or maybe report some bug :) Regards, -- Alejandro Vazquez Fente -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.wobben@REDACTED Wed Sep 23 22:30:33 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Wed, 23 Sep 2015 22:30:33 +0200 Subject: [erlang-questions] feedback please Message-ID: <56030BE9.1070902@home.nl> Hello, I did chapter 4 of Programming Erlang which is about pattern matching. So as a challenge I tried to make a little cash on hands programm. Here is the code : -module(balance). -export([get_balance/1]). get_balance(Balance) -> io:format("~n~n"), io:format("Your account balance is $~w.~n", [Balance]), Entered_action = io:get_line("Which Action do you need (w)ithdraw/(d)eposit? "), Action = string:strip(Entered_action, right, $\n), Entered_amount = io:get_line("Which Amount do you want to withdraw/deposit? "), Amount2 = string:to_integer(string:strip(Entered_amount, right, $\n)), Amount = element(1,Amount2), get_balance(Balance, {Action, Amount}). get_balance(0,{"w",_ }) -> io:format("You have no funds to withdraw"), get_balance(0); get_balance(Balance, {"w", Amount}) when Amount > Balance -> io:format("You cannot withdraw more then you have"), get_balance(Balance); get_balance(Balance, {"w", Amount}) -> get_balance(Balance - Amount); get_balance(Balance, {"d", Amount}) -> get_balance(Balance + Amount); Are there things I could do better without processes or OTP because they are still not explained. Roelof From zxq9@REDACTED Thu Sep 24 06:44:55 2015 From: zxq9@REDACTED (zxq9) Date: Thu, 24 Sep 2015 13:44:55 +0900 Subject: [erlang-questions] feedback please In-Reply-To: <56030BE9.1070902@home.nl> References: <56030BE9.1070902@home.nl> Message-ID: <3340168.gZWebXRfOB@burrito> On Wednesday 23 September 2015 22:30:33 Roelof Wobben wrote: > Hello, > > I did chapter 4 of Programming Erlang which is about pattern matching. > > So as a challenge I tried to make a little cash on hands programm. > > Here is the code : > > -module(balance). > > -export([get_balance/1]). > > get_balance(Balance) -> > io:format("~n~n"), > io:format("Your account balance is $~w.~n", [Balance]), > > Entered_action = io:get_line("Which Action do you need (w)ithdraw/(d)eposit? "), > Action = string:strip(Entered_action, right, $\n), > > Entered_amount = io:get_line("Which Amount do you want to withdraw/deposit? "), > Amount2 = string:to_integer(string:strip(Entered_amount, right, $\n)), > Amount = element(1,Amount2), > > get_balance(Balance, {Action, Amount}). > > get_balance(0,{"w",_ }) -> > io:format("You have no funds to withdraw"), > get_balance(0); > > get_balance(Balance, {"w", Amount}) when Amount > Balance -> > io:format("You cannot withdraw more then you have"), > get_balance(Balance); > > get_balance(Balance, {"w", Amount}) -> > get_balance(Balance - Amount); > get_balance(Balance, {"d", Amount}) -> get_balance(Balance + Amount); I'm assuming the last clause got cut off; maybe something like: get_balance(Balance, BadInput) -> ok = io:format("CUT THAT OUT! \"~tp\" is not a valid option!~n", [BadInput]), get_balance(Balance). > Are there things I could do better without processes or OTP because they > are still not explained. On the point of pattern matching, your program makes natural use of of it. I think it is clear you basically understand how pattern matching works in function heads -- and that is the point of the chapter. You made good use of a guard, also, which is sort of a natural extension of pattern matching (both in function heads and case statements). I'm sure we could spend a week blathering about user interface issues, how to structure functions around UI elements, events, etc., but that's not the point. There is a whole school of thought on how to structure functional UI code, but talking about that distracts from the point of pattern matching and guards. There are a *lot* of opportunities to use pattern matching in Erlang. You will realize this the more you write little toy programs like the one above for yourself, and especially when as you write, hack around on and refactor programs you care about. It is super useful and something you will sorely miss in languages that lack it. Something you may notice in other people's code is conversion of external input to internal atoms that represent something about the input. This can make programs easier to read, and incidentally can make them faster (though usually not noticeably so). For example, you accept two types of user actions, Withdrawl and Deposit. The user can input "w" or "d". We could broaden the variety of input that is acceptable, make the pattern-matching part of the code more readable (more self-documenting -- important if you have a huge menu of options), and keep the "bad external input" case away from the get_balance/2 processing function: -module(roelof1). -export([get_balance/1]). get_balance(Balance) -> io:format("~n~n"), io:format("Your account balance is $~w.~n", [Balance]), Action = ask_action(), Amount = ask_amount(), get_balance(Balance, {Action, Amount}). get_balance(0, {withdrawl, _Amount}) -> ok = io:format("Hey! You're broke!~n"), get_balance(0); get_balance(Balance, {withdrawl, Amount}) when Amount > Balance -> ok = io:format("You cannot withdraw more than you have~n"), get_balance(Balance); get_balance(Balance, {withdrawl, Amount}) -> get_balance(Balance - Amount); get_balance(Balance, {deposit, Amount}) -> get_balance(Balance + Amount). ask_action() -> Input = io:get_line("Which action do you need? (W)ithdrawl/(D)eposit? "), case Input of [$w | _] -> withdrawl; [$W | _] -> withdrawl; [$d | _] -> deposit; [$D | _] -> deposit; BadInput -> ok = io:format("You can't seem to '~tp' today...~n" "Let's try that again...~n", [BadInput]), ask_action() end. ask_amount() -> Input = io:get_line("How much? "), case scrub_amount(Input) of {ok, Number} -> Number; {error, bad_input} -> ok = io:format("Hrm... '~tp' does not seem to be a number~n" "Now, one more time...~n", [Input]), ask_amount() end. scrub_amount(Input) -> case string:to_integer(Input) of {error, no_integer} -> {error, bad_input}; {Int, _} -> {ok, Int} end. There are still plenty of odd cases up there above (What about {dollars, pennies, fractional} values? What if the user wants to withdraw $1.50? How to gracefully exit? etc.) -- but the point is not to present a case for fully-massaged external input, my point is that pattern matching is used everywhere and makes it very natural to use symbolic references to certain types of input (user commands, in this case) as well as present a lot of chances to break your large functions into smaller, much more narrowly focused functions. Pattern matching is GREAT! Its really nice to see that you've got a grip on this. Anyway, if you are able to read the code above without your head exploding then it is time to move on to the next chapter in your book. I think you've got pattern matching covered. Keep going! -Craig From r.wobben@REDACTED Thu Sep 24 07:13:15 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Thu, 24 Sep 2015 07:13:15 +0200 Subject: [erlang-questions] feedback please In-Reply-To: <3340168.gZWebXRfOB@burrito> References: <56030BE9.1070902@home.nl> <3340168.gZWebXRfOB@burrito> Message-ID: <5603866B.1010306@home.nl> Op 24-9-2015 om 06:44 schreef zxq9: > On Wednesday 23 September 2015 22:30:33 Roelof Wobben wrote: >> Hello, >> >> I did chapter 4 of Programming Erlang which is about pattern matching. >> >> So as a challenge I tried to make a little cash on hands programm. >> >> Here is the code : >> >> -module(balance). >> >> -export([get_balance/1]). >> >> get_balance(Balance) -> >> io:format("~n~n"), >> io:format("Your account balance is $~w.~n", [Balance]), >> >> Entered_action = io:get_line("Which Action do you need (w)ithdraw/(d)eposit? "), >> Action = string:strip(Entered_action, right, $\n), >> >> Entered_amount = io:get_line("Which Amount do you want to withdraw/deposit? "), >> Amount2 = string:to_integer(string:strip(Entered_amount, right, $\n)), >> Amount = element(1,Amount2), >> >> get_balance(Balance, {Action, Amount}). >> >> get_balance(0,{"w",_ }) -> >> io:format("You have no funds to withdraw"), >> get_balance(0); >> >> get_balance(Balance, {"w", Amount}) when Amount > Balance -> >> io:format("You cannot withdraw more then you have"), >> get_balance(Balance); >> >> get_balance(Balance, {"w", Amount}) -> >> get_balance(Balance - Amount); >> get_balance(Balance, {"d", Amount}) -> get_balance(Balance + Amount); > I'm assuming the last clause got cut off; maybe something like: > > get_balance(Balance, BadInput) -> > ok = io:format("CUT THAT OUT! \"~tp\" is not a valid option!~n", > [BadInput]), > get_balance(Balance). > >> Are there things I could do better without processes or OTP because they >> are still not explained. > On the point of pattern matching, your program makes natural use of of it. I think it is clear you basically understand how pattern matching works in function heads -- and that is the point of the chapter. You made good use of a guard, also, which is sort of a natural extension of pattern matching (both in function heads and case statements). > > I'm sure we could spend a week blathering about user interface issues, how to structure functions around UI elements, events, etc., but that's not the point. There is a whole school of thought on how to structure functional UI code, but talking about that distracts from the point of pattern matching and guards. > > There are a *lot* of opportunities to use pattern matching in Erlang. You will realize this the more you write little toy programs like the one above for yourself, and especially when as you write, hack around on and refactor programs you care about. It is super useful and something you will sorely miss in languages that lack it. > > Something you may notice in other people's code is conversion of external input to internal atoms that represent something about the input. This can make programs easier to read, and incidentally can make them faster (though usually not noticeably so). For example, you accept two types of user actions, Withdrawl and Deposit. The user can input "w" or "d". We could broaden the variety of input that is acceptable, make the pattern-matching part of the code more readable (more self-documenting -- important if you have a huge menu of options), and keep the "bad external input" case away from the get_balance/2 processing function: > > > -module(roelof1). > -export([get_balance/1]). > > get_balance(Balance) -> > io:format("~n~n"), > io:format("Your account balance is $~w.~n", [Balance]), > Action = ask_action(), > Amount = ask_amount(), > get_balance(Balance, {Action, Amount}). > > get_balance(0, {withdrawl, _Amount}) -> > ok = io:format("Hey! You're broke!~n"), > get_balance(0); > get_balance(Balance, {withdrawl, Amount}) when Amount > Balance -> > ok = io:format("You cannot withdraw more than you have~n"), > get_balance(Balance); > get_balance(Balance, {withdrawl, Amount}) -> > get_balance(Balance - Amount); > get_balance(Balance, {deposit, Amount}) -> > get_balance(Balance + Amount). > > ask_action() -> > Input = io:get_line("Which action do you need? (W)ithdrawl/(D)eposit? "), > case Input of > [$w | _] -> withdrawl; > [$W | _] -> withdrawl; > [$d | _] -> deposit; > [$D | _] -> deposit; > BadInput -> > ok = io:format("You can't seem to '~tp' today...~n" > "Let's try that again...~n", > [BadInput]), > ask_action() > end. > > ask_amount() -> > Input = io:get_line("How much? "), > case scrub_amount(Input) of > {ok, Number} -> > Number; > {error, bad_input} -> > ok = io:format("Hrm... '~tp' does not seem to be a number~n" > "Now, one more time...~n", > [Input]), > ask_amount() > end. > > scrub_amount(Input) -> > case string:to_integer(Input) of > {error, no_integer} -> {error, bad_input}; > {Int, _} -> {ok, Int} > end. > > > There are still plenty of odd cases up there above (What about {dollars, pennies, fractional} values? What if the user wants to withdraw $1.50? How to gracefully exit? etc.) -- but the point is not to present a case for fully-massaged external input, my point is that pattern matching is used everywhere and makes it very natural to use symbolic references to certain types of input (user commands, in this case) as well as present a lot of chances to break your large functions into smaller, much more narrowly focused functions. > > Pattern matching is GREAT! Its really nice to see that you've got a grip on this. Anyway, if you are able to read the code above without your head exploding then it is time to move on to the next chapter in your book. I think you've got pattern matching covered. > > Keep going! > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > ----- > Geen virus gevonden in dit bericht. > Gecontroleerd door AVG - www.avg.com > Versie: 2015.0.6140 / Virusdatabase: 4419/10685 - datum van uitgifte: 09/23/15 > Thanks for the feedback. I understand the code and I see you use error handling. That Is the next chapter I need to study. So after that chapter I can change it to the code you build. But L like the way to handle the input a lot. Roelof From zxq9@REDACTED Thu Sep 24 07:23:28 2015 From: zxq9@REDACTED (zxq9) Date: Thu, 24 Sep 2015 14:23:28 +0900 Subject: [erlang-questions] feedback please In-Reply-To: <5603866B.1010306@home.nl> References: <56030BE9.1070902@home.nl> <3340168.gZWebXRfOB@burrito> <5603866B.1010306@home.nl> Message-ID: <4932605.QsZFrd1HLM@burrito> On Thursday 24 September 2015 07:13:15 Roelof Wobben wrote: > Thanks for the feedback. > > I understand the code and I see you use error handling. That Is the next > chapter I need to study. > So after that chapter I can change it to the code you build. But L like > the way to handle the input a lot. That was an example of match-or-crash as error handling. Any place that code would have crashed is a place there wasn't a match, which meant something deeper than our expected input case was faulty -- and in that case it is better to stop execution and crash completely than carry on with crazy data or a screwed up underlying runtime. If the next chapter covers "error handling" it probably talks about how crashes are dealt with (and the link/monitor messages that are sent whenever a process crashes/exits) and how to use `try.. catch` to handle exceptions without crashing. There aren't very many places you will want to use try..catch (many folks make a point of trying very hard not to use it, myself included), but it is important to know. Have fun! -Craig From fritchie@REDACTED Thu Sep 24 07:37:22 2015 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Thu, 24 Sep 2015 00:37:22 -0500 Subject: [erlang-questions] user process suspended In-Reply-To: Message of "Tue, 22 Sep 2015 16:33:12 +0200." Message-ID: <2595.1443073042@snookles.snookles.com> Samuel wrote: sr> We are starting to suspect that it is the lager console_backed that sr> hangs in a send to the io port. We have just disabled it and we'll sr> see if it helps. Hi, Samuel. Basho has fought a similar problem with console I/O hanging. IIRC we had the most problems with Solaris/Illumos/SmartOS. My poor memory is forgetting which BEAM runtimes were succeptible. I also don't know if or how often the problem was reported to us on other OS platforms. There were a couple of diagnostics that we used: 1. Getting process_info(Pid, backtrace) for all procs. Then we'd see that the ... hrm, which process ... hrrrrm, Riak version 1.2 used R15B01, I think ... the 'user' process? This may or may not resemble the hang, see the end of this message. 2. Connecting to the Riak console via the "to_erl" scheme to talk to The VM's Actual Console. Then the admin would press Return ... and the Riak node would spring to life, problem solved. Until next time. IIRC disabling console output was the workaround. I've lost the ops history of when that problem was fixed VM/runtime-wise and/or Riak's workarounds. But if this sounds like your problem, then it's worth digging for that history. Oh, here's a piece of luck. I've rediscovered that I'd written an Expect script to automate the process for triggering the problem. I'll attach that, also, in case it's helpful. -Scott --- snip --- snip --- snip --- snip --- =proc:<0.30.0> State: Waiting Name: user Spawned as: erlang:apply/2 Current call: user:get_chars_more/7 Spawned by: <0.29.0> Started: Tue Apr 9 16:16:46 2013 Message queue length: 1 Message queue: [{io_request,<0.186.0>,<0.179.0>,{put_chars,unicode,<<821 bytes>>}}] Number of heap fragments: 0 Heap fragment data: 0 Link list: [] Dictionary: [{shell,<0.31.0>},{read_mode,list},{unicode,false}] Reductions: 357542 Stack+heap: 610 OldHeap: 0 Heap unused: 152 OldHeap unused: 0 Stack dump: Program counter: 0x00007f0c51d92930 (user:get_chars_more/7 + 232) CP: 0x0000000000000000 (invalid) arity = 0 0x00007f0c51931710 Return addr 0x00007f0c51d8c498 (user:do_io_request/5 + 88) y(0) unicode y(1) {[],[]} y(2) #Port<0.630> y(3) {erl_scan,tokens,[1]} y(4) get_until y(5) io_lib y(6) {erl_scan_continuation,[],no_col,[],2,{erl_scan,#Fun,false,false,false},0,#Fun} 0x00007f0c51931750 Return addr 0x00007f0c51d8c308 (user:server_loop/2 + 1408) y(0) #Port<0.630> y(1) <0.30.0> y(2) <0.25333.1728> 0x00007f0c51931770 Return addr 0x00007f0c51d8b968 (user:catch_loop/3 + 112) y(0) #Port<0.630> 0x00007f0c51931780 Return addr 0x0000000000836c78 () y(0) <0.31.0> y(1) #Port<0.630> y(2) Catch 0x00007f0c51d8b968 (user:catch_loop/3 + 112) --- snip --- snip --- snip --- snip --- #!/usr/bin/expect -- ## Assumptions: ## ## 1. SSH has been used to log into "hostname" once before: we aren't ## smart enough to deal with SSH host key verification stuff. ## ## 2. This script can successfully use SSH to log into hostname using ## the "riak" user. ## ## 3. If SSH does not require a password for the "riak" user on ## hostname, then please break that feature so that the SSH ## password authentication method is required. ## ## 4. We assume that the "riak" account's shell prompt will include ## a "$" in it. If that is not a correct assumption, please fix ## the login_prompt procedure below. ## ## 5. This script will stop Riak and kill any other beam.smp processes ## that happen to be running on hostname. ## ## 6. We assume riak is SMP-capable and look at beam.smp processes only. ## Usage: this-script hostname password riak_path erl_path cookie riak_node_name ## ## For example, if the remote machine is using a Linux CentOS Riak package: ## ## expect this-script 172.16.72.130 foofoo /usr/sbin/riak /usr/lib64/riak/erts-5.9.1/bin/erl riak riak@REDACTED ## Reminder reminder reminder: never use single quotes ## Reminder reminder reminder: never use single quotes ## Reminder reminder reminder: never use single quotes set timeout 30 set erl_prompt {@[^)]+\)\d+> } set hostname [lindex $argv 0] set password [lindex $argv 1] set riak_path [lindex $argv 2] set erl_path [lindex $argv 3] set cookie [lindex $argv 4] set riak_node_name [lindex $argv 5] proc login_prompt {} { expect -re {\$} # If a dollar sign does not appear in the prompt, comment out # the line above, uncomment the line below, then edit the line # below to include the correct prompt string. # expect -re {some prompt regexp here} } proc erlang_prompt {} { global erl_prompt expect -re $erl_prompt } proc cmd {cmd} { send "$cmd\r" login_prompt } proc async_cmd {cmd} { send "$cmd\r" } proc erlang_cmd {cmd} { send "$cmd\r" erlang_prompt } proc announce {msg} { puts "\n\n*** [exec date]\n*** $msg" } proc switch_to {id} { global spawn_id announce "Switching to session $id" set spawn_id $id } # send_user "Password for host $hostname: " # stty -echo # expect_user -re "(.*)\n" # stty echo # set password $expect_out(1,string) # puts "" spawn ssh -l riak $hostname set session1 $spawn_id expect {password} send "$password\r" login_prompt announce "Killing Riak and all other beam.smp processes" switch_to $session1 cmd {ps axww | grep beam.smp | grep -v grep | awk '{print $1}' | xargs kill -9 ; sleep 1} cmd {ps -ef | grep beam.smp | grep -v grep | awk '{print $2}' | xargs kill -9 ; sleep 1} announce "Starting Riak" spawn ssh -l riak $hostname "$riak_path start" set session2 $spawn_id expect {password} send "$password\r" expect "ok" set after_start 10 announce "Got ok from '$riak_path start', sleeping $after_start seconds..." sleep $after_start announce "Attach to riak, do sanity test" switch_to $session1 async_cmd "$riak_path attach" send "hello_world.\r" erlang_prompt announce "2nd login session to $hostname, attach to riak via remsh" spawn ssh -l riak $hostname set session2 $spawn_id expect {password} send "$password\r" login_prompt switch_to $session2 async_cmd "$erl_path -setcookie $cookie -name tmp@REDACTED -remsh $riak_node_name" erlang_prompt erlang_cmd "hello." announce "Log a message via lager, watch for it over on Riak console" switch_to $session2 erlang_cmd "lager:log(warning, self(), \"WARNING 1\", \[\])." switch_to $session1 expect { "WARNING 1" { announce "Found our first warning, excellent!" } timeout { puts "BUMMER 1!" ; exit 4 } } announce "Now, to hang the logger to demonstrate the bug.\n Any incomplete term is fine, we'll use an unfinished tuple...." switch_to $session1 async_cmd "{tuple_item_number_1," set bug_timeout 12 announce "Now that console is hung, call lager:log().\n Set timeout time to $bug_timeout seconds.\n This function should execute in only a few milliseconds." set timeout $bug_timeout switch_to $session2 async_cmd "lager:log(warning, self(), \"WARNING 2\", \[\])." switch_to $session1 expect { "WARNING 2" { announce "BUMMER, Found 2nd warning at wrong time"; exit 4} timeout { announce "TIMEOUT! Excellent, the bug has been demonstrated (1)" } } announce "When we finish the erlang term over on $session1 ('riak attach' session),\n then the logger will become unblocked." switch_to $session1 async_cmd "element_number_2}." expect { -re $erl_prompt { announce "Excellent, we have 'riak attach' prompt" } timeout { announce "BUMMER, timeout when not expected (2)"; exit 4} } announce "Now go to session $session2 and expect the 2nd warning\n _and_ the -remsh's prompt." switch_to $session2 expect { "WARNING 2" { announce "Excellent, I see the 'WARNING 2' log message" } timeout { announce "BUMMER, timeout when not expected @ 2nd w"; exit 4} } expect { -re $erl_prompt { announce "Excellent, we have the 'riak -remsh' prompt" } timeout { announce "BUMMER, timeout when not expected (3)"; exit 4} } announce "Time to clean up both sessions" switch_to $session1 async_cmd "\004" login_prompt async_cmd "exit\r" expect eof switch_to $session2 async_cmd "\003" ; sleep 1 async_cmd "\003" ; sleep 1 login_prompt async_cmd "exit\r" expect eof announce "Excellent, this successful test is finished." exit 0 From zxq9@REDACTED Thu Sep 24 07:39:44 2015 From: zxq9@REDACTED (zxq9) Date: Thu, 24 Sep 2015 14:39:44 +0900 Subject: [erlang-questions] feedback please In-Reply-To: <5603866B.1010306@home.nl> References: <56030BE9.1070902@home.nl> <3340168.gZWebXRfOB@burrito> <5603866B.1010306@home.nl> Message-ID: <5235970.BP7GLh1TPO@burrito> Ah, one thing worth pointing out... some people really hate case statements. I'm not religious about this, but very often I will do something slightly different than you might see in books: "name" my case statements by making them into function calls. (I *do* hate nested case statements, though, especially when there are a lot of them!) Consider the way ask_action/0 was implemented earlier: ask_action() -> Input = io:get_line("Which action do you need? (W)ithdrawl/(D)eposit? "), case Input of [$w | _] -> withdrawl; [$W | _] -> withdrawl; [$d | _] -> deposit; [$D | _] -> deposit; BadInput -> ok = io:format("You can't seem to '~tp' today...~n" "Let's try that again...~n", [BadInput]), ask_action() end. We might change this to... ask_action() -> ask_action(io:get_line("Which action do you need? (W)ithdrawl/(D)eposit? ")). ask_action([$w | _]) -> withdrawl; ask_action([$W | _]) -> withdrawl; ask_action([$d | _]) -> deposit; ask_action([$D | _]) -> deposit; ask_action(BadInput) -> ok = io:format("You can't seem to '~tp' today...~n" "Let's try that again...~n", [BadInput]), ask_action(). The advantages here are not universal, but you will almost certainly see this caseless style of code occasionally. Sometimes it can be really useful, sometimes it can be annoying to read if the calls are always in-lined with huge compositions-as-arguments. For example, if we had removed ask_action/0 and included the user prompt in-line within get_balance/1 it would have been slightly more distracting to the eye. I favor verb-focused readability in the places I can have it; but not everyone does (and that wouldn't have been so bad in this case, anyway -- but there are cases where you can spend a few minutes working out what the hell is going on in the arguments to a function call, and this is super annoying to me, especially when you know that part the original author's reason for making some particular case into a function call was readability). Anyway, I digress. Just be aware that matching allows you to move a case into a function call, and make the match on the head -- and this is a somewhat common technique to "name" case statements (it can often make the trace of a crashed process much easier to read/understand/fix). -Craig From r.wobben@REDACTED Thu Sep 24 07:41:40 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Thu, 24 Sep 2015 07:41:40 +0200 Subject: [erlang-questions] feedback please In-Reply-To: <4932605.QsZFrd1HLM@burrito> References: <56030BE9.1070902@home.nl> <3340168.gZWebXRfOB@burrito> <5603866B.1010306@home.nl> <4932605.QsZFrd1HLM@burrito> Message-ID: <56038D14.30008@home.nl> Op 24-9-2015 om 07:23 schreef zxq9: > On Thursday 24 September 2015 07:13:15 Roelof Wobben wrote: >> Thanks for the feedback. >> >> I understand the code and I see you use error handling. That Is the next >> chapter I need to study. >> So after that chapter I can change it to the code you build. But L like >> the way to handle the input a lot. > That was an example of match-or-crash as error handling. Any place that code would have crashed is a place there wasn't a match, which meant something deeper than our expected input case was faulty -- and in that case it is better to stop execution and crash completely than carry on with crazy data or a screwed up underlying runtime. > > If the next chapter covers "error handling" it probably talks about how crashes are dealt with (and the link/monitor messages that are sent whenever a process crashes/exits) and how to use `try.. catch` to handle exceptions without crashing. There aren't very many places you will want to use try..catch (many folks make a point of trying very hard not to use it, myself included), but it is important to know. > > Have fun! > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > ----- > Geen virus gevonden in dit bericht. > Gecontroleerd door AVG - www.avg.com > Versie: 2015.0.6140 / Virusdatabase: 4419/10690 - datum van uitgifte: 09/24/15 > > Thanks, You are right. I did take a quick look and almost the whole chapter is try .. catch. So match or chrash is better ? Roelof From zxq9@REDACTED Thu Sep 24 07:56:18 2015 From: zxq9@REDACTED (zxq9) Date: Thu, 24 Sep 2015 14:56:18 +0900 Subject: [erlang-questions] feedback please In-Reply-To: <56038D14.30008@home.nl> References: <56030BE9.1070902@home.nl> <4932605.QsZFrd1HLM@burrito> <56038D14.30008@home.nl> Message-ID: <1496684.U5ec38t2A0@burrito> On Thursday 24 September 2015 07:41:40 Roelof Wobben wrote: > Thanks, > > You are right. I did take a quick look and almost the whole chapter is > try .. catch. So match or chrash is better ? It is not so much of a matter of which is "better" -- but that *most* of the time you want to follow the mantra of "let it crash" and this is a very natural fit for Erlang. For example, if I start a process that is supposed to read some configuration settings in its init/1 section before it gets going, and the config data is all screwed up (wrong format, maybe its not even the right data -- like binary image data instead of a settings file or something) then I don't want to try..catch and then write extra code to force the process into some semi-workable state and carry on. I want to just crash right away. In development this might just be something that explodes in my terminal. BOOM! And then I know I need to look at something right away: "Why the devil am I sending image data where a file name is supposed to go?" In production it may be an indicator that some remote resource you normally query has changed its API: "Oh boy, our [very important to your business] service changed their protocol unannounced... again...". In either case the last thing you want to do is have the processes silently carry on and crash *other* processes (or other services!) *way later* for reasons that are now at least one degree removed from the actual problem. But occasionally you will want a try..catch. Sometimes you just want to be able to stay alive long enough to log a very specific/detailed error in a natural way; sometimes you are dealing with an unknown platform and interacting with a user at the same time, and you're not sure what sort of data you might receive -- but if the data is bad you need to stay alive long enough to tell the user about it in detail. In cases like these you *can* accomplish this without try..catch, but its really annoying. In most cases, though, try..catch is much more annoying (and downright dangerous compared to crashing immediately). In my experience try..catch is something you'll know when you need, and that need is infrequent. Almost any time you use try..catch you are not writing for the "success case" (or "happy path") -- the rare times that try..catch fits into the "happy path" not having it would be an annoying handicap. So its not that "crashing is better than try..catch" so much as "standard Erlang practice is to crash on faults, and try..catch is a very useful tool in some specific cases", so you do need to know about it. -Craig From r.wobben@REDACTED Thu Sep 24 08:04:44 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Thu, 24 Sep 2015 08:04:44 +0200 Subject: [erlang-questions] feedback please In-Reply-To: <1496684.U5ec38t2A0@burrito> References: <56030BE9.1070902@home.nl> <4932605.QsZFrd1HLM@burrito> <56038D14.30008@home.nl> <1496684.U5ec38t2A0@burrito> Message-ID: <5603927C.7090200@home.nl> Op 24-9-2015 om 07:56 schreef zxq9: > On Thursday 24 September 2015 07:41:40 Roelof Wobben wrote: >> Thanks, >> >> You are right. I did take a quick look and almost the whole chapter is >> try .. catch. So match or chrash is better ? > It is not so much of a matter of which is "better" -- but that *most* of the time you want to follow the mantra of "let it crash" and this is a very natural fit for Erlang. For example, if I start a process that is supposed to read some configuration settings in its init/1 section before it gets going, and the config data is all screwed up (wrong format, maybe its not even the right data -- like binary image data instead of a settings file or something) then I don't want to try..catch and then write extra code to force the process into some semi-workable state and carry on. I want to just crash right away. > > In development this might just be something that explodes in my terminal. BOOM! And then I know I need to look at something right away: > "Why the devil am I sending image data where a file name is supposed to go?" > > In production it may be an indicator that some remote resource you normally query has changed its API: > "Oh boy, our [very important to your business] service changed their protocol unannounced... again...". > > In either case the last thing you want to do is have the processes silently carry on and crash *other* processes (or other services!) *way later* for reasons that are now at least one degree removed from the actual problem. > > But occasionally you will want a try..catch. Sometimes you just want to be able to stay alive long enough to log a very specific/detailed error in a natural way; sometimes you are dealing with an unknown platform and interacting with a user at the same time, and you're not sure what sort of data you might receive -- but if the data is bad you need to stay alive long enough to tell the user about it in detail. In cases like these you *can* accomplish this without try..catch, but its really annoying. In most cases, though, try..catch is much more annoying (and downright dangerous compared to crashing immediately). > > In my experience try..catch is something you'll know when you need, and that need is infrequent. Almost any time you use try..catch you are not writing for the "success case" (or "happy path") -- the rare times that try..catch fits into the "happy path" not having it would be an annoying handicap. > > So its not that "crashing is better than try..catch" so much as "standard Erlang practice is to crash on faults, and try..catch is a very useful tool in some specific cases", so you do need to know about it. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > ----- > Geen virus gevonden in dit bericht. > Gecontroleerd door AVG - www.avg.com > Versie: 2015.0.6140 / Virusdatabase: 4419/10690 - datum van uitgifte: 09/24/15 > > Thanks for the explanation. it's very clear what point you try to make. Roelof From r.wobben@REDACTED Thu Sep 24 08:22:27 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Thu, 24 Sep 2015 08:22:27 +0200 Subject: [erlang-questions] feedback please In-Reply-To: <5603927C.7090200@home.nl> References: <56030BE9.1070902@home.nl> <4932605.QsZFrd1HLM@burrito> <56038D14.30008@home.nl> <1496684.U5ec38t2A0@burrito> <5603927C.7090200@home.nl> Message-ID: <560396A3.7090500@home.nl> Op 24-9-2015 om 08:04 schreef Roelof Wobben: > Op 24-9-2015 om 07:56 schreef zxq9: >> On Thursday 24 September 2015 07:41:40 Roelof Wobben wrote: >>> Thanks, >>> >>> You are right. I did take a quick look and almost the whole chapter is >>> try .. catch. So match or chrash is better ? >> It is not so much of a matter of which is "better" -- but that *most* >> of the time you want to follow the mantra of "let it crash" and this >> is a very natural fit for Erlang. For example, if I start a process >> that is supposed to read some configuration settings in its init/1 >> section before it gets going, and the config data is all screwed up >> (wrong format, maybe its not even the right data -- like binary image >> data instead of a settings file or something) then I don't want to >> try..catch and then write extra code to force the process into some >> semi-workable state and carry on. I want to just crash right away. >> >> In development this might just be something that explodes in my >> terminal. BOOM! And then I know I need to look at something right away: >> "Why the devil am I sending image data where a file name is supposed >> to go?" >> >> In production it may be an indicator that some remote resource you >> normally query has changed its API: >> "Oh boy, our [very important to your business] service changed their >> protocol unannounced... again...". >> >> In either case the last thing you want to do is have the processes >> silently carry on and crash *other* processes (or other services!) >> *way later* for reasons that are now at least one degree removed from >> the actual problem. >> >> But occasionally you will want a try..catch. Sometimes you just want >> to be able to stay alive long enough to log a very specific/detailed >> error in a natural way; sometimes you are dealing with an unknown >> platform and interacting with a user at the same time, and you're not >> sure what sort of data you might receive -- but if the data is bad >> you need to stay alive long enough to tell the user about it in >> detail. In cases like these you *can* accomplish this without >> try..catch, but its really annoying. In most cases, though, >> try..catch is much more annoying (and downright dangerous compared to >> crashing immediately). >> >> In my experience try..catch is something you'll know when you need, >> and that need is infrequent. Almost any time you use try..catch you >> are not writing for the "success case" (or "happy path") -- the rare >> times that try..catch fits into the "happy path" not having it would >> be an annoying handicap. >> >> So its not that "crashing is better than try..catch" so much as >> "standard Erlang practice is to crash on faults, and try..catch is a >> very useful tool in some specific cases", so you do need to know >> about it. >> >> -Craig >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> ----- >> Geen virus gevonden in dit bericht. >> Gecontroleerd door AVG - www.avg.com >> Versie: 2015.0.6140 / Virusdatabase: 4419/10690 - datum van uitgifte: >> 09/24/15 >> >> > > Thanks for the explanation. it's very clear what point you try to make. > > Roelof > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > ----- > Geen virus gevonden in dit bericht. > Gecontroleerd door AVG - www.avg.com > Versie: 2015.0.6140 / Virusdatabase: 4419/10690 - datum van uitgifte: > 09/24/15 > > On question pops into my mind. You have this code for input the amount : ask_amount() -> Input = io:get_line("How much? "), case scrub_amount(Input) of {ok, Number} -> Number; {error, bad_input} -> ok = io:format("Hrm... '~tp' does not seem to be a number~n" "Now, one more time...~n", [Input]), ask_amount() end. If I want to check if the input is postive do I need to put a if then into the {ok, number} piece. Roelof From zxq9@REDACTED Thu Sep 24 08:45:45 2015 From: zxq9@REDACTED (zxq9) Date: Thu, 24 Sep 2015 15:45:45 +0900 Subject: [erlang-questions] feedback please In-Reply-To: <560396A3.7090500@home.nl> References: <56030BE9.1070902@home.nl> <5603927C.7090200@home.nl> <560396A3.7090500@home.nl> Message-ID: <5970572.k0S0IWtsRU@burrito> On Thursday 24 September 2015 08:22:27 Roelof Wobben wrote: > On question pops into my mind. > > You have this code for input the amount : > > ask_amount() -> > Input = io:get_line("How much? "), > case scrub_amount(Input) of > {ok, Number} -> > Number; > {error, bad_input} -> > ok = io:format("Hrm... '~tp' does not seem to be a number~n" > "Now, one more time...~n", > [Input]), > ask_amount() > end. > > If I want to check if the input is postive do I need to put a if then > into the {ok, number} piece. It depends on where you want to catch it, and this is halfway to being a UI question. Before we had asked the user whether they wanted to deposit or withdraw money. In some really old games this was itself indicated by whether your bank activity was negative or positive. On the other hand, some games have bugs where you can invent money by performing negative deposits or negative withdrawls. So, first you have to ask yourself whether you want to make it illegal for a user to input a negative amount. If you want this to be illegal, then you want to stop things right here at ask_amount/0 and force the user to input something else. You can use guards in case statements the same way you do in function heads. This provides a simpler solution: ask_amount() -> Input = io:get_line("How much? "), case scrub_amount(Input) of {ok, Number} when Number > 0 -> Number; {ok, 0} -> ok = io:format("$0? Giving up on the idea already?~n"), 0; {ok, Number} -> ok = io:format("Hey, you! ~tp is less than no money!~n", [Number]), ask_amount(); {error, bad_input} -> ok = io:format("Hrm... '~tp' does not seem to be a number~n" "Now, one more time...~n", [Input]), ask_amount() end. If you do want to make it legal to act on negative amounts then let it go through, but re-consider your guards in get_balance/2. Not that there is *definitely* some weird case in get_balance/2 now, but almost any time you realize you have an edge case there *very often is* some possible behavior that your program says is correct but is not what you intend. So you have to catch things like this, and stopping it all at the place you validate external input is the best way to do it. Once you accept data into your program you should trust it. Rigorously validate it *once* when it comes in, and from then on trust that the data you are receiving is what you say it is. Dialyzer can actually help with this quite a bit once you learn how to use it (it pays attention to guards as well as -type definitions). There are cases where it is just hopelessly outmatched by reality, but you can usually collect side effects into a smallish number of functions/modules and type the rest of a program in a way that makes you confident that it will work as intended or crash right away. Anyway, don't get bogged down in nitpicking this little example program too much. You have bigger fish to fry. You know most of sequential Erlang already (yes, it is a very tiny language). You need to move on to concurrent Erlang and get your feet wet with spawning, linking, monitoring, etc. It will change the way you think about programs, and this is the key benefit you should be getting out of this experience. -Craig PS: You can spend days arguing/worrying/paralyzed over input validation, UI details, lists/strings VS binaries, maps vs dicts, records VS tuples, blahblahblahblah. All of that is just background noise. Forget about that stuff for now. Keep moving forward and score some early victories in the arena of concurrency -- these little details will become much less mysterious in the course of writing programs you care about, and by that time the discussions about them will make sense (enough sense that you will develop a sense for which of those discussions to ignore). From vasdeveloper@REDACTED Thu Sep 24 09:16:34 2015 From: vasdeveloper@REDACTED (Theepan) Date: Thu, 24 Sep 2015 12:46:34 +0530 Subject: [erlang-questions] feedback please In-Reply-To: <56038D14.30008@home.nl> References: <56030BE9.1070902@home.nl> <3340168.gZWebXRfOB@burrito> <5603866B.1010306@home.nl> <4932605.QsZFrd1HLM@burrito> <56038D14.30008@home.nl> Message-ID: >> So match or chrash is better ? If something must not happen, and if happens crash. For example, for a programers error it should crash. For off schema data it should crash. For un-matching case clauses it should crash. If something has a chance of happening, although it is expected, use try-catch. Most of the time it is used at the inlets of a program where unexpected inputs are possible. For example, user input. And it is not uncommon to use try-catch and forced-crash together. Theepan On Thu, Sep 24, 2015 at 11:11 AM, Roelof Wobben wrote: > Op 24-9-2015 om 07:23 schreef zxq9: > >> On Thursday 24 September 2015 07:13:15 Roelof Wobben wrote: >> >>> Thanks for the feedback. >>> >>> I understand the code and I see you use error handling. That Is the next >>> chapter I need to study. >>> So after that chapter I can change it to the code you build. But L like >>> the way to handle the input a lot. >>> >> That was an example of match-or-crash as error handling. Any place that >> code would have crashed is a place there wasn't a match, which meant >> something deeper than our expected input case was faulty -- and in that >> case it is better to stop execution and crash completely than carry on with >> crazy data or a screwed up underlying runtime. >> >> If the next chapter covers "error handling" it probably talks about how >> crashes are dealt with (and the link/monitor messages that are sent >> whenever a process crashes/exits) and how to use `try.. catch` to handle >> exceptions without crashing. There aren't very many places you will want to >> use try..catch (many folks make a point of trying very hard not to use it, >> myself included), but it is important to know. >> >> Have fun! >> -Craig >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> ----- >> Geen virus gevonden in dit bericht. >> Gecontroleerd door AVG - www.avg.com >> Versie: 2015.0.6140 / Virusdatabase: 4419/10690 - datum van uitgifte: >> 09/24/15 >> >> >> > Thanks, > > You are right. I did take a quick look and almost the whole chapter is try > .. catch. So match or chrash is better ? > > Roelof > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennethlakin@REDACTED Thu Sep 24 09:29:52 2015 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Thu, 24 Sep 2015 00:29:52 -0700 Subject: [erlang-questions] feedback please In-Reply-To: References: <56030BE9.1070902@home.nl> <3340168.gZWebXRfOB@burrito> <5603866B.1010306@home.nl> <4932605.QsZFrd1HLM@burrito> <56038D14.30008@home.nl> Message-ID: <5603A670.3060902@gmail.com> On 09/24/2015 12:16 AM, Theepan wrote: > For off schema data it should crash. > If something has a chance of happening, although it > is expected, use try-catch. This isn't a universal rule. I have a section in one of my programs where one of a *very* wide array of errors can come back from an operation on external system. For the *few* errors that I know how to handle and *expect* to happen, I have matching case clauses. For the *rest* of the errors that I either don't know how to handle, are aren't really expecting, I have a catch-all clause with a WARN print that spews details to the system log. If I hit an unexpected error, I figure out if I need to and how to handle it, then add a clause in the case statement. In my case, for this part of the software, it's better to keep running than it is to crash because I hit one of the billion errors that I never expected to have to handle. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From r.wobben@REDACTED Thu Sep 24 09:03:26 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Thu, 24 Sep 2015 09:03:26 +0200 Subject: [erlang-questions] feedback please In-Reply-To: <5970572.k0S0IWtsRU@burrito> References: <56030BE9.1070902@home.nl> <5603927C.7090200@home.nl> <560396A3.7090500@home.nl> <5970572.k0S0IWtsRU@burrito> Message-ID: <5603A03E.8020800@home.nl> Op 24-9-2015 om 08:45 schreef zxq9: > On Thursday 24 September 2015 08:22:27 Roelof Wobben wrote: >> On question pops into my mind. >> >> You have this code for input the amount : >> >> ask_amount() -> >> Input = io:get_line("How much? "), >> case scrub_amount(Input) of >> {ok, Number} -> >> Number; >> {error, bad_input} -> >> ok = io:format("Hrm... '~tp' does not seem to be a number~n" >> "Now, one more time...~n", >> [Input]), >> ask_amount() >> end. >> >> If I want to check if the input is postive do I need to put a if then >> into the {ok, number} piece. > It depends on where you want to catch it, and this is halfway to being a UI question. Before we had asked the user whether they wanted to deposit or withdraw money. In some really old games this was itself indicated by whether your bank activity was negative or positive. On the other hand, some games have bugs where you can invent money by performing negative deposits or negative withdrawls. > > So, first you have to ask yourself whether you want to make it illegal for a user to input a negative amount. If you want this to be illegal, then you want to stop things right here at ask_amount/0 and force the user to input something else. You can use guards in case statements the same way you do in function heads. This provides a simpler solution: > > ask_amount() -> > Input = io:get_line("How much? "), > case scrub_amount(Input) of > {ok, Number} when Number > 0 -> > Number; > {ok, 0} -> > ok = io:format("$0? Giving up on the idea already?~n"), > 0; > {ok, Number} -> > ok = io:format("Hey, you! ~tp is less than no money!~n", > [Number]), > ask_amount(); > {error, bad_input} -> > ok = io:format("Hrm... '~tp' does not seem to be a number~n" > "Now, one more time...~n", > [Input]), > ask_amount() > end. > > If you do want to make it legal to act on negative amounts then let it go through, but re-consider your guards in get_balance/2. Not that there is *definitely* some weird case in get_balance/2 now, but almost any time you realize you have an edge case there *very often is* some possible behavior that your program says is correct but is not what you intend. So you have to catch things like this, and stopping it all at the place you validate external input is the best way to do it. Once you accept data into your program you should trust it. Rigorously validate it *once* when it comes in, and from then on trust that the data you are receiving is what you say it is. > > Dialyzer can actually help with this quite a bit once you learn how to use it (it pays attention to guards as well as -type definitions). There are cases where it is just hopelessly outmatched by reality, but you can usually collect side effects into a smallish number of functions/modules and type the rest of a program in a way that makes you confident that it will work as intended or crash right away. > > Anyway, don't get bogged down in nitpicking this little example program too much. You have bigger fish to fry. You know most of sequential Erlang already (yes, it is a very tiny language). You need to move on to concurrent Erlang and get your feet wet with spawning, linking, monitoring, etc. It will change the way you think about programs, and this is the key benefit you should be getting out of this experience. > > -Craig > > PS: You can spend days arguing/worrying/paralyzed over input validation, UI details, lists/strings VS binaries, maps vs dicts, records VS tuples, blahblahblahblah. All of that is just background noise. Forget about that stuff for now. Keep moving forward and score some early victories in the arena of concurrency -- these little details will become much less mysterious in the course of writing programs you care about, and by that time the discussions about them will make sense (enough sense that you will develop a sense for which of those discussions to ignore). > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > ----- > Geen virus gevonden in dit bericht. > Gecontroleerd door AVG - www.avg.com > Versie: 2015.0.6140 / Virusdatabase: 4419/10690 - datum van uitgifte: 09/24/15 > > that quick to concencurry. The book has a chapter of error handling, maps , binaries, dialyzer and compiling with a make file, So in my oponion I not done with sequantial erlang. I think it wll take me a week before Im there. Roelof From vasdeveloper@REDACTED Thu Sep 24 09:50:30 2015 From: vasdeveloper@REDACTED (Theepan) Date: Thu, 24 Sep 2015 13:20:30 +0530 Subject: [erlang-questions] feedback please In-Reply-To: <5603A670.3060902@gmail.com> References: <56030BE9.1070902@home.nl> <3340168.gZWebXRfOB@burrito> <5603866B.1010306@home.nl> <4932605.QsZFrd1HLM@burrito> <56038D14.30008@home.nl> <5603A670.3060902@gmail.com> Message-ID: >> In my case, for this part of the software, it's better to keep running >> than it is to crash because I hit one of the billion errors that I never >> expected to have to handle. Crashing can happen at the process level or system system level. It depends. For a configuration error, you will have to crash the whole system. Theepan On Thu, Sep 24, 2015 at 12:59 PM, Kenneth Lakin wrote: > On 09/24/2015 12:16 AM, Theepan wrote: > > For off schema data it should crash. > > If something has a chance of happening, although it > > is expected, use try-catch. > > This isn't a universal rule. > > I have a section in one of my programs where one of a *very* wide array > of errors can come back from an operation on external system. For the > *few* errors that I know how to handle and *expect* to happen, I have > matching case clauses. For the *rest* of the errors that I either don't > know how to handle, are aren't really expecting, I have a catch-all > clause with a WARN print that spews details to the system log. If I hit > an unexpected error, I figure out if I need to and how to handle it, > then add a clause in the case statement. > > In my case, for this part of the software, it's better to keep running > than it is to crash because I hit one of the billion errors that I never > expected to have to handle. > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Thu Sep 24 09:55:21 2015 From: zxq9@REDACTED (zxq9) Date: Thu, 24 Sep 2015 16:55:21 +0900 Subject: [erlang-questions] feedback please In-Reply-To: <5603A03E.8020800@home.nl> References: <56030BE9.1070902@home.nl> <5970572.k0S0IWtsRU@burrito> <5603A03E.8020800@home.nl> Message-ID: <4677610.TleYrx7IUj@burrito> On Thursday 24 September 2015 09:03:26 Roelof Wobben wrote: > that quick to concencurry. The book has a chapter of error handling, > maps , binaries, dialyzer and compiling with a make file, > So in my oponion I not done with sequantial erlang. I think it wll take > me a week before Im there. Nah, maps, binaries and dialyzer are not huge subjects by themselves. Well, dialyzer can be, but I doubt it is treated in full detail in the book. Maps will be familiar to you if you've ever used a language that has hashes/maps/dictionaries (or whatever key:value store) -- or if you've used Erlang dicts. The functional interface to maps (http://www.erlang.org/doc/man/maps.html) and dicts (http://www.erlang.org/doc/man/dict.html) is very similar, and neither is very far from the spirit of many of the lists module functions (http://www.erlang.org/doc/man/lists.html). Your initial understanding of binaries will focus on two things: 1- The difference between lists and binaries (in particular, binary strings and lists-as-strings) 2- Erlang's magical syntax for binaries (which is really quite nice) Dialyzer is interesting. Hopefully you will get a solid introduction to it here. But, like many of the advanced Erlang tools, it is something that will only start to seem painfully relevant to your life once you've worked on a project long enough to confuse yourself with it (and people are depending on it to work). You'll probably do this once, and then sit down and really learn how and when to use Dialyzer (among other tools -- there are a ton of really slick things available you'll learn about as you use Erlang more). Make is its own little world and learning it could be its own week. But you won't be going all that far into it in an Erlang book, I imagine. Make is just a very nice little tool to automate a lot of the command line drugery of building things that depend on other things. It is not even a part of Erlang, just a very comfortable script-my-build-steps-away tool once you get to know how it works (and not just for Erlang, for nearly anything). So no, you're not that far away. Don't despair. :-) -Craig From kennethlakin@REDACTED Thu Sep 24 10:03:33 2015 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Thu, 24 Sep 2015 01:03:33 -0700 Subject: [erlang-questions] feedback please In-Reply-To: References: <56030BE9.1070902@home.nl> <3340168.gZWebXRfOB@burrito> <5603866B.1010306@home.nl> <4932605.QsZFrd1HLM@burrito> <56038D14.30008@home.nl> <5603A670.3060902@gmail.com> Message-ID: <5603AE55.2080103@gmail.com> On 09/24/2015 12:50 AM, Theepan wrote: > Crashing can happen at the process level or system system level. Yes. Erlang has *very* nice fault-containment tools. > For a configuration error, you will have to crash the whole > system. Or... log a warning and fall back to hard-coded defaults. It *really* depends on the nature of the error and the system. You keep speaking in absolutes, but the "reasonable solution" space isn't quite *that* small. :) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From vasdeveloper@REDACTED Thu Sep 24 13:08:46 2015 From: vasdeveloper@REDACTED (Theepan) Date: Thu, 24 Sep 2015 16:38:46 +0530 Subject: [erlang-questions] feedback please In-Reply-To: <5603AE55.2080103@gmail.com> References: <56030BE9.1070902@home.nl> <3340168.gZWebXRfOB@burrito> <5603866B.1010306@home.nl> <4932605.QsZFrd1HLM@burrito> <56038D14.30008@home.nl> <5603A670.3060902@gmail.com> <5603AE55.2080103@gmail.com> Message-ID: >> You keep speaking in absolutes, but the "reasonable solution" space isn't quite *that* small. Certainly, programing is a democratic space, and you have many ways to get things done. It is also more religious. What is right in a small system may be wrong in a big system, and vice versa. I just express the ways I do things, which may be helpful to somebody who value it. It also helps me in knowing other views, and if I value them, I will stand corrected. Theepan On Thu, Sep 24, 2015 at 1:33 PM, Kenneth Lakin wrote: > On 09/24/2015 12:50 AM, Theepan wrote: > > Crashing can happen at the process level or system system level. > > Yes. Erlang has *very* nice fault-containment tools. > > > For a configuration error, you will have to crash the whole > > system. > > Or... log a warning and fall back to hard-coded defaults. It *really* > depends on the nature of the error and the system. You keep speaking in > absolutes, but the "reasonable solution" space isn't quite *that* small. :) > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.wobben@REDACTED Thu Sep 24 13:56:53 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Thu, 24 Sep 2015 13:56:53 +0200 Subject: [erlang-questions] feedback please In-Reply-To: References: <56030BE9.1070902@home.nl> <3340168.gZWebXRfOB@burrito> <5603866B.1010306@home.nl> <4932605.QsZFrd1HLM@burrito> <56038D14.30008@home.nl> <5603A670.3060902@gmail.com> <5603AE55.2080103@gmail.com> Message-ID: <5603E505.40801@home.nl> An HTML attachment was scrubbed... URL: From r.wobben@REDACTED Thu Sep 24 14:45:06 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Thu, 24 Sep 2015 14:45:06 +0200 Subject: [erlang-questions] Where is the syntax error. Message-ID: <5603F052.8070704@home.nl> Hello, For a exercise I had to write a function which keeps track how long a anymous function takes with the help of time:now. So I have this : -module(time). -export([elapsed_time/1]). elapsed_time(F) -> Begintime = erlang:monotic_time(), F(), Endtime = erlang:monotonic_time(), Time = erlang:convert_timeunit(Endtime - Begintime,native , micro_seconds), Time. Now I want to test it in erl. But then I see this output: time:elapsed_time(Answer = fun() -> io:format("Sucess") end. ). * 1: syntax error before: '.' Can someone give me a tip where the syntax errror is. Roelof From zxq9@REDACTED Thu Sep 24 14:47:48 2015 From: zxq9@REDACTED (zxq9) Date: Thu, 24 Sep 2015 21:47:48 +0900 Subject: [erlang-questions] Where is the syntax error. In-Reply-To: <5603F052.8070704@home.nl> References: <5603F052.8070704@home.nl> Message-ID: <2510111.5bg7a4AXeJ@changa> On 2015?9?24? ??? 14:45:06 Roelof Wobben wrote: > Now I want to test it in erl. > But then I see this output: > > time:elapsed_time(Answer = fun() -> io:format("Sucess") end. ). > * 1: syntax error before: '.' > > Can someone give me a tip where the syntax errror is. The message is confusing because you have two periods. This is also the error. Try this way: time:elapsed_time(Answer = fun() -> io:format("Sucess") end). -Craig From hugo@REDACTED Thu Sep 24 14:51:00 2015 From: hugo@REDACTED (Hugo Mills) Date: Thu, 24 Sep 2015 12:51:00 +0000 Subject: [erlang-questions] Where is the syntax error. In-Reply-To: <5603F052.8070704@home.nl> References: <5603F052.8070704@home.nl> Message-ID: <20150924125100.GA3749@carfax.org.uk> On Thu, Sep 24, 2015 at 02:45:06PM +0200, Roelof Wobben wrote: > Hello, > > For a exercise I had to write a function which keeps track how long > a anymous function takes with the help of time:now. > > So I have this : > > -module(time). > > -export([elapsed_time/1]). > > elapsed_time(F) -> > Begintime = erlang:monotic_time(), > F(), > Endtime = erlang:monotonic_time(), > Time = erlang:convert_timeunit(Endtime - Begintime,native , > micro_seconds), > Time. > > > Now I want to test it in erl. > But then I see this output: > > time:elapsed_time(Answer = fun() -> io:format("Sucess") end. ). > * 1: syntax error before: '.' > > Can someone give me a tip where the syntax errror is. Well, like the error message says, it's something to do with a dot. You've got two dots in that line of code. Which one is wrong? Hint: a dot is the end of an expression to be evaluated (in the shell) or the end of a function definition (in a module). Hugo. -- Hugo Mills | 2 + 2 = 5, for sufficiently large values of 2. hugo@REDACTED carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 | -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From zxq9@REDACTED Thu Sep 24 14:54:52 2015 From: zxq9@REDACTED (zxq9) Date: Thu, 24 Sep 2015 21:54:52 +0900 Subject: [erlang-questions] Where is the syntax error. In-Reply-To: <20150924125100.GA3749@carfax.org.uk> References: <5603F052.8070704@home.nl> <20150924125100.GA3749@carfax.org.uk> Message-ID: <1904714.MRkcGXILy0@changa> On 2015?9?24? ??? 12:51:00 Hugo Mills wrote: > On Thu, Sep 24, 2015 at 02:45:06PM +0200, Roelof Wobben wrote: > > Can someone give me a tip where the syntax errror is. > > Well, like the error message says, it's something to do with a dot. > > You've got two dots in that line of code. Which one is wrong? > Hint: a dot is the end of an expression to be evaluated (in the shell) > or the end of a function definition (in a module). Bah, this was a more fun answer. I gave it up without thinking. Debugging all day makes me bad at find-it-yourself hints... -Craig From r.wobben@REDACTED Thu Sep 24 14:57:07 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Thu, 24 Sep 2015 14:57:07 +0200 Subject: [erlang-questions] Where is the syntax error. In-Reply-To: <1904714.MRkcGXILy0@changa> References: <5603F052.8070704@home.nl> <20150924125100.GA3749@carfax.org.uk> <1904714.MRkcGXILy0@changa> Message-ID: <5603F323.2090008@home.nl> Op 24-9-2015 om 14:54 schreef zxq9: > On 2015?9?24? ??? 12:51:00 Hugo Mills wrote: >> On Thu, Sep 24, 2015 at 02:45:06PM +0200, Roelof Wobben wrote: >>> Can someone give me a tip where the syntax errror is. >> Well, like the error message says, it's something to do with a dot. >> >> You've got two dots in that line of code. Which one is wrong? >> Hint: a dot is the end of an expression to be evaluated (in the shell) >> or the end of a function definition (in a module). > Bah, this was a more fun answer. I gave it up without thinking. > > Debugging all day makes me bad at find-it-yourself hints... > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > ----- > Geen virus gevonden in dit bericht. > Gecontroleerd door AVG - www.avg.com > Versie: 2015.0.6140 / Virusdatabase: 4419/10690 - datum van uitgifte: 09/24/15 Thanks both, I will write now a nice expression which give the answer. Roelof From garry@REDACTED Thu Sep 24 17:01:37 2015 From: garry@REDACTED (Garry Hodgson) Date: Thu, 24 Sep 2015 11:01:37 -0400 Subject: [erlang-questions] Where is the syntax error. In-Reply-To: <1904714.MRkcGXILy0@changa> References: <5603F052.8070704@home.nl> <20150924125100.GA3749@carfax.org.uk> <1904714.MRkcGXILy0@changa> Message-ID: <56041051.3030103@research.att.com> On 09/24/2015 08:54 AM, zxq9 wrote: > Debugging all day makes me bad at find-it-yourself hints... -Craig _ I like to split my time between putting bugs in and taking them out. -- Garry Hodgson Lead Member of Technical Staff AT&T Chief Security Office (CSO) "This e-mail and any files transmitted with it are AT&T property, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited." From r.wobben@REDACTED Thu Sep 24 18:05:13 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Thu, 24 Sep 2015 18:05:13 +0200 Subject: [erlang-questions] Where is the syntax error. In-Reply-To: <56041051.3030103@research.att.com> References: <5603F052.8070704@home.nl> <20150924125100.GA3749@carfax.org.uk> <1904714.MRkcGXILy0@changa> <56041051.3030103@research.att.com> Message-ID: <56041F39.9010500@home.nl> Op 24-9-2015 om 17:01 schreef Garry Hodgson: > On 09/24/2015 08:54 AM, zxq9 wrote: >> Debugging all day makes me bad at find-it-yourself hints... -Craig _ > > I like to split my time between putting bugs in and taking them out. > Thanks, One thing I do not see. If I want to run this with a function with a argument. -module(time). -export([elapsed_time/1]). elapsed_time(F) -> Begintime = erlang:monotonic_time(), F(), Endtime = erlang:monotonic_time(), Time = erlang:convert_time_unit(Endtime - Begintime,native , micro_seconds), io:format("This took ~w microseconds", [Time]). This is the wrong way to do this : time:elapsed_time((Answer = fun(x) -> x + 2 end )2). Can I solve this by chancing the function to this elapsed_time(F, arg) and then do F(arg) ? Roelof time:elapsed_time((Answer = fun(x) -> x + 2 end )2). time:elapsed_time((Answer = fun(x) -> x + 2 end )2). From hugo@REDACTED Thu Sep 24 18:17:01 2015 From: hugo@REDACTED (Hugo Mills) Date: Thu, 24 Sep 2015 16:17:01 +0000 Subject: [erlang-questions] Where is the syntax error. In-Reply-To: <56041F39.9010500@home.nl> References: <5603F052.8070704@home.nl> <20150924125100.GA3749@carfax.org.uk> <1904714.MRkcGXILy0@changa> <56041051.3030103@research.att.com> <56041F39.9010500@home.nl> Message-ID: <20150924161701.GB3749@carfax.org.uk> On Thu, Sep 24, 2015 at 06:05:13PM +0200, Roelof Wobben wrote: > Op 24-9-2015 om 17:01 schreef Garry Hodgson: > >On 09/24/2015 08:54 AM, zxq9 wrote: > >>Debugging all day makes me bad at find-it-yourself hints... -Craig _ > > > >I like to split my time between putting bugs in and taking them out. > > > > Thanks, > > One thing I do not see. > > If I want to run this with a function with a argument. > > -module(time). > > -export([elapsed_time/1]). > > elapsed_time(F) -> > Begintime = erlang:monotonic_time(), > F(), > Endtime = erlang:monotonic_time(), > Time = erlang:convert_time_unit(Endtime - Begintime,native , > micro_seconds), > io:format("This took ~w microseconds", [Time]). > > This is the wrong way to do this : > > time:elapsed_time((Answer = fun(x) -> x + 2 end )2). > > Can I solve this by chancing the function to this elapsed_time(F, > arg) and then do F(arg) ? Yes, exactly. Although you probably want to use Arg (variable), not arg (atom) :) Hugo. > Roelof > > time:elapsed_time((Answer = fun(x) -> x + 2 end )2). > > time:elapsed_time((Answer = fun(x) -> x + 2 end )2). > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Hugo Mills | Putting U back in Honor, Valor, and Trth. hugo@REDACTED carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 | -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From n.oxyde@REDACTED Thu Sep 24 19:12:15 2015 From: n.oxyde@REDACTED (Anthony Ramine) Date: Thu, 24 Sep 2015 19:12:15 +0200 Subject: [erlang-questions] Abstract code in .beam files In-Reply-To: References: Message-ID: <9C5F0A8F-584E-4F8A-B833-3D4E94FD74C8@gmail.com> Le 23 sept. 2015 ? 17:51, Robert Virding a ?crit : > Anyway that was my idea. I see I will have to wait until the Erlang compiler is fixed to do this. And dialyzer. I'm not sure I understand your email, you mostly paraphrased Jos?'s. Regards. From send2philip@REDACTED Thu Sep 24 23:58:26 2015 From: send2philip@REDACTED (Philip Clarke) Date: Thu, 24 Sep 2015 22:58:26 +0100 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: Hi Lukas, One common use case for tracing I would like is the ability to trace a function call for just the next request in the system. Currently the PidSpec in erlang:trace/3 allows for pid, existing, new or all. We would use pid if we knew the pid of the next short lived process to call a function (but we don't). We cannot use new either as tracing a function call for all new processes might mean that this function is traced 1000s times per second. What I would like is some new PidSpec like 'next_pid' which would specify that I want to trace all new processes until the function call I am tracing has been triggered. Once the trace is triggered then it should turn off tracing on all processes. e.g. dbg:p(next_pid, call). dbg:tp(M, F, x). This would be a great help in my work where quite often I want to see the typical format of a request from a client and just capturing 1 request is enough. I believe that libraries such as redbug have this kind of ability, but perhaps the feature is fundamental enough to include as part of trace ? Thanks for considering Philip On Wed, Sep 23, 2015 at 4:42 PM, ILYA Khlopotov wrote: > +1 for multiple independent tracers. > > It would be also very helpful to support following: > 1. Cluster avare seq_trace. Currently trace token is not send over erlang > distribution protocol. It could be solved if new control message is added > to distribution protocol for example. > 2. There are a lot of different formats for trace messages and any tracing > library need to be able to understand all of them. Almost any tracing > librbary has to do something like: > > https://github.com/erlang/otp/blob/a2670f0822fc6729df956c8ec8c381340ff0a5fb/lib/et/src/et_selector.erl#L218 > > https://github.com/erlang/otp/blob/a2670f0822fc6729df956c8ec8c381340ff0a5fb/lib/et/src/et_selector.erl#L268 > > https://github.com/erlang/otp/blob/a2670f0822fc6729df956c8ec8c381340ff0a5fb/lib/runtime_tools/src/dbg.erl#L942 > https://github.com/ferd/recon/blob/master/src/recon_trace.erl#L482 > It is problematic though since tuples are of different size. > Would it be better to send this info as an erlang record which anyone > could import and use? It could be two records as well. Since every trace > message has a meta info which describes where and what happend and list of > arguments of the message. I.e. something like > meta: { > pid, > ts, > type, > serial, > ctx > } > > event: { > meta :: #meta{}, > args :: [], > sequence = 0 > } > > 3. For seq_trace in particular it could be useful to be able to inject > some context into the tracing process. Consider following use case. Let's > say we have a notion of sessions. First user logins in order to start > session, then he issues some commands using session_cookie as a mean to > prove his identity. Since calls to login and command are independent http > requests there is no way to link them together. We could have following: > maybe_trace_login(_User, _Cookie) -> > ok. %% function we use for dbg:fun2ms > login(User, Password) -> > %% check password raise if it is invalid > Cookie = ..., > maybe_trace_login(User, Cookie), > ... > command(Cookie, Command, Args) -> > %% check Cookie raise if unknown > do_command(Command, Args), %% we want to get trace of this > ... > However if we could store {User, Cookie} in the context of a tracer > process from match spec. > We could produce nice trace messages for calls to command which would > include user name. > This is the reason why ctx field is added in #meta{}. > We also would need to be able to initiate tracing of command when cookie > match somehow. > > BR, > ILYA > > On Mon, Sep 21, 2015 at 1:52 AM, Lukas Larsson wrote: > >> Hello everyone. >> >> As you may know, one of the OTP teams focus areas for the coming year is >> make tracing better. At the moment we are gathering ideas and attempting to >> put together a vision of what we would like to have, before deciding what >> we can make. >> >> I'm pretty sure that many of you have much more experience with using >> Erlang tracing while developing and in production than I do, which is we >> would love to have your input as to what you would like to change about >> tracing. >> >> To set the scope of the discussion, when I say tracing I include; erlang >> tracing, dtrace/systemtap, trace outputs (stdout/file/IP), filtering >> through match specs, sequence tracing, tool integration (dbg, fprof, >> redbug, recon to mention some) and probably more. >> >> To start the discussion, here are a few of my ideas in no particular >> order: >> * Allow multiple tracers. Today only one port/process can be the >> receiver of trace data. >> * Create a couple of scalable high throughput tracing output backends >> with different overflow mechanics. Today all tracing is funneled through >> one bottleneck and has no overflow handling at all. >> * Expose vm probes (today dtrace probes) to the erlang tracer. >> * Better integration of dtrace/lttngt/systemtap into the erlang trace. >> * Allow the erlang tracer to be an Erlang callback module. Today only >> ports/processes are allowed. >> * Optimize trace output to file/ip. Maybe use something like the Common >> Trace Format ( >> http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.md;hb=master), >> instead of the term_to_binary that we have today. >> * Write much much better documentation for dbg :) >> >> We are looking for feedback from both beginners as well as seasoned >> veterans to make erlang tracing the best it can be. So if you have any >> thoughts or ideas, join the discussion to make Erlang tracing better for >> you and everyone else. >> >> Thanks in advance, >> Lukas >> Erlang/OTP team >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Sep 25 02:50:41 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 25 Sep 2015 12:50:41 +1200 Subject: [erlang-questions] feedback please In-Reply-To: <56030BE9.1070902@home.nl> References: <56030BE9.1070902@home.nl> Message-ID: <17A6AAB5-5937-4B9D-A72A-07973AD7B170@cs.otago.ac.nz> First, let's clean up the code you posted, which seems to have been garbled a bit in transmission. -module(balance). -export([get_balance/1]). get_balance(Balance) -> io:format("~n~n"), io:format("Your account balance is $~w.~n", [Balance]), Entered_action = io:get_line( "Which Action do you need (w)ithdraw/(d)eposit? "), Action = string:strip(Entered_action, right, $\n), Entered_amount = io:get_line( "What Amount do you want to withdraw/deposit? "), Amount2 = string:to_integer(string:strip(Entered_amount, right, $\n)), Amount = element(1, Amount2), get_balance(Balance, Action, Amount). get_balance(0, "w", _) -> io:format("You have no funds to withdraw.~n"), get_balance(0); get_balance(Balance, "w", Amount) when Amount > Balance -> io:format("You cannot withdraw more than you have.~n"), get_balance(Balance); get_balance(Balance, "w", Amount) -> get_balance(Balance - Amount); get_balance(Balance, "d", Amount) -> get_balance(Balance + Amount). Second, let's look at the names. What does "get" mean? NOTHING. So why is it even there? If anything "get_balance" implies something that *reports* a balance *without changing it*, which makes "get" worse than useless. Call it something like "maintain_balance/1." Now, let's look at your sanity checking. - You ensure that the *change* to a balance is an integer, but you nowhere check that the *initial* balance is one. - You check that someone doesn't withdraw more than they have, but you do not check that they do not deposit a large negative amount. Your account balance is $1050. Which Action do you need (w)ithdraw/(d)eposit? d What Amount do you want to withdraw/deposit? -2000 Finally, it's nice to be able to stop the loop without having to crash it. There are several further improvements possible, but at this point here's what we get: -module(balance). -export([maintain_balance/1]). maintain_balance(Initial_Balance) when is_integer(Initial_Balance), Initial_Balance >= 0 -> loop(Initial_Balance). loop(Balance) -> io:format("~nYour account balance is $~w.~n", [Balance]), case prompted_character( "Which Action do you need (w)ithdraw/(d)eposit/(q)uit? ") of $q -> Balance ; $w -> Amount = prompted_integer( "What amount do you want to withdraw? "), loop(withdraw(Balance, Amount)) ; $d -> Amount = prompted_integer( "What amount do you want to deposit? "), loop(deposit(Balance, Amount)) end. prompted_character(Prompt) -> hd(io:get_line(Prompt)). prompted_integer(Prompt) -> Amount = list_to_integer(string:strip(io:get_line(Prompt), right, $\n)), if Amount < 0 -> io:format("Amounts should not be negative.~n"), 0 ; Amount >= 0 -> Amount end. withdraw(Balance, Amount) when Balance >= Amount -> Balance - Amount; withdraw(Balance, _) -> io:format("Insufficient funds.~n"), Balance. deposit(Balance, Amount) -> Balance + Amount. From ok@REDACTED Fri Sep 25 02:53:58 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 25 Sep 2015 12:53:58 +1200 Subject: [erlang-questions] feedback please In-Reply-To: <5235970.BP7GLh1TPO@burrito> References: <56030BE9.1070902@home.nl> <3340168.gZWebXRfOB@burrito> <5603866B.1010306@home.nl> <5235970.BP7GLh1TPO@burrito> Message-ID: <4B64CD01-BBEF-4995-BDA8-2DC17C32FAB8@cs.otago.ac.nz> On 24/09/2015, at 5:39 pm, zxq9 quoted: > > ask_action() -> > Input = io:get_line("Which action do you need? (W)ithdrawl/(D)eposit? "), > case Input of > [$w | _] -> withdrawl; > [$W | _] -> withdrawl; > [$d | _] -> deposit; > [$D | _] -> deposit; > BadInput -> > ok = io:format("You can't seem to '~tp' today...~n" > "Let's try that again...~n", > [BadInput]), > ask_action() > end. It's withdrawAl, the action of withdrawing, not speaking in a funny way. I'd be more concerned about moving the *string literals* out into a tuple of strings (for localisation, spelling correction, &c) than about moving the case expression out. From ok@REDACTED Fri Sep 25 02:58:38 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 25 Sep 2015 12:58:38 +1200 Subject: [erlang-questions] feedback please In-Reply-To: <1496684.U5ec38t2A0@burrito> References: <56030BE9.1070902@home.nl> <4932605.QsZFrd1HLM@burrito> <56038D14.30008@home.nl> <1496684.U5ec38t2A0@burrito> Message-ID: On 24/09/2015, at 5:56 pm, zxq9 wrote: > On Thursday 24 September 2015 07:41:40 Roelof Wobben wrote: >> Thanks, >> >> You are right. I did take a quick look and almost the whole chapter is >> try .. catch. So match or chrash is better ? > > It is not so much of a matter of which is "better" -- but that *most* of the time you want to follow the mantra of "let it crash" and this is a very natural fit for Erlang. It must be said that *this* specific example is one where crashing is an extremely bad thing. If you went to your ATM and a process in the bank's machines had crashed and completely lost track of your money, you would be EXTREMELY unhappy. Lawyer-level unhappy. A lot of values can be thrown away and no harm done. But if a process is supposed to be managing persistent state there may be serious legal penalties attached to losing it. This would make a nice little DETS example: does it really make sense to keep the balance in a persistent store? Too right it does! From ok@REDACTED Fri Sep 25 04:14:28 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 25 Sep 2015 14:14:28 +1200 Subject: [erlang-questions] Where is the syntax error. In-Reply-To: <5603F052.8070704@home.nl> References: <5603F052.8070704@home.nl> Message-ID: On 25/09/2015, at 12:45 am, Roelof Wobben wrote: > Now I want to test it in erl. > But then I see this output: > > time:elapsed_time(Answer = fun() -> io:format("Sucess") end. ). > * 1: syntax error before: '.' > > Can someone give me a tip where the syntax errror is. The error message says "before: '.'" So look for dots. I can see two of them. v v time:elapsed_time(Answer = fun() -> io:format("Sucess") end. ). ^ ^ So it has to be one of those places. Hint: when is it legal to have a full stop *inside* a function definition? PS: it's "Success", not "Sucess". From gpadovani@REDACTED Thu Sep 24 22:35:56 2015 From: gpadovani@REDACTED (Gianluca Padovani) Date: Thu, 24 Sep 2015 22:35:56 +0200 Subject: [erlang-questions] Elixir/Erlang Meetup in Bologna Italy on 17th October Message-ID: Hello, I'm happy to announce that also in Italy we are trying to grow the community. We have a full day meetup with a lot of interesting talks about Elixir and Erlang. This is our site http://www.italian-elixir.org/ and you can register here: https://www.eventbrite.it/e/biglietti-bologna-elixir-erlang-meetup-18440536140 I hope to see you there!!! bye Gianluca -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Sep 25 04:20:46 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 25 Sep 2015 14:20:46 +1200 Subject: [erlang-questions] Where is the syntax error. In-Reply-To: <56041F39.9010500@home.nl> References: <5603F052.8070704@home.nl> <20150924125100.GA3749@carfax.org.uk> <1904714.MRkcGXILy0@changa> <56041051.3030103@research.att.com> <56041F39.9010500@home.nl> Message-ID: On 25/09/2015, at 4:05 am, Roelof Wobben wrote: > elapsed_time(F) -> > Begintime = erlang:monotonic_time(), > F(), > Endtime = erlang:monotonic_time(), ... Here you are calling F with *NO* arguments. > time:elapsed_time((Answer = fun(x) -> x + 2 end )2). There are four things wrong with that. (1) What's with end)2)? What's the 2 doing there? (2) You have written a function of ONE argument, but the module wants a function of NO arguments. Writing 'x' as the argument is not an error in itself, but you will seldom have reason to write a function that accepts *only* the literal atom 'x' as argument and nothing else. (3) x + 2 cannot possibly work. x is an atom. Erlang has no idea what arithmetic on atoms might mean. You *probably* meant fun (X) -> X + 2 end. (4) You have Answer = ... which is not a syntax error but is both pointless (because the variable is never used again) and confusing (because the value that is bound to the variable is NOT the answer to any question you're asking). > time:elapsed_time(fun () -> 2 end). is what you want. Except that a trivial function like that is not going to consume enough time for you to measure it. Try something slower, like opening and closing an existing file. From r.wobben@REDACTED Fri Sep 25 08:22:31 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Fri, 25 Sep 2015 08:22:31 +0200 Subject: [erlang-questions] feedback please In-Reply-To: References: <56030BE9.1070902@home.nl> <4932605.QsZFrd1HLM@burrito> <56038D14.30008@home.nl> <1496684.U5ec38t2A0@burrito> Message-ID: <5604E827.2060203@home.nl> Op 25-9-2015 om 02:58 schreef Richard A. O'Keefe: > On 24/09/2015, at 5:56 pm, zxq9 wrote: > >> On Thursday 24 September 2015 07:41:40 Roelof Wobben wrote: >>> Thanks, >>> >>> You are right. I did take a quick look and almost the whole chapter is >>> try .. catch. So match or chrash is better ? >> It is not so much of a matter of which is "better" -- but that *most* of the time you want to follow the mantra of "let it crash" and this is a very natural fit for Erlang. > It must be said that *this* specific example is one where > crashing is an extremely bad thing. If you went to your > ATM and a process in the bank's machines had crashed and > completely lost track of your money, you would be EXTREMELY > unhappy. Lawyer-level unhappy. > > A lot of values can be thrown away and no harm done. > But if a process is supposed to be managing persistent > state there may be serious legal penalties attached to > losing it. > > This would make a nice little DETS example: does it > really make sense to keep the balance in a persistent > store? Too right it does! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > ----- > Geen virus gevonden in dit bericht. > Gecontroleerd door AVG - www.avg.com > Versie: 2015.0.6140 / Virusdatabase: 4419/10692 - datum van uitgifte: 09/24/15 > Hello, Here is my version with a quit choice and a check if the beginning balance is zero or more : -module(balance). -export([maintain_balance/1]). maintain_balance(Balance) when Balance >= 0 -> io:format("~n~n"), io:format("Your account balance is $~w.~n", [Balance]), Action = ask_action(), case Action of quit -> io:format("Thanks for using our software"); _ -> Amount = ask_amount(), case Action of withdraw -> withdraw(Balance, {Action, Amount}); deposit -> deposit(Balance, {Action, Amount}) end end; maintain_balance(_) -> io:format("The balance cannot be negative"). withdraw(0, {withdraw, _}) -> ok = io:format("Hey! You're broke!~n"), maintain_balance(0); withdraw(Balance, {withdraw, Amount}) when Amount > Balance -> ok = io:format("You cannot withdraw more than you have~n"), maintain_balance(Balance); withdraw(Balance, {withdraw, Amount}) -> maintain_balance(Balance - Amount). deposit(Balance, {deposit, Amount}) -> maintain_balance(Balance + Amount). ask_action() -> Input = io:get_line("Which action do you need? (W)ithdraw/(D)eposit?/(Q)uit "), case Input of [$w | _] -> withdraw; [$W | _] -> withdraw; [$d | _] -> deposit; [$D | _] -> deposit; [$q | _] -> quit; [$Q | _] -> quit; BadInput -> ok = io:format("You can't seem to '~tp' today...~n" "Let's try that again...~n", [BadInput]), ask_action() end. ask_amount() -> Input = io:get_line("How much? "), case scrub_amount(Input) of {ok, Number} when Number > 0 -> Number; {ok, 0} -> ok = io:format("$0? Giving up on the idea already?~n"), 0; {ok, Number} -> ok = io:format("Hey, you! ~tp is less than no money!~n", [Number]), ask_amount(); {error, bad_input} -> ok = io:format("Hrm... '~tp' does not seem to be a number~n" "Now, one more time...~n", [Input]), ask_amount() end. scrub_amount(Input) -> case string:to_integer(Input) of {error, no_integer} -> {error, bad_input}; {Int, _} -> {ok, Int} end. Roelof From r.wobben@REDACTED Fri Sep 25 08:54:38 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Fri, 25 Sep 2015 08:54:38 +0200 Subject: [erlang-questions] feedback please In-Reply-To: <5604E827.2060203@home.nl> References: <56030BE9.1070902@home.nl> <4932605.QsZFrd1HLM@burrito> <56038D14.30008@home.nl> <1496684.U5ec38t2A0@burrito> <5604E827.2060203@home.nl> Message-ID: <5604EFAE.7010308@home.nl> Op 25-9-2015 om 08:22 schreef Roelof Wobben: > Op 25-9-2015 om 02:58 schreef Richard A. O'Keefe: >> On 24/09/2015, at 5:56 pm, zxq9 wrote: >> >>> On Thursday 24 September 2015 07:41:40 Roelof Wobben wrote: >>>> Thanks, >>>> >>>> You are right. I did take a quick look and almost the whole chapter is >>>> try .. catch. So match or chrash is better ? >>> It is not so much of a matter of which is "better" -- but that >>> *most* of the time you want to follow the mantra of "let it crash" >>> and this is a very natural fit for Erlang. >> It must be said that *this* specific example is one where >> crashing is an extremely bad thing. If you went to your >> ATM and a process in the bank's machines had crashed and >> completely lost track of your money, you would be EXTREMELY >> unhappy. Lawyer-level unhappy. >> >> A lot of values can be thrown away and no harm done. >> But if a process is supposed to be managing persistent >> state there may be serious legal penalties attached to >> losing it. >> >> This would make a nice little DETS example: does it >> really make sense to keep the balance in a persistent >> store? Too right it does! >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> ----- >> Geen virus gevonden in dit bericht. >> Gecontroleerd door AVG - www.avg.com >> Versie: 2015.0.6140 / Virusdatabase: 4419/10692 - datum van uitgifte: >> 09/24/15 >> > > Hello, > > Here is my version with a quit choice and a check if the beginning > balance is zero or more : > > -module(balance). > -export([maintain_balance/1]). > > maintain_balance(Balance) when Balance >= 0 -> > io:format("~n~n"), > io:format("Your account balance is $~w.~n", [Balance]), > Action = ask_action(), > case Action of > quit -> > io:format("Thanks for using our software"); > _ -> > Amount = ask_amount(), > case Action of > withdraw -> withdraw(Balance, {Action, Amount}); > deposit -> deposit(Balance, {Action, Amount}) > end > end; > > maintain_balance(_) -> > io:format("The balance cannot be negative"). > > withdraw(0, {withdraw, _}) -> > ok = io:format("Hey! You're broke!~n"), > maintain_balance(0); > > withdraw(Balance, {withdraw, Amount}) when Amount > Balance -> > ok = io:format("You cannot withdraw more than you have~n"), > maintain_balance(Balance); > > withdraw(Balance, {withdraw, Amount}) -> > maintain_balance(Balance - Amount). > > deposit(Balance, {deposit, Amount}) -> > maintain_balance(Balance + Amount). > > ask_action() -> > Input = io:get_line("Which action do you need? > (W)ithdraw/(D)eposit?/(Q)uit "), > case Input of > [$w | _] -> withdraw; > [$W | _] -> withdraw; > [$d | _] -> deposit; > [$D | _] -> deposit; > [$q | _] -> quit; > [$Q | _] -> quit; > BadInput -> > ok = io:format("You can't seem to '~tp' today...~n" > "Let's try that again...~n", > [BadInput]), > ask_action() > end. > > ask_amount() -> > Input = io:get_line("How much? "), > case scrub_amount(Input) of > {ok, Number} when Number > 0 -> > Number; > {ok, 0} -> > ok = io:format("$0? Giving up on the idea already?~n"), > 0; > {ok, Number} -> > ok = io:format("Hey, you! ~tp is less than no money!~n", > [Number]), > ask_amount(); > {error, bad_input} -> > ok = io:format("Hrm... '~tp' does not seem to be a number~n" > "Now, one more time...~n", > [Input]), > ask_amount() > end. > > scrub_amount(Input) -> > case string:to_integer(Input) of > {error, no_integer} -> {error, bad_input}; > {Int, _} -> {ok, Int} > end. > > > Roelof > > And here one without the nested case -module(balance). -export([maintain_balance/1]). maintain_balance(Balance) when Balance >= 0 -> io:format("~n~n"), io:format("Your account balance is $~w.~n", [Balance]), Action = ask_action(), if Action =:= quit -> io:format("Thanks for using our software"); Action =:= withdraw -> withdraw(Balance, {Action, ask_amount()}); Action =:= deposit -> deposit(Balance, {Action, ask_amount()}) end; maintain_balance(_) -> io:format("The balance cannot be negative"). withdraw(0, {withdraw, _}) -> ok = io:format("Hey! You're broke!~n"), maintain_balance(0); withdraw(Balance, {withdraw, Amount}) when Amount > Balance -> ok = io:format("You cannot withdraw more than you have~n"), maintain_balance(Balance); withdraw(Balance, {withdraw, Amount}) -> maintain_balance(Balance - Amount). deposit(Balance, {deposit, Amount}) -> maintain_balance(Balance + Amount). ask_action() -> Input = io:get_line("Which action do you need? (W)ithdraw/(D)eposit?/(Q)uit "), case Input of [$w | _] -> withdraw; [$W | _] -> withdraw; [$d | _] -> deposit; [$D | _] -> deposit; [$q | _] -> quit; [$Q | _] -> quit; BadInput -> ok = io:format("You can't seem to '~tp' today...~n" "Let's try that again...~n", [BadInput]), ask_action() end. ask_amount() -> Input = io:get_line("How much? "), case scrub_amount(Input) of {ok, Number} when Number > 0 -> Number; {ok, 0} -> ok = io:format("$0? Giving up on the idea already?~n"), 0; {ok, Number} -> ok = io:format("Hey, you! ~tp is less than no money!~n", [Number]), ask_amount(); {error, bad_input} -> ok = io:format("Hrm... '~tp' does not seem to be a number~n" "Now, one more time...~n", [Input]), ask_amount() end. scrub_amount(Input) -> case string:to_integer(Input) of {error, no_integer} -> {error, bad_input}; {Int, _} -> {ok, Int} end. Roelof From lukas@REDACTED Fri Sep 25 11:38:01 2015 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 25 Sep 2015 11:38:01 +0200 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: Hello, On Thu, Sep 24, 2015 at 11:58 PM, Philip Clarke wrote: > One common use case for tracing I would like is the ability to trace a > function call for just the next request in the system. > > Currently the PidSpec in erlang:trace/3 allows for pid, existing, new or > all. > > We would use pid if we knew the pid of the next short lived process to > call a function (but we don't). > We cannot use new either as tracing a function call for all new processes > might mean that this function is traced 1000s times per second. > > What I would like is some new PidSpec like 'next_pid' which would specify > that I want to trace all new processes until the function call I am tracing > has been triggered. Once the trace is triggered then it should turn off > tracing on all processes. > > e.g. > dbg:p(next_pid, call). > dbg:tp(M, F, x). > > This would be a great help in my work where quite often I want to see the > typical format of a request from a client and just capturing 1 request is > enough. I believe that libraries such as redbug have this kind of ability, > but perhaps the feature is fundamental enough to include as part of trace ? > > Doing this kind of thing and guaranteeing that just one trace will be printed cannot be done without suspending all schedulers running in the system, which is something that I'm pretty sure you do not want :) You can however come pretty close even today by doing something like this: -module(single_trace). -export([trace/2,trace/4]). trace(M,F) -> trace(M,F,'_',[]). trace(M,F,A,MS) -> DisableTrace = {'_',[],[{disable_trace,[call]}]}, erlang:trace_pattern({M,F,A}, [ DisableTrace | MS], []), Tracer = spawn(fun trace_loop/0), erlang:trace(all, true, [{tracer, Tracer}, call]). trace_loop() -> receive Msg -> io:format("~p~n",[Msg]), erlang:trace(all, false, [call]), trace_loop() end. 12> spawn(fun F() -> lists:map(fun(I) -> I + 1 end, lists:seq(1,100)), F() end). <0.53.0> 13> spawn(fun F() -> lists:map(fun(I) -> I + 1 end, lists:seq(1,100)), F() end). <0.55.0> 14> spawn(fun F() -> lists:map(fun(I) -> I + 1 end, lists:seq(1,100)), F() end). <0.57.0> 15> spawn(fun F() -> lists:map(fun(I) -> I + 1 end, lists:seq(1,100)), F() end). <0.59.0> 16> spawn(fun F() -> lists:map(fun(I) -> I + 1 end, lists:seq(1,100)), F() end). <0.61.0> 17> single_trace:trace(lists,map). {trace,<0.61.0>,call, {lists,map, [#Fun, [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23, 24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43, 44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63, 64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83, 84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100]]}} 18> Writing your own little tracer process can sometimes do wonders :) This will occasionally print a couple of more messages, but it is quite fast at turning things off. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From leonard.boyce@REDACTED Fri Sep 25 16:08:07 2015 From: leonard.boyce@REDACTED (Leonard Boyce) Date: Fri, 25 Sep 2015 10:08:07 -0400 Subject: [erlang-questions] Mnesia disc_copies Message-ID: Hi all, I'm hoping someone can clarify this for me as I feel like a bit of an idiot right now. For some reason I've always been under the impression that mnesia disc_copies does not keep *all* data memory resident. IE it allows for 'most recent' data to remain memory resident, while persisting all data to disc. Reading the doc, it appears I am 100% incorrect and that *all* data in a disc_copies table is both memory resident *and* persisted to disc. from the doc, "disc_copies ... If a table is of type disc_copies at a certain node, the entire table is resident in RAM memory and on disc.". Is this correct? Thanks, Leonard From samuelrivas@REDACTED Fri Sep 25 16:44:12 2015 From: samuelrivas@REDACTED (Samuel) Date: Fri, 25 Sep 2015 16:44:12 +0200 Subject: [erlang-questions] user process suspended In-Reply-To: <2595.1443073042@snookles.snookles.com> References: <2595.1443073042@snookles.snookles.com> Message-ID: Thanks a lot, that seems very helpful :) For what is worth, after disabling the console backed we have been running for a while without experiencing problems, but I want to see it running flawlessly for a few more days before calling it a victory. On 24 September 2015 at 07:37, Scott Lystig Fritchie wrote: > Samuel wrote: > > sr> We are starting to suspect that it is the lager console_backed that > sr> hangs in a send to the io port. We have just disabled it and we'll > sr> see if it helps. > > Hi, Samuel. Basho has fought a similar problem with console I/O > hanging. IIRC we had the most problems with Solaris/Illumos/SmartOS. > My poor memory is forgetting which BEAM runtimes were succeptible. I > also don't know if or how often the problem was reported to us on other > OS platforms. > > There were a couple of diagnostics that we used: > > 1. Getting process_info(Pid, backtrace) for all procs. Then we'd see > that the ... hrm, which process ... hrrrrm, Riak version 1.2 used R15B01, > I think ... the 'user' process? This may or may not resemble the hang, > see the end of this message. > > 2. Connecting to the Riak console via the "to_erl" scheme to talk to The > VM's Actual Console. Then the admin would press Return ... and the Riak > node would spring to life, problem solved. Until next time. > > IIRC disabling console output was the workaround. > > I've lost the ops history of when that problem was fixed VM/runtime-wise > and/or Riak's workarounds. But if this sounds like your problem, then > it's worth digging for that history. > > Oh, here's a piece of luck. I've rediscovered that I'd written an > Expect script to automate the process for triggering the problem. I'll > attach that, also, in case it's helpful. > > -Scott > > --- snip --- snip --- snip --- snip --- > > =proc:<0.30.0> > State: Waiting > Name: user > Spawned as: erlang:apply/2 > Current call: user:get_chars_more/7 > Spawned by: <0.29.0> > Started: Tue Apr 9 16:16:46 2013 > Message queue length: 1 > Message queue: [{io_request,<0.186.0>,<0.179.0>,{put_chars,unicode,<<821 bytes>>}}] > Number of heap fragments: 0 > Heap fragment data: 0 > Link list: [] > Dictionary: [{shell,<0.31.0>},{read_mode,list},{unicode,false}] > Reductions: 357542 > Stack+heap: 610 > OldHeap: 0 > Heap unused: 152 > OldHeap unused: 0 > Stack dump: > Program counter: 0x00007f0c51d92930 (user:get_chars_more/7 + 232) > CP: 0x0000000000000000 (invalid) > arity = 0 > > 0x00007f0c51931710 Return addr 0x00007f0c51d8c498 (user:do_io_request/5 + 88) > y(0) unicode > y(1) {[],[]} > y(2) #Port<0.630> > y(3) {erl_scan,tokens,[1]} > y(4) get_until > y(5) io_lib > y(6) {erl_scan_continuation,[],no_col,[],2,{erl_scan,#Fun,false,false,false},0,#Fun} > > 0x00007f0c51931750 Return addr 0x00007f0c51d8c308 (user:server_loop/2 + 1408) > y(0) #Port<0.630> > y(1) <0.30.0> > y(2) <0.25333.1728> > > 0x00007f0c51931770 Return addr 0x00007f0c51d8b968 (user:catch_loop/3 + 112) > y(0) #Port<0.630> > > 0x00007f0c51931780 Return addr 0x0000000000836c78 () > y(0) <0.31.0> > y(1) #Port<0.630> > y(2) Catch 0x00007f0c51d8b968 (user:catch_loop/3 + 112) > > --- snip --- snip --- snip --- snip --- > > #!/usr/bin/expect -- > > ## Assumptions: > ## > ## 1. SSH has been used to log into "hostname" once before: we aren't > ## smart enough to deal with SSH host key verification stuff. > ## > ## 2. This script can successfully use SSH to log into hostname using > ## the "riak" user. > ## > ## 3. If SSH does not require a password for the "riak" user on > ## hostname, then please break that feature so that the SSH > ## password authentication method is required. > ## > ## 4. We assume that the "riak" account's shell prompt will include > ## a "$" in it. If that is not a correct assumption, please fix > ## the login_prompt procedure below. > ## > ## 5. This script will stop Riak and kill any other beam.smp processes > ## that happen to be running on hostname. > ## > ## 6. We assume riak is SMP-capable and look at beam.smp processes only. > > > ## Usage: this-script hostname password riak_path erl_path cookie riak_node_name > ## > ## For example, if the remote machine is using a Linux CentOS Riak package: > ## > ## expect this-script 172.16.72.130 foofoo /usr/sbin/riak /usr/lib64/riak/erts-5.9.1/bin/erl riak riak@REDACTED > > ## Reminder reminder reminder: never use single quotes > ## Reminder reminder reminder: never use single quotes > ## Reminder reminder reminder: never use single quotes > > set timeout 30 > set erl_prompt {@[^)]+\)\d+> } > > set hostname [lindex $argv 0] > set password [lindex $argv 1] > set riak_path [lindex $argv 2] > set erl_path [lindex $argv 3] > set cookie [lindex $argv 4] > set riak_node_name [lindex $argv 5] > > proc login_prompt {} { > expect -re {\$} > # If a dollar sign does not appear in the prompt, comment out > # the line above, uncomment the line below, then edit the line > # below to include the correct prompt string. > # expect -re {some prompt regexp here} > } > > proc erlang_prompt {} { > global erl_prompt > > expect -re $erl_prompt > } > > proc cmd {cmd} { > send "$cmd\r" > login_prompt > } > > proc async_cmd {cmd} { > send "$cmd\r" > } > > proc erlang_cmd {cmd} { > send "$cmd\r" > erlang_prompt > } > > proc announce {msg} { > puts "\n\n*** [exec date]\n*** $msg" > } > > proc switch_to {id} { > global spawn_id > > announce "Switching to session $id" > set spawn_id $id > } > > # send_user "Password for host $hostname: " > # stty -echo > # expect_user -re "(.*)\n" > # stty echo > # set password $expect_out(1,string) > # puts "" > > spawn ssh -l riak $hostname > set session1 $spawn_id > expect {password} > send "$password\r" > login_prompt > > announce "Killing Riak and all other beam.smp processes" > switch_to $session1 > cmd {ps axww | grep beam.smp | grep -v grep | awk '{print $1}' | xargs kill -9 ; sleep 1} > cmd {ps -ef | grep beam.smp | grep -v grep | awk '{print $2}' | xargs kill -9 ; sleep 1} > > announce "Starting Riak" > spawn ssh -l riak $hostname "$riak_path start" > set session2 $spawn_id > expect {password} > send "$password\r" > expect "ok" > set after_start 10 > announce "Got ok from '$riak_path start', sleeping $after_start seconds..." > sleep $after_start > > announce "Attach to riak, do sanity test" > switch_to $session1 > async_cmd "$riak_path attach" > send "hello_world.\r" > erlang_prompt > > announce "2nd login session to $hostname, attach to riak via remsh" > > spawn ssh -l riak $hostname > set session2 $spawn_id > expect {password} > send "$password\r" > login_prompt > > switch_to $session2 > async_cmd "$erl_path -setcookie $cookie -name tmp@REDACTED -remsh $riak_node_name" > erlang_prompt > erlang_cmd "hello." > > announce "Log a message via lager, watch for it over on Riak console" > switch_to $session2 > erlang_cmd "lager:log(warning, self(), \"WARNING 1\", \[\])." > > switch_to $session1 > expect { > "WARNING 1" { announce "Found our first warning, excellent!" } > timeout { puts "BUMMER 1!" ; exit 4 } > } > > announce "Now, to hang the logger to demonstrate the bug.\n Any incomplete term is fine, we'll use an unfinished tuple...." > switch_to $session1 > async_cmd "{tuple_item_number_1," > > set bug_timeout 12 > announce "Now that console is hung, call lager:log().\n Set timeout time to $bug_timeout seconds.\n This function should execute in only a few milliseconds." > > set timeout $bug_timeout > > switch_to $session2 > async_cmd "lager:log(warning, self(), \"WARNING 2\", \[\])." > > switch_to $session1 > expect { > "WARNING 2" { announce "BUMMER, Found 2nd warning at wrong time"; exit 4} > timeout { announce "TIMEOUT! Excellent, the bug has been demonstrated (1)" } > } > > announce "When we finish the erlang term over on $session1 ('riak attach' session),\n then the logger will become unblocked." > > switch_to $session1 > async_cmd "element_number_2}." > expect { > -re $erl_prompt { announce "Excellent, we have 'riak attach' prompt" } > timeout { announce "BUMMER, timeout when not expected (2)"; exit 4} > } > > announce "Now go to session $session2 and expect the 2nd warning\n _and_ the -remsh's prompt." > switch_to $session2 > expect { > "WARNING 2" { announce "Excellent, I see the 'WARNING 2' log message" } > timeout { announce "BUMMER, timeout when not expected @ 2nd w"; exit 4} > } > expect { > -re $erl_prompt { announce "Excellent, we have the 'riak -remsh' prompt" } > timeout { announce "BUMMER, timeout when not expected (3)"; exit 4} > } > > announce "Time to clean up both sessions" > > switch_to $session1 > async_cmd "\004" > login_prompt > async_cmd "exit\r" > expect eof > > switch_to $session2 > async_cmd "\003" ; sleep 1 > async_cmd "\003" ; sleep 1 > login_prompt > async_cmd "exit\r" > expect eof > > announce "Excellent, this successful test is finished." > exit 0 -- Samuel From vasdeveloper@REDACTED Fri Sep 25 17:26:40 2015 From: vasdeveloper@REDACTED (Theepan) Date: Fri, 25 Sep 2015 20:56:40 +0530 Subject: [erlang-questions] Mnesia disc_copies In-Reply-To: References: Message-ID: >> I feel like a bit of an idiot right now. Don't take it too personal, it often happens to everybody.. Just kidding.. :-) Only caches provide for near-time, near-space data, and Mnesia RAM is not a cache. Therefore it is full copy of the tables. Theepan On Fri, Sep 25, 2015 at 7:38 PM, Leonard Boyce wrote: > Hi all, > > I'm hoping someone can clarify this for me as I feel like a bit of an > idiot right now. > > For some reason I've always been under the impression that mnesia > disc_copies does not keep *all* data memory resident. IE it allows for > 'most recent' data to remain memory resident, while persisting all > data to disc. > > Reading the doc, it appears I am 100% incorrect and that *all* data in > a disc_copies table is both memory resident *and* persisted to disc. > from the doc, "disc_copies ... If a table is of type disc_copies at a > certain node, the entire table is resident in RAM memory and on > disc.". > > Is this correct? > > Thanks, > Leonard > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vances@REDACTED Fri Sep 25 17:36:45 2015 From: vances@REDACTED (Vance Shipley) Date: Fri, 25 Sep 2015 21:06:45 +0530 Subject: [erlang-questions] Mnesia disc_copies In-Reply-To: References: Message-ID: On Sep 25, 2015 7:38 PM, "Leonard Boyce" wrote: > If a table is of type disc_copies at a certain node, the entire table is resident in RAM memory and on disc.". Yes, mnesia is a memory resident database, that's why it was originally called "amnesia"; because when you turn it off it forgets everything. Persistence was added later. -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.wobben@REDACTED Fri Sep 25 17:42:52 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Fri, 25 Sep 2015 17:42:52 +0200 Subject: [erlang-questions] json parsing Message-ID: <56056B7C.6030304@home.nl> Hello, I like to try to parse a json file in Erlang. I found the jsx library where I noticed that all "strings" where excaped by a / Do I have to do it myself or can I write a function which checks if something is a list and then concat a / the string and another / ? Roelof From vasdeveloper@REDACTED Fri Sep 25 18:13:36 2015 From: vasdeveloper@REDACTED (Theepan) Date: Fri, 25 Sep 2015 21:43:36 +0530 Subject: [erlang-questions] json parsing In-Reply-To: <56056B7C.6030304@home.nl> References: <56056B7C.6030304@home.nl> Message-ID: Check the documentation of the "file" module. You can get the contents of a file in list of binary format. That means, you don't have to do your own escaping. Theepan On Fri, Sep 25, 2015 at 9:12 PM, Roelof Wobben wrote: > Hello, > > I like to try to parse a json file in Erlang. > > I found the jsx library where I noticed that all "strings" where excaped > by a / > > Do I have to do it myself or can I write a function which checks if > something is a list and then concat a / the string and another / ? > > Roelof > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.wobben@REDACTED Fri Sep 25 18:15:49 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Fri, 25 Sep 2015 18:15:49 +0200 Subject: [erlang-questions] json parsing In-Reply-To: References: <56056B7C.6030304@home.nl> Message-ID: <56057335.8020408@home.nl> An HTML attachment was scrubbed... URL: From damien@REDACTED Fri Sep 25 18:03:46 2015 From: damien@REDACTED (Damien Krotkine) Date: Fri, 25 Sep 2015 18:03:46 +0200 Subject: [erlang-questions] Feedback on webmachine route Message-ID: <1443197026.1700912.393584041.707866DD@webmail.messagingengine.com> Hi, I'm kinda new to Erlang, and I'm extending the ReST API of Riak, so that a single request will make it fetch multiple keys and return their values, concatenated. It's spawning a process for each keys for fetching the values, then wait to receive all the values to stich them together and return it to the client. I'm sure this code is horrible from the point of view of a seasoned Erlang developer, and I welcome any feedback. Especially regarding speed and robustness against faults. Thanks, dams Link to code here: http://lpaste.net/6968618518325493760 Also pasted below: -module(events_api). %% webmachine resource exports -export([ init/1, content_types_provided/2, service_available/2, malformed_request/2, to_textplain/2 ]). -include_lib("webmachine/include/webmachine.hrl"). -record(ctx, { bucket = undefined, key = undefined }). init(_) -> {ok, #ctx{}}. content_types_provided(ReqData, Context) -> {[{"plain/text", to_textplain}], ReqData, Context}. malformed_request(ReqData, Ctx) -> {false, ReqData, Ctx}. service_available(ReqData, Ctx) -> {true, ReqData, Ctx}. to_textplain(ReqData, Ctx) -> Epoch = wrq:get_qs_value("epoch", "n/a", ReqData), DC = wrq:get_qs_value("dc", ReqData), Types = wrq:get_qs_value("types", "n/a", ReqData), Personas = wrq:get_qs_value("personas", "n/a", ReqData), error_logger:info_msg("epoch:~p DC:~p types:~p personas:~p",[ Epoch, DC, Types, Personas ]), {ok, Client} = riak:local_client(), % fetch metadata, returns a list of data keys DataKeys = get_datakeys(list_to_binary(Epoch), list_to_binary(DC), Client), error_logger:info_msg("DataKeys: ~p ",[ DataKeys ]), % get the actual event blobs from the data keys Blobs = get_values(Client, DataKeys), error_logger:info_msg(" got ~p blobs",[ lists:flatlength(Blobs) ]), % join blobs together, and return it Body = binary_join(Blobs), {[Body], ReqData, Ctx}. % 1442926965 get_datakeys(Epoch, DC, Client) -> Bucket = <<"epochs">>, Key = <>, Options = [ {r, 1}, {timeout, 1000} ], error_logger:info_msg("fetching metadata bucket:~p key:~p ...",[ Bucket, Key ]), { ok, RiakObject } = riak_client:get(Bucket, Key, Options, Client), Data = riak_object:get_value(RiakObject), Partials = binary:split(Data, <<"|">>, [ global ]), % data keys have the epoch prepended, let's do that lists:map( fun(Partial) -> <> end, Partials). % easy api get_values(Client, DataKeys) -> get_values(Client, DataKeys, [], []). % main code: spawn a child that fetches, accumulate child pid get_values(Client, [DataKey | Tail], Children, []) -> Parent = self(), Child = spawn_link( fun() -> Options = [ {r, 1}, {timeout, 1000} ], Bucket = <<"events">>, { ok, RiakObject } = riak_client:get(Bucket, DataKey, Options, Client), Blob = riak_object:get_value(RiakObject), Parent ! { self(), Blob } end ), get_values(Client, Tail, [ Child | Children ], []). % all keys fetches are spawn, wait and collect all the results get_values(_Client, [], Children, Blobs) -> receive {Child, Blob} -> case lists:member(Child, Children) of true -> get_values(undefined, [], lists:delete(Child, Children), [Blob| Blobs]); false -> erlang:error("Wrong signal: doesn't match protocol") end end; % all results collected, return the blobs. get_values(_Client, [], [], Blobs) -> Blobs; binary_join([]) -> <<>>; binary_join([Part]) -> Part; binary_join([Head|Tail]) -> lists:foldl(fun (Value, Acc) -> <> end, Head, Tail). From r.wobben@REDACTED Sat Sep 26 09:01:30 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Sat, 26 Sep 2015 09:01:30 +0200 Subject: [erlang-questions] problem making my first parser Message-ID: <560642CA.5060204@home.nl> Hello, I try to make a very easy parser to practice to make a json parser in the future. As first I try to make a parser which can calculate some basic things. So far I have this : -module(number_parser). -export([scan/1]). scan(String) -> scan(String, []). scan([], List) -> List; scan([Head | Rest], List_parse) when Head >= $0, Head =< $9 -> scan(Rest, [(Head - 48) | List_parse]); scan([Head | Rest], List_parse) when Head =:= "+" -> scan(Rest, [Head | List_parse] ). but when I try this code on scan("1+1") I see this error message : ** exception error: no function clause matching number_parser:scan("+1",[1]) (number_parser.erl, line 8) Which does not make sense. +1 makes as far as I know [+, 1] so I have a head and tail so the last clause schould work. Why do I then see this error message ? Roelof From zxq9@REDACTED Sat Sep 26 09:08:35 2015 From: zxq9@REDACTED (zxq9) Date: Sat, 26 Sep 2015 16:08:35 +0900 Subject: [erlang-questions] problem making my first parser In-Reply-To: <560642CA.5060204@home.nl> References: <560642CA.5060204@home.nl> Message-ID: <2946856.ALr5DkKQnJ@burrito> On Saturday 26 September 2015 09:01:30 Roelof Wobben wrote: > Hello, > > I try to make a very easy parser to practice to make a json parser in > the future. > As first I try to make a parser which can calculate some basic things. > > So far I have this : > > -module(number_parser). > > -export([scan/1]). > > scan(String) -> > scan(String, []). > > scan([], List) -> > List; > > scan([Head | Rest], List_parse) when Head >= $0, Head =< $9 -> > scan(Rest, [(Head - 48) | List_parse]); > > scan([Head | Rest], List_parse) when Head =:= "+" -> > scan(Rest, [Head | List_parse] ). > > > but when I try this code on scan("1+1") I see this error message : > > ** exception error: no function clause matching > number_parser:scan("+1",[1]) (number_parser.erl, line 8) > > Which does not make sense. +1 makes as far as I know [+, 1] so I have a > head and tail so the last clause schould work. > > > Why do I then see this error message ? What happens if you try `[1] >= $0, [1] =< $9.`? How about `[1] =:= "+".` ? What are you actually comparing here? Do you care about plus signs as strings, or do you care about plus as a symbol that represents an operation? -Craig From r.wobben@REDACTED Sat Sep 26 09:31:31 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Sat, 26 Sep 2015 09:31:31 +0200 Subject: [erlang-questions] problem making my first parser In-Reply-To: <2946856.ALr5DkKQnJ@burrito> References: <560642CA.5060204@home.nl> <2946856.ALr5DkKQnJ@burrito> Message-ID: <560649D3.4000509@home.nl> Op 26-9-2015 om 09:08 schreef zxq9: > On Saturday 26 September 2015 09:01:30 Roelof Wobben wrote: >> Hello, >> >> I try to make a very easy parser to practice to make a json parser in >> the future. >> As first I try to make a parser which can calculate some basic things. >> >> So far I have this : >> >> -module(number_parser). >> >> -export([scan/1]). >> >> scan(String) -> >> scan(String, []). >> >> scan([], List) -> >> List; >> >> scan([Head | Rest], List_parse) when Head >= $0, Head =< $9 -> >> scan(Rest, [(Head - 48) | List_parse]); >> >> scan([Head | Rest], List_parse) when Head =:= "+" -> >> scan(Rest, [Head | List_parse] ). >> >> >> but when I try this code on scan("1+1") I see this error message : >> >> ** exception error: no function clause matching >> number_parser:scan("+1",[1]) (number_parser.erl, line 8) >> >> Which does not make sense. +1 makes as far as I know [+, 1] so I have a >> head and tail so the last clause schould work. >> >> >> Why do I then see this error message ? > What happens if you try `[1] >= $0, [1] =< $9.`? > > How about `[1] =:= "+".` ? > > What are you actually comparing here? A list with a string/char. But I check the tail of a list which contains a char with + or 0 - 9. > Do you care about plus signs as strings, or do you care about plus as a symbol that represents an operation? At this point I only care about the plus sign as a string. Later on the parse function I will make a operation based on this. Roelof From zxq9@REDACTED Sat Sep 26 09:51:00 2015 From: zxq9@REDACTED (zxq9) Date: Sat, 26 Sep 2015 16:51 +0900 Subject: [erlang-questions] problem making my first parser In-Reply-To: <560649D3.4000509@home.nl> References: <560642CA.5060204@home.nl> <2946856.ALr5DkKQnJ@burrito> <560649D3.4000509@home.nl> Message-ID: <360820909.ln3eIfI47t@burrito> On Saturday 26 September 2015 09:31:31 Roelof Wobben wrote: > Op 26-9-2015 om 09:08 schreef zxq9: > > On Saturday 26 September 2015 09:01:30 Roelof Wobben wrote: > >> Why do I then see this error message ? > > What happens if you try `[1] >= $0, [1] =< $9.`? > > > > How about `[1] =:= "+".` ? > > > > What are you actually comparing here? > > A list with a string/char. But I check the tail of a list which > contains a char with + or 0 - 9. > > > Do you care about plus signs as strings, or do you care about plus as a symbol that represents an operation? > > At this point I only care about the plus sign as a string. Later on the > parse function I will make a operation based on this. You care about the plus sign as a token, not a string. There is a difference. What is the difference among the following? scan([H|R], L) when H =:= "+" -> scan(R, [plus|L]. scan([H|R], L) when H =:= $+ -> scan(R, [H|L]). scan([$+|R], L) -> scan(R, [fun plus/2|L]). Three different ways to decide what you received, and three different ways to continue on. Think about which combination might be most useful to you, and which might be difficult to deal with (or even broken). In other words: Since you are going to have to deal with the resulting list later, which list will be the easiest to deal with? But that is forward thinking about the output. What might be the most useful way to define the *input* that you are going to accept? Maybe "1+1" all crunched together isn't the friendliest way to receive this input? Simple delimitation like "1 + 1" or even "1,+,1" certainly allows more opportunities for "tokenize-or-reject/crash" on bad input. What about the ordering of the symbols? After all "1" is not actually 1, it is a character that represents 1 (hence the conversion from a character to an integer), the same way that neither "+" nor $+ are actually `plus(A, B) -> A + B`. Hm... that's interesting, when writing functions we write function(Thingy1, Thingy2) -- which is not infix notation. So infix "1 + 1" would look like infix "1 plus 1" but converted to a function would look like "plus(1, 1)" which as an S-expression would look like "(plus 1 1)" or "(+ 1 1)", which in Reverse Polish notation would look like "1 1 +" and as a list might look like [1, 1, plus] or [plus, 1, 1]. Think about this a bit. Why are these notations useful? Why have they been invented? Why do programming languages have functions written like "plus(A, B)" when there are operators like "A + B"? These are not merely philosophic questions. They are central to understanding how notations, grammars, parsing, etc. work. -Craig From r.wobben@REDACTED Sat Sep 26 10:07:46 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Sat, 26 Sep 2015 10:07:46 +0200 Subject: [erlang-questions] problem making my first parser In-Reply-To: <360820909.ln3eIfI47t@burrito> References: <560642CA.5060204@home.nl> <2946856.ALr5DkKQnJ@burrito> <560649D3.4000509@home.nl> <360820909.ln3eIfI47t@burrito> Message-ID: <56065252.2000300@home.nl> Op 26-9-2015 om 09:51 schreef zxq9: > On Saturday 26 September 2015 09:31:31 Roelof Wobben wrote: >> Op 26-9-2015 om 09:08 schreef zxq9: >>> On Saturday 26 September 2015 09:01:30 Roelof Wobben wrote: >>>> Why do I then see this error message ? >>> What happens if you try `[1] >= $0, [1] =< $9.`? >>> >>> How about `[1] =:= "+".` ? >>> >>> What are you actually comparing here? >> A list with a string/char. But I check the tail of a list which >> contains a char with + or 0 - 9. >> >>> Do you care about plus signs as strings, or do you care about plus as a symbol that represents an operation? >> At this point I only care about the plus sign as a string. Later on the >> parse function I will make a operation based on this. > You care about the plus sign as a token, not a string. There is a difference. > > What is the difference among the following? > > scan([H|R], L) when H =:= "+" -> scan(R, [plus|L]. > This one is broken. Apperently you cannot compare this way because a string is a list of integers. > scan([H|R], L) when H =:= $+ -> scan(R, [H|L]). This one works fine but can be improved. > scan([$+|R], L) -> scan(R, [fun plus/2|L]). This one cannot work because when reading the + it's not know what the next will be. > Three different ways to decide what you received, and three different ways to continue on. Think about which combination might be most useful to you, and which might be difficult to deal with (or even broken). > > In other words: > Since you are going to have to deal with the resulting list later, which list will be the easiest to deal with? The second one mixed with the thirth. like this scan([$+ | R], L) -> scan (R, [$+ | L]). > But that is forward thinking about the output. What might be the most useful way to define the *input* that you are going to accept? Maybe "1+1" all crunched together isn't the friendliest way to receive this input? Simple delimitation like "1 + 1" or even "1,+,1" certainly allows more opportunities for "tokenize-or-reject/crash" on bad input. What about the ordering of the symbols? After all "1" is not actually 1, it is a character that represents 1 (hence the conversion from a character to an integer), the same way that neither "+" nor $+ are actually `plus(A, B) -> A + B`. > > Hm... that's interesting, when writing functions we write function(Thingy1, Thingy2) -- which is not infix notation. So infix "1 + 1" would look like infix "1 plus 1" but converted to a function would look like "plus(1, 1)" which as an S-expression would look like "(plus 1 1)" or "(+ 1 1)", which in Reverse Polish notation would look like "1 1 +" and as a list might look like [1, 1, plus] or [plus, 1, 1]. Think about this a bit. These problem I try to solve in the parse part. > Why are these notations useful? Why have they been invented? Why do programming languages have functions written like "plus(A, B)" when there are operators like "A + B"? > > These are not merely philosophic questions. They are central to understanding how notations, grammars, parsing, etc. work. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > ----- > Geen virus gevonden in dit bericht. > Gecontroleerd door AVG - www.avg.com > Versie: 2015.0.6140 / Virusdatabase: 4419/10702 - datum van uitgifte: 09/26/15 > > From r.wobben@REDACTED Sat Sep 26 10:18:52 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Sat, 26 Sep 2015 10:18:52 +0200 Subject: [erlang-questions] problem making my first parser In-Reply-To: <56065252.2000300@home.nl> References: <560642CA.5060204@home.nl> <2946856.ALr5DkKQnJ@burrito> <560649D3.4000509@home.nl> <360820909.ln3eIfI47t@burrito> <56065252.2000300@home.nl> Message-ID: <560654EC.1000509@home.nl> Op 26-9-2015 om 10:07 schreef Roelof Wobben: > Op 26-9-2015 om 09:51 schreef zxq9: >> On Saturday 26 September 2015 09:31:31 Roelof Wobben wrote: >>> Op 26-9-2015 om 09:08 schreef zxq9: >>>> On Saturday 26 September 2015 09:01:30 Roelof Wobben wrote: >>>>> Why do I then see this error message ? >>>> What happens if you try `[1] >= $0, [1] =< $9.`? >>>> >>>> How about `[1] =:= "+".` ? >>>> >>>> What are you actually comparing here? >>> A list with a string/char. But I check the tail of a list which >>> contains a char with + or 0 - 9. >>> >>>> Do you care about plus signs as strings, or do you care about plus >>>> as a symbol that represents an operation? >>> At this point I only care about the plus sign as a string. Later on the >>> parse function I will make a operation based on this. >> You care about the plus sign as a token, not a string. There is a >> difference. >> >> What is the difference among the following? >> >> scan([H|R], L) when H =:= "+" -> scan(R, [plus|L]. >> > This one is broken. Apperently you cannot compare this way because a > string is a list of integers. > >> scan([H|R], L) when H =:= $+ -> scan(R, [H|L]). > > This one works fine but can be improved. >> scan([$+|R], L) -> scan(R, [fun plus/2|L]). > > This one cannot work because when reading the + it's not know what the > next will be. > > >> Three different ways to decide what you received, and three different >> ways to continue on. Think about which combination might be most >> useful to you, and which might be difficult to deal with (or even >> broken). >> >> In other words: >> Since you are going to have to deal with the resulting list later, >> which list will be the easiest to deal with? > > The second one mixed with the thirth. like this scan([$+ | R], L) -> > scan (R, [$+ | L]). > >> But that is forward thinking about the output. What might be the most >> useful way to define the *input* that you are going to accept? Maybe >> "1+1" all crunched together isn't the friendliest way to receive this >> input? Simple delimitation like "1 + 1" or even "1,+,1" certainly >> allows more opportunities for "tokenize-or-reject/crash" on bad >> input. What about the ordering of the symbols? After all "1" is not >> actually 1, it is a character that represents 1 (hence the conversion >> from a character to an integer), the same way that neither "+" nor $+ >> are actually `plus(A, B) -> A + B`. >> >> Hm... that's interesting, when writing functions we write >> function(Thingy1, Thingy2) -- which is not infix notation. So infix >> "1 + 1" would look like infix "1 plus 1" but converted to a function >> would look like "plus(1, 1)" which as an S-expression would look like >> "(plus 1 1)" or "(+ 1 1)", which in Reverse Polish notation would >> look like "1 1 +" and as a list might look like [1, 1, plus] or >> [plus, 1, 1]. Think about this a bit. > > These problem I try to solve in the parse part. > >> Why are these notations useful? Why have they been invented? Why do >> programming languages have functions written like "plus(A, B)" when >> there are operators like "A + B"? >> >> These are not merely philosophic questions. They are central to >> understanding how notations, grammars, parsing, etc. work. >> >> -Craig >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> ----- >> Geen virus gevonden in dit bericht. >> Gecontroleerd door AVG - www.avg.com >> Versie: 2015.0.6140 / Virusdatabase: 4419/10702 - datum van uitgifte: >> 09/26/15 >> >> > Finisched the first complete one -module(number_parser). -export([scan/1]). scan(String) -> scan(String, []). scan([], List) -> parse(List); scan([Head | Rest], List_parse) when Head >= $0, Head =< $9 -> scan(Rest, [(Head - 48) | List_parse]); scan([$+ | Rest], List_parse) -> scan(Rest, [$+ | List_parse] ); scan([32 | Rest], List_parse) -> scan(Rest, List_parse). parse([Number1, $+, Number2]) -> Number1 + Number2; parse([$+, Number1, Number2]) -> Number1 + Number2; parse([Number1, Number2, $+]) -> Number1 + Number2. Things I still have to think about is how I deal with numbers above the 10. Roelof From vimal7370@REDACTED Sat Sep 26 10:21:15 2015 From: vimal7370@REDACTED (Vimal Kumar) Date: Sat, 26 Sep 2015 13:51:15 +0530 Subject: [erlang-questions] OTP Cheatsheet Message-ID: Hi all, Erlang newbie here, learning from the book "Learn You Some Erlang For Great Good". Things were superfine until I reached OTP section. Currently scratching my head seeing all the behaviours and complexities involved, but taking everything slowly in. I was looking for whether there is any quick cheatsheet available on OTP - somesort of quick reference that lists the different OTP behaviours, its required functions and return types. Googled, but didn't find any such. Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Sat Sep 26 11:41:28 2015 From: zxq9@REDACTED (zxq9) Date: Sat, 26 Sep 2015 18:41:28 +0900 Subject: [erlang-questions] OTP Cheatsheet In-Reply-To: References: Message-ID: <2139291.c0agkM9Fnv@burrito> On Saturday 26 September 2015 13:51:15 Vimal Kumar wrote: > Hi all, > > Erlang newbie here, learning from the book "Learn You Some Erlang For Great > Good". Things were superfine until I reached OTP section. Currently > scratching my head seeing all the behaviours and complexities involved, but > taking everything slowly in. > > I was looking for whether there is any quick cheatsheet available on OTP - > somesort of quick reference that lists the different OTP behaviours, its > required functions and return types. Googled, but didn't find any such. It is hard to beat the Erlang docs for this. Unfortunately, it is hard to navigate the Erlang docs until you've spent some time bumbling around within them. The place to start getting a once-over of all this stuff is in the "OTP Design Principles" part of the User's Guide. There is a once-over of what an OTP application looks like, a section on each of the main behaviors, and them some further discussion: http://www.erlang.org/doc/design_principles/des_princ.html The module docs themselves provide a cut-and-dried rundown of each component of the form you are asking for. The gen_server, gen_fsm, gen_event and supervisor docs are under "Basic -> stdlib": gen_server: http://www.erlang.org/doc/man/gen_server.html gen_fsm: http://www.erlang.org/doc/man/gen_fsm.html gen_event: http://www.erlang.org/doc/man/gen_event.html supervisor: http://www.erlang.org/doc/man/supervisor.html Other stuff worth knowing in the stdlib that you may want to read is the documentation for proc_lib, supervisor_bridge and sys. proc_lib: http://www.erlang.org/doc/man/proc_lib.html supervisor_bridge: http://www.erlang.org/doc/man/supervisor_bridge.html sys: http://www.erlang.org/doc/man/sys.html All of the module docs can be found from the command line as well by typing `erl -man [module_name]`, so to read the gen_server docs you can do `erl -man gen_server`. The ideas are more alien than complex, imo, and for me it was helpful to read different discussions that approached bits of OTP from several angles as I worked on projects I cared about. So skim the docs, work through LYSE, code a little, read the User's Guide (tons of interesting stuff in there), pick up some other books (even the ones that are currently over your head might give you some food for thought) and read/skim through those (knowing you'll come back later), read the mailing list archives (http://erlang.org/pipermail/erlang-questions/), etc. Immersion via various sources seems to be the way most people finally get around to really understanding what is going on. Federico Carrone put together a site that is pretty much just an aggregation of useful Erlang links that you might want to skim over when you have time. There is a lot of interesting stuff linked there: http://spawnedshelter.com/ -Craig From vasdeveloper@REDACTED Sat Sep 26 12:10:00 2015 From: vasdeveloper@REDACTED (Theepan) Date: Sat, 26 Sep 2015 15:40:00 +0530 Subject: [erlang-questions] OTP Cheatsheet In-Reply-To: References: Message-ID: All the behaviours in high-level have three abstractions: 1. The Erlang process that is core of the behaviour - We don't have to worry much about it, as the OTP platform is taking care of it. 2. The API to interact with the process mentioned above - The other Erlang process use these APIs to interact with the behaviour process. 3. The call back module - When the Erlang process (mentioned in 1 above) gets messages through APIs (mentioned in 2 above), the functions in this module will be called back by the Erlang process. How the process should behave is defined by this module. What you might want to note is that, most of the time you find mapping between the APIs mentioned in 2 above, and the predefined callback function headers mentioned in 3 above. All the behaviours adhere to the above. The easiest way to learn behaviours is, take gen_server and understand it thoroughly. Others will be easy. It is important, before going into the implementation, to know the role of each behaviour, and where it is appropriate to use them. Erlang/OTP documentation has enough information. Also, it is good to have a look at some open source implementations. Theepan On Sat, Sep 26, 2015 at 1:51 PM, Vimal Kumar wrote: > Hi all, > > Erlang newbie here, learning from the book "Learn You Some Erlang For > Great Good". Things were superfine until I reached OTP section. Currently > scratching my head seeing all the behaviours and complexities involved, but > taking everything slowly in. > > I was looking for whether there is any quick cheatsheet available on OTP - > somesort of quick reference that lists the different OTP behaviours, its > required functions and return types. Googled, but didn't find any such. > > Thank you! > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sat Sep 26 13:07:42 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 26 Sep 2015 13:07:42 +0200 Subject: [erlang-questions] Feedback on webmachine route In-Reply-To: <1443197026.1700912.393584041.707866DD@webmail.messagingengine.com> References: <1443197026.1700912.393584041.707866DD@webmail.messagingengine.com> Message-ID: On Fri, Sep 25, 2015 at 6:03 PM, Damien Krotkine wrote: > I'm sure this code is horrible from the point of view of a seasoned > Erlang developer, and I welcome any feedback. Especially regarding speed > and robustness against faults. > The key two questions to think about: * What happens if your spawned children die? * If a child locks up, does you system exhibit liveness/progress? Since you spawn children with spawn_link, they all link to the parent process. So if one child dies, so does the parent. This gives you all-or-nothing semantics: either all keys are returned, or the system fails. An alternative semantics, based on spawn_monitor for instance, could have you handle each key individually and return an error for keys which fail. Your receive expression never times out. So if a key fetch blocks, so does your call. In some situations, when you can guarantee children have progress, this is no problem, but it is often clever to add some kind of timeout. However, when doing so, you must handle the case where a blob arrives late to the process. This usually means you need to make sure the mailbox flushes appropriately (which is something you need to peruse the webmachine documentation for). It is often impossible to a-priori declare what kind of semantics you want for these things. It is a weakness of "frameworks" that tries to automate these kinds of things. You get a predetermined semantics, but then you can't switch it up when you need a different semantics. So you have to make the analysis of how you want your system to behave if it starts failing. Finally, binary() values form a monoid under concatenation, with <<>> as the neutral element. Thus you can write your binary join as binary_join(Parts) -> lists:foldl(fun(P, Acc) -> <> end, <<>>, Parts). The special case [] then returns <<>> and the case [X] forms << (<<>>)/binary, X/binary>> which evaluates to X as expected. With these changes, binary_join/1 is so simple it is probably worth just folding into the call site directly. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eshikafe@REDACTED Sat Sep 26 13:44:16 2015 From: eshikafe@REDACTED (austin aigbe) Date: Sat, 26 Sep 2015 12:44:16 +0100 Subject: [erlang-questions] Browser not displaying images from Mochiweb server Message-ID: Hello, My browser is unable to display images from my mochiweb server - please see attached file. I am using erlydtl for rendering. What could be the reason for this? Setup: Mochiweb --------------------------- Erlang/OTP 18 --------------------------- Ubuntu 14.04.3 --------------------------- VirtualBox 5.0.4 -------------------------- Windows 7 ------------------------- File structure: austin@REDACTED:~/fe$ ls -l total 232 -rwxr-xr-x 1 austin austin 310 Sep 25 18:25 bench.sh drwxrwxr-x 5 austin austin 4096 Sep 25 19:17 deps drwxrwxr-x 2 austin austin 4096 Sep 26 12:06 ebin -rw-rw-r-- 1 austin austin 418 Sep 25 18:25 Makefile drwxrwxr-x 3 austin austin 4096 Sep 25 18:25 priv -rwxrw-r-- 1 austin austin 199962 Sep 25 19:17 rebar -rw-rw-r-- 1 austin austin 313 Sep 25 19:13 rebar.config drwxrwxr-x 2 austin austin 4096 Sep 26 12:06 src -rwxr-xr-x 1 austin austin 119 Sep 25 18:25 start-dev.sh drwxrwxr-x 3 austin austin 4096 Sep 26 11:34 templates austin@REDACTED:~/fe$ ls -l templates/ total 20 drwxrwxr-x 2 austin austin 4096 Sep 26 11:33 data -rw-rw-r-- 1 austin austin 16339 Sep 26 11:34 login.dtl austin@REDACTED:~/fe$ ls -l templates/data/ total 668 -rwxrwxr-x 1 austin austin 57348 Sep 25 20:05 canvas-tools.js -rwxrwxr-x 1 austin austin 150767 Sep 25 20:05 d3.5.5_dynatrace.js -rwxrwxr-x 1 austin austin 46151 Sep 25 20:05 d3.geo.projection.v0.2.12.min.js -rwxrwxr-x 1 austin austin 6916 Sep 25 20:05 d3map.js -rwxrwxr-x 1 austin austin 2679 Sep 25 20:05 d3.tip.v0.6.3.js -rw-rw-r-- 1 austin austin 43649 Sep 26 11:29 dynatraceLogo.jpg -rwxrwxr-x 1 austin austin 7365 Sep 25 20:05 exporting.js -rwxrwxr-x 1 austin austin 1229 Sep 25 20:05 exportingOffline.js -rwxrwxr-x 1 austin austin 3773 Sep 25 20:05 fonts.css -rwxrwxr-x 1 austin austin 153218 Sep 25 20:05 highcharts-4.0.3.js -rwxrwxr-x 1 austin austin 23135 Sep 25 20:05 highcharts-4.0.3-more.js -rwxrwxr-x 1 austin austin 84995 Sep 25 20:05 jquery-2.1.1.min.js -rwxrwxr-x 1 austin austin 2287 Sep 25 20:05 jquery.gridster.extras.js -rwxrwxr-x 1 austin austin 43758 Sep 25 20:05 jquery.gridster.js -rwxrwxr-x 1 austin austin 21325 Sep 25 20:05 login_logo.png -rwxrwxr-x 1 austin austin 6142 Sep 25 20:05 topojson.v1.6.18.min.js austin@REDACTED:~/fe$ Thanks. Austin -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: frontend_login.PNG Type: image/png Size: 214322 bytes Desc: not available URL: From r.wobben@REDACTED Sat Sep 26 14:44:17 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Sat, 26 Sep 2015 14:44:17 +0200 Subject: [erlang-questions] why is the the output not a list Message-ID: <56069321.2020407@home.nl> Hello, I tried to make a number parser which can parse numbers above the 10. So I change my code to this : -module(number_parser). -export([scan/1]). scan(String) -> scan(String, []). scan([], List) -> List; scan([Head | Rest], List_parse) when Head >= $0, Head =< $9 -> digits(Head, Rest, List_parse); scan( [$+ | Rest], List_parse) -> scan(Rest, [$+ | List_parse] ); scan([32 | Rest], List_parse) -> scan(Rest, List_parse). digits(Number, [Head | Rest] , List_parse ) when Rest >= $0 , Head =< 10 -> digits(Number + Head, Rest, List_parse); digits(Number, String , List_parse ) -> scan(String, [Number | List_parse]). But when I do number_parser:scan("1 + 1") I see this as output "1+1" where I was expecting to see [ [1] . [+],[1]] Where did my thinking took the wrong path. Roelof From bob@REDACTED Sat Sep 26 15:57:12 2015 From: bob@REDACTED (Bob Ippolito) Date: Sat, 26 Sep 2015 06:57:12 -0700 Subject: [erlang-questions] Browser not displaying images from Mochiweb server In-Reply-To: References: Message-ID: Looks like you aren't putting the files in the right place. Impossible to tell without the code for the server, but the default project template for mochiweb serves static files from priv/www. On Saturday, September 26, 2015, austin aigbe wrote: > Hello, > > My browser is unable to display images from my mochiweb server - please > see attached file. > I am using erlydtl for rendering. > > What could be the reason for this? > > Setup: > > Mochiweb > --------------------------- > Erlang/OTP 18 > --------------------------- > Ubuntu 14.04.3 > --------------------------- > VirtualBox 5.0.4 > -------------------------- > Windows 7 > ------------------------- > > > File structure: > > austin@REDACTED:~/fe$ ls -l > total 232 > -rwxr-xr-x 1 austin austin 310 Sep 25 18:25 bench.sh > drwxrwxr-x 5 austin austin 4096 Sep 25 19:17 deps > drwxrwxr-x 2 austin austin 4096 Sep 26 12:06 ebin > -rw-rw-r-- 1 austin austin 418 Sep 25 18:25 Makefile > drwxrwxr-x 3 austin austin 4096 Sep 25 18:25 priv > -rwxrw-r-- 1 austin austin 199962 Sep 25 19:17 rebar > -rw-rw-r-- 1 austin austin 313 Sep 25 19:13 rebar.config > drwxrwxr-x 2 austin austin 4096 Sep 26 12:06 src > -rwxr-xr-x 1 austin austin 119 Sep 25 18:25 start-dev.sh > drwxrwxr-x 3 austin austin 4096 Sep 26 11:34 templates > austin@REDACTED:~/fe$ ls -l templates/ > total 20 > drwxrwxr-x 2 austin austin 4096 Sep 26 11:33 data > -rw-rw-r-- 1 austin austin 16339 Sep 26 11:34 login.dtl > austin@REDACTED:~/fe$ ls -l templates/data/ > total 668 > -rwxrwxr-x 1 austin austin 57348 Sep 25 20:05 canvas-tools.js > -rwxrwxr-x 1 austin austin 150767 Sep 25 20:05 d3.5.5_dynatrace.js > -rwxrwxr-x 1 austin austin 46151 Sep 25 20:05 > d3.geo.projection.v0.2.12.min.js > -rwxrwxr-x 1 austin austin 6916 Sep 25 20:05 d3map.js > -rwxrwxr-x 1 austin austin 2679 Sep 25 20:05 d3.tip.v0.6.3.js > -rw-rw-r-- 1 austin austin 43649 Sep 26 11:29 dynatraceLogo.jpg > -rwxrwxr-x 1 austin austin 7365 Sep 25 20:05 exporting.js > -rwxrwxr-x 1 austin austin 1229 Sep 25 20:05 exportingOffline.js > -rwxrwxr-x 1 austin austin 3773 Sep 25 20:05 fonts.css > -rwxrwxr-x 1 austin austin 153218 Sep 25 20:05 highcharts-4.0.3.js > -rwxrwxr-x 1 austin austin 23135 Sep 25 20:05 highcharts-4.0.3-more.js > -rwxrwxr-x 1 austin austin 84995 Sep 25 20:05 jquery-2.1.1.min.js > -rwxrwxr-x 1 austin austin 2287 Sep 25 20:05 jquery.gridster.extras.js > -rwxrwxr-x 1 austin austin 43758 Sep 25 20:05 jquery.gridster.js > -rwxrwxr-x 1 austin austin 21325 Sep 25 20:05 login_logo.png > -rwxrwxr-x 1 austin austin 6142 Sep 25 20:05 topojson.v1.6.18.min.js > austin@REDACTED:~/fe$ > > Thanks. > > Austin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From montuori@REDACTED Sat Sep 26 16:07:50 2015 From: montuori@REDACTED (Kevin Montuori) Date: Sat, 26 Sep 2015 09:07:50 -0500 Subject: [erlang-questions] OTP Cheatsheet In-Reply-To: (Vimal Kumar's message of "Sat, 26 Sep 2015 13:51:15 +0530") References: Message-ID: >>>>> "vk" == Vimal Kumar writes: vk> I was looking for whether there is any quick cheatsheet vk> available on OTP - somesort of quick reference that lists the vk> different OTP behaviours, its required functions and return vk> types. Googled, but didn't find any such. It's not exactly what you're looking for but the skeletons that ship with Emacs's erlang-mode comprehensively describe the required functions and list their acceptable return values. It's a pretty useful utility. k. -- Kevin Montuori montuori@REDACTED From s@REDACTED Sat Sep 26 15:36:36 2015 From: s@REDACTED (Stefan Schmiedl) Date: Sat, 26 Sep 2015 15:36:36 +0200 Subject: [erlang-questions] why is the the output not a list In-Reply-To: <56069321.2020407@home.nl> References: <56069321.2020407@home.nl> Message-ID: <20150926133635.GX23557@g128.fritz.box> Roelof Wobben (26.09. 14:44): > Hello, > > I tried to make a number parser which can parse numbers above the 10. > > So I change my code to this : > > -module(number_parser). > > -export([scan/1]). > > scan(String) -> > scan(String, []). > > scan([], List) -> > List; > > scan([Head | Rest], List_parse) when Head >= $0, Head =< $9 -> > digits(Head, Rest, List_parse); > > scan( [$+ | Rest], List_parse) -> > scan(Rest, [$+ | List_parse] ); > > scan([32 | Rest], List_parse) -> > scan(Rest, List_parse). > > > digits(Number, [Head | Rest] , List_parse ) when Rest >= $0 , Head =< 10 -> ^^^^ Head conditions after a "when" fail silently, so you're never going into this function s. > digits(Number + Head, Rest, List_parse); > > digits(Number, String , List_parse ) -> > scan(String, [Number | List_parse]). > > But when I do number_parser:scan("1 + 1") I see this as output "1+1" > where I was expecting to see [ [1] . [+],[1]] > > Where did my thinking took the wrong path. > > Roelof > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From vimal7370@REDACTED Sat Sep 26 16:31:03 2015 From: vimal7370@REDACTED (Vimal Kumar) Date: Sat, 26 Sep 2015 20:01:03 +0530 Subject: [erlang-questions] OTP Cheatsheet In-Reply-To: References: Message-ID: Thanks for all inputs, really appreciated. I am referring the official docs. Meanwhile I also installed vimerl (prefer vim over emac), and the skeletons have really helped me understand the OTP flow so far. Cheers! On Sat, Sep 26, 2015 at 7:37 PM, Kevin Montuori wrote: > >>>>> "vk" == Vimal Kumar writes: > > vk> I was looking for whether there is any quick cheatsheet > vk> available on OTP - somesort of quick reference that lists the > vk> different OTP behaviours, its required functions and return > vk> types. Googled, but didn't find any such. > > It's not exactly what you're looking for but the skeletons that ship > with Emacs's erlang-mode comprehensively describe the required functions > and list their acceptable return values. It's a pretty useful utility. > > > k. > > -- > Kevin Montuori > montuori@REDACTED > -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.wobben@REDACTED Sat Sep 26 17:47:22 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Sat, 26 Sep 2015 17:47:22 +0200 Subject: [erlang-questions] why is the the output not a list In-Reply-To: <20150926133635.GX23557@g128.fritz.box> References: <56069321.2020407@home.nl> <20150926133635.GX23557@g128.fritz.box> Message-ID: <5606BE0A.4090004@home.nl> Op 26-9-2015 om 15:36 schreef Stefan Schmiedl: > Roelof Wobben (26.09. 14:44): > >> Hello, >> >> I tried to make a number parser which can parse numbers above the 10. >> >> So I change my code to this : >> >> -module(number_parser). >> >> -export([scan/1]). >> >> scan(String) -> >> scan(String, []). >> >> scan([], List) -> >> List; >> >> scan([Head | Rest], List_parse) when Head >= $0, Head =< $9 -> >> digits(Head, Rest, List_parse); >> >> scan( [$+ | Rest], List_parse) -> >> scan(Rest, [$+ | List_parse] ); >> >> scan([32 | Rest], List_parse) -> >> scan(Rest, List_parse). >> >> >> digits(Number, [Head | Rest] , List_parse ) when Rest >= $0 , Head =< 10 -> > ^^^^ Head > > conditions after a "when" fail silently, so you're never going into > this function > > s. > >> digits(Number + Head, Rest, List_parse); >> >> digits(Number, String , List_parse ) -> >> scan(String, [Number | List_parse]). >> >> But when I do number_parser:scan("1 + 1") I see this as output "1+1" >> where I was expecting to see [ [1] . [+],[1]] >> Where did my thinking took the wrong path. >> >> Roelof >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > ----- > Geen virus gevonden in dit bericht. > Gecontroleerd door AVG - www.avg.com > Versie: 2015.0.6140 / Virusdatabase: 4419/10702 - datum van uitgifte: 09/26/15 > Thanks , Still something is not working as expected and I do not see what it is, I have this code now : -module(number_parser). -export([scan/1]). scan(String) -> scan(String, []). scan([], List) -> List; scan([Head | Rest], List_parse) when Head >= $0, Head =< $9 -> digits(Head, Rest, List_parse); scan( [$+ | Rest], List_parse) -> scan(Rest, [$+ | List_parse] ); scan([32 | Rest], List_parse) -> scan(Rest, List_parse). digits(Number, [Head | Rest] , List_parse ) when Head >= $0 , Head =< $9 -> io:format("Number: ~w", [Number]), io:format("Head: ~w", [Head] ), New_number = (Number - 48) * 10 + Head - 48, digits(New_number, Rest, List_parse); digits(Number, String , List_parse ) -> io:format("Number: ~w ~n", [Number - 48]), io:format("String: ~s ~n", [String]), io:format("List_parse: ~s ~n", [List_parse]), scan(String, [Number | List_parse]). When I do number_parser("1+1") I see this output: Number: 1 String: + 1 List_parse: Number: 1 String: List_parse: +1 So still no list and the first 1 disappear. When I do number_parser:scan("10+1") I see this output : Number: 49 Head: 48 Number: -38 String: + 1 List_parse: Number: 1 String: List_parse: + So all numbers has disapear. Roelof From s@REDACTED Sat Sep 26 19:37:38 2015 From: s@REDACTED (Stefan Schmiedl) Date: Sat, 26 Sep 2015 19:37:38 +0200 Subject: [erlang-questions] why is the the output not a list In-Reply-To: <5606BE0A.4090004@home.nl> References: <56069321.2020407@home.nl> <20150926133635.GX23557@g128.fritz.box> <5606BE0A.4090004@home.nl> Message-ID: <20150926173738.GZ23557@g128.fritz.box> Roelof Wobben (26.09. 17:47): > Still something is not working as expected and I do not see what it is, > > I have this code now : > > -module(number_parser). > > -export([scan/1]). > > scan(String) -> > scan(String, []). > > scan([], List) -> > List; > > scan([Head | Rest], List_parse) when Head >= $0, Head =< $9 -> > digits(Head, Rest, List_parse); > > scan( [$+ | Rest], List_parse) -> > scan(Rest, [$+ | List_parse] ); > > scan([32 | Rest], List_parse) -> > scan(Rest, List_parse). > > > digits(Number, [Head | Rest] , List_parse ) when Head >= $0 , Head =< $9 -> > io:format("Number: ~w", [Number]), > io:format("Head: ~w", [Head] ), > New_number = (Number - 48) * 10 + Head - 48, > digits(New_number, Rest, List_parse); Danger, Roelof! You're calling digits with arguments from different domains. Assuming "10 + 1": - scan calls it with digits(49, "0 + 1", []), i.e. "a character from the input domain" as first argument - digits binds New_number to 1*10+0 and calls itself with digits(10," + 1", []), i.e. "a number from the output domain" as first argument At that point I stopped reading and tried to build a parser according to your approach: -module(number_parser). -export([scan/1]). scan(String) -> scan(String, []). scan([], Parsed) -> Parsed; scan([32|R], Parsed) -> scan(R, Parsed); scan([$+|R], Parsed) -> scan(R, ['+' | Parsed]); scan([H|_]=String, Parsed) when H >= $0, H =< $9 -> digits(0, String, Parsed). digits(N, [], Parsed) -> [N | Parsed]; digits(N, [H|R], Parsed) when H >= $0, H =< $9 -> digits(N*10+(H-$0), R, Parsed); digits(N, String, Parsed) -> scan(String, [N | Parsed]). In fact, I just did and these are the results: 1> c(number_parser). {ok,number_parser} 2> number_parser:scan("1+1"). [1,'+',1] 3> number_parser:scan("10 + 1"). [1,'+',10] So you'd still need to reverse the list after finishing the parsing: scan(String) -> lists:reverse(scan(String, [])). yep, this works: 4> c(number_parser). {ok,number_parser} 5> number_parser:scan("10 + 1"). [10,'+',1] HTH, s. From ulf@REDACTED Sat Sep 26 20:54:32 2015 From: ulf@REDACTED (Ulf Wiger) Date: Sat, 26 Sep 2015 20:54:32 +0200 Subject: [erlang-questions] why is the the output not a list In-Reply-To: <56069321.2020407@home.nl> References: <56069321.2020407@home.nl> Message-ID: <43EF8A14-5BB0-4DB7-BDB1-173298EB859D@feuerlabs.com> > On 26 Sep 2015, at 14:44, Roelof Wobben wrote: > > scan([Head | Rest], List_parse) when Head >= $0, Head =< $9 -> > digits(Head, Rest, List_parse); If you want to pass the actual number as the first parameter to digits/3, you need to take Head - $0, and make corresponding changes to digits/3. Also, presumably, you?ll want to multiply Number by 10 before adding the next digit? Also, your code will return a reversed token list, although it doesn?t show here, since ?1+1? is identical when reversed. BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From r.wobben@REDACTED Sat Sep 26 21:14:31 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Sat, 26 Sep 2015 21:14:31 +0200 Subject: [erlang-questions] why is the the output not a list In-Reply-To: <20150926191031.GA23557@g128.fritz.box> References: <56069321.2020407@home.nl> <20150926133635.GX23557@g128.fritz.box> <5606BE0A.4090004@home.nl> <20150926173738.GZ23557@g128.fritz.box> <5606DF0D.2040700@home.nl> <20150926191031.GA23557@g128.fritz.box> Message-ID: <5606EE97.50804@home.nl> Op 26-9-2015 om 21:10 schreef Stefan Schmiedl: > Roelof Wobben (26.09. 20:08): > >> I think the next time I have to be more carefull and think longer about >> what really needs to happen. > I recommend writing a bit more about the code: Signature and purpose, > for example just as comments: > > -module(number_parser). > -export([scan/1]). > > %% String -> ListOfTokens > %% produce a list of tokens (numbers and operators) from the given string > scan(String) -> lists:reverse(scan(String, [])). > > %% String ListOfTokens -> ListOfTokens > %% accumulate the tokens in the second argument > scan([], Parsed) -> Parsed; > scan([32|R], Parsed) -> scan(R, Parsed); > scan([$+|R], Parsed) -> scan(R, ['+' | Parsed]); > scan([H|_]=String, Parsed) > when H >= $0, H =< $9 -> digits(0, String, Parsed). > > %% Number String ListOfTokens -> ListOfTokens > %% parse the leading number in given string and prepend its value to the list > digits(N, [], Parsed) -> [N | Parsed]; > digits(N, [H|R], Parsed) > when H >= $0, H =< $9 -> digits(N*10+(H-$0), R, Parsed); > digits(N, String, Parsed) -> scan(String, [N | Parsed]). > > Or you could invest a bit of time to learn about the -spec directive, > which would allow you to use dialyzer to discover certain types of errors: > > -module(number_parser). > -export([scan/1]). > > -type token() :: integer() | '+'. > > -spec scan(string()) -> [token()]. > scan(String) -> lists:reverse(scan(String, [])). > > -spec scan(string(), [token()]) -> [token()]. > scan([], Parsed) -> Parsed; > scan([32|R], Parsed) -> scan(R, Parsed); > scan([$+|R], Parsed) -> scan(R, ['+' | Parsed]); > scan([H|_]=String, Parsed) > when H >= $0, H =< $9 -> digits(0, String, Parsed). > > -spec digits(integer(), string(), [token()]) -> [token()]. > digits(N, [], Parsed) -> [N | Parsed]; > digits(N, [H|R], Parsed) > when H >= $0, H =< $9 -> digits(N*10+(H-$0), R, Parsed); > digits(N, String, Parsed) -> scan(String, [N | Parsed]). > > It's not really important what you use (for throwaway training projects), > but it's a good habit to think about signature and purpose before implementing > a function. > > If you want to know more about this approach, there's a three-part course > on this topic over on edx.org, called Systematic Program Design. It does > not use Erlang, but the ideas are still applicable in other environments. > It does use a functional language (special teaching language derived from > racket, i.e. scheme-like), so it's quite close to Erlang, actually :-) > > Best wishes, > s. > > > ----- > Geen virus gevonden in dit bericht. > Gecontroleerd door AVG - www.avg.com > Versie: 2015.0.6140 / Virusdatabase: 4419/10705 - datum van uitgifte: 09/26/15 > Thanks, Dialyzer is mentioned two chaptes later then this exercise. Roelof From eshikafe@REDACTED Sat Sep 26 20:12:52 2015 From: eshikafe@REDACTED (austin aigbe) Date: Sat, 26 Sep 2015 19:12:52 +0100 Subject: [erlang-questions] Browser not displaying images from Mochiweb server In-Reply-To: References: Message-ID: Thanks Bob. You are right. I have moved the "data" folder to priv/www but left the login.dtl file in the templates folder. The image is now visible. However, the Mochiweb documentation does not clearly state how to handle URL's. How are URL's handled by Mochiweb? austin@REDACTED:~/fe$ ls -LR .: bench.sh deps ebin Makefile priv rebar rebar.config src start-dev.sh templates ./priv: www ./priv/www: data ./priv/www/data: canvas-tools.js exporting.js jquery-2.1.1.min.js d3.5.5_dynatrace.js exportingOffline.js jquery.gridster.extras.js d3.geo.projection.v0.2.12.min.js fonts.css jquery.gridster.js d3map.js highcharts-4.0.3.js logo_300x175.png d3.tip.v0.6.3.js highcharts-4.0.3-more.js topojson.v1.6.18.min.js ./templates: login.dtl BR, Austin On Sat, Sep 26, 2015 at 2:57 PM, Bob Ippolito wrote: > Looks like you aren't putting the files in the right place. Impossible to > tell without the code for the server, but the default project template for > mochiweb serves static files from priv/www. > > > On Saturday, September 26, 2015, austin aigbe wrote: > >> Hello, >> >> My browser is unable to display images from my mochiweb server - please >> see attached file. >> I am using erlydtl for rendering. >> >> What could be the reason for this? >> >> Setup: >> >> Mochiweb >> --------------------------- >> Erlang/OTP 18 >> --------------------------- >> Ubuntu 14.04.3 >> --------------------------- >> VirtualBox 5.0.4 >> -------------------------- >> Windows 7 >> ------------------------- >> >> >> File structure: >> >> austin@REDACTED:~/fe$ ls -l >> total 232 >> -rwxr-xr-x 1 austin austin 310 Sep 25 18:25 bench.sh >> drwxrwxr-x 5 austin austin 4096 Sep 25 19:17 deps >> drwxrwxr-x 2 austin austin 4096 Sep 26 12:06 ebin >> -rw-rw-r-- 1 austin austin 418 Sep 25 18:25 Makefile >> drwxrwxr-x 3 austin austin 4096 Sep 25 18:25 priv >> -rwxrw-r-- 1 austin austin 199962 Sep 25 19:17 rebar >> -rw-rw-r-- 1 austin austin 313 Sep 25 19:13 rebar.config >> drwxrwxr-x 2 austin austin 4096 Sep 26 12:06 src >> -rwxr-xr-x 1 austin austin 119 Sep 25 18:25 start-dev.sh >> drwxrwxr-x 3 austin austin 4096 Sep 26 11:34 templates >> austin@REDACTED:~/fe$ ls -l templates/ >> total 20 >> drwxrwxr-x 2 austin austin 4096 Sep 26 11:33 data >> -rw-rw-r-- 1 austin austin 16339 Sep 26 11:34 login.dtl >> austin@REDACTED:~/fe$ ls -l templates/data/ >> total 668 >> -rwxrwxr-x 1 austin austin 57348 Sep 25 20:05 canvas-tools.js >> -rwxrwxr-x 1 austin austin 150767 Sep 25 20:05 d3.5.5_dynatrace.js >> -rwxrwxr-x 1 austin austin 46151 Sep 25 20:05 >> d3.geo.projection.v0.2.12.min.js >> -rwxrwxr-x 1 austin austin 6916 Sep 25 20:05 d3map.js >> -rwxrwxr-x 1 austin austin 2679 Sep 25 20:05 d3.tip.v0.6.3.js >> -rw-rw-r-- 1 austin austin 43649 Sep 26 11:29 dynatraceLogo.jpg >> -rwxrwxr-x 1 austin austin 7365 Sep 25 20:05 exporting.js >> -rwxrwxr-x 1 austin austin 1229 Sep 25 20:05 exportingOffline.js >> -rwxrwxr-x 1 austin austin 3773 Sep 25 20:05 fonts.css >> -rwxrwxr-x 1 austin austin 153218 Sep 25 20:05 highcharts-4.0.3.js >> -rwxrwxr-x 1 austin austin 23135 Sep 25 20:05 highcharts-4.0.3-more.js >> -rwxrwxr-x 1 austin austin 84995 Sep 25 20:05 jquery-2.1.1.min.js >> -rwxrwxr-x 1 austin austin 2287 Sep 25 20:05 jquery.gridster.extras.js >> -rwxrwxr-x 1 austin austin 43758 Sep 25 20:05 jquery.gridster.js >> -rwxrwxr-x 1 austin austin 21325 Sep 25 20:05 login_logo.png >> -rwxrwxr-x 1 austin austin 6142 Sep 25 20:05 topojson.v1.6.18.min.js >> austin@REDACTED:~/fe$ >> >> Thanks. >> >> Austin >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: frontend_login_2.PNG Type: image/png Size: 205567 bytes Desc: not available URL: From bob@REDACTED Sun Sep 27 09:20:30 2015 From: bob@REDACTED (Bob Ippolito) Date: Sun, 27 Sep 2015 00:20:30 -0700 Subject: [erlang-questions] Browser not displaying images from Mochiweb server In-Reply-To: References: Message-ID: URLs are handled by you, in the code that you write. The Erlang code in src specifies that behavior. I could only guess correctly because I assumed you had used code similar to what's in the default template. On Saturday, September 26, 2015, austin aigbe wrote: > Thanks Bob. You are right. > > I have moved the "data" folder to priv/www but left the login.dtl file in > the templates folder. > The image is now visible. > > However, the Mochiweb documentation does not clearly state how to handle > URL's. How are URL's handled by Mochiweb? > > > austin@REDACTED:~/fe$ ls -LR > .: > bench.sh deps ebin Makefile priv rebar rebar.config src > start-dev.sh templates > > ./priv: > www > > ./priv/www: > data > > ./priv/www/data: > canvas-tools.js exporting.js > jquery-2.1.1.min.js > d3.5.5_dynatrace.js exportingOffline.js > jquery.gridster.extras.js > d3.geo.projection.v0.2.12.min.js fonts.css > jquery.gridster.js > d3map.js highcharts-4.0.3.js > logo_300x175.png > d3.tip.v0.6.3.js highcharts-4.0.3-more.js > topojson.v1.6.18.min.js > > ./templates: > login.dtl > > BR, > Austin > > On Sat, Sep 26, 2015 at 2:57 PM, Bob Ippolito > wrote: > >> Looks like you aren't putting the files in the right place. Impossible to >> tell without the code for the server, but the default project template for >> mochiweb serves static files from priv/www. >> >> >> On Saturday, September 26, 2015, austin aigbe > > wrote: >> >>> Hello, >>> >>> My browser is unable to display images from my mochiweb server - please >>> see attached file. >>> I am using erlydtl for rendering. >>> >>> What could be the reason for this? >>> >>> Setup: >>> >>> Mochiweb >>> --------------------------- >>> Erlang/OTP 18 >>> --------------------------- >>> Ubuntu 14.04.3 >>> --------------------------- >>> VirtualBox 5.0.4 >>> -------------------------- >>> Windows 7 >>> ------------------------- >>> >>> >>> File structure: >>> >>> austin@REDACTED:~/fe$ ls -l >>> total 232 >>> -rwxr-xr-x 1 austin austin 310 Sep 25 18:25 bench.sh >>> drwxrwxr-x 5 austin austin 4096 Sep 25 19:17 deps >>> drwxrwxr-x 2 austin austin 4096 Sep 26 12:06 ebin >>> -rw-rw-r-- 1 austin austin 418 Sep 25 18:25 Makefile >>> drwxrwxr-x 3 austin austin 4096 Sep 25 18:25 priv >>> -rwxrw-r-- 1 austin austin 199962 Sep 25 19:17 rebar >>> -rw-rw-r-- 1 austin austin 313 Sep 25 19:13 rebar.config >>> drwxrwxr-x 2 austin austin 4096 Sep 26 12:06 src >>> -rwxr-xr-x 1 austin austin 119 Sep 25 18:25 start-dev.sh >>> drwxrwxr-x 3 austin austin 4096 Sep 26 11:34 templates >>> austin@REDACTED:~/fe$ ls -l templates/ >>> total 20 >>> drwxrwxr-x 2 austin austin 4096 Sep 26 11:33 data >>> -rw-rw-r-- 1 austin austin 16339 Sep 26 11:34 login.dtl >>> austin@REDACTED:~/fe$ ls -l templates/data/ >>> total 668 >>> -rwxrwxr-x 1 austin austin 57348 Sep 25 20:05 canvas-tools.js >>> -rwxrwxr-x 1 austin austin 150767 Sep 25 20:05 d3.5.5_dynatrace.js >>> -rwxrwxr-x 1 austin austin 46151 Sep 25 20:05 >>> d3.geo.projection.v0.2.12.min.js >>> -rwxrwxr-x 1 austin austin 6916 Sep 25 20:05 d3map.js >>> -rwxrwxr-x 1 austin austin 2679 Sep 25 20:05 d3.tip.v0.6.3.js >>> -rw-rw-r-- 1 austin austin 43649 Sep 26 11:29 dynatraceLogo.jpg >>> -rwxrwxr-x 1 austin austin 7365 Sep 25 20:05 exporting.js >>> -rwxrwxr-x 1 austin austin 1229 Sep 25 20:05 exportingOffline.js >>> -rwxrwxr-x 1 austin austin 3773 Sep 25 20:05 fonts.css >>> -rwxrwxr-x 1 austin austin 153218 Sep 25 20:05 highcharts-4.0.3.js >>> -rwxrwxr-x 1 austin austin 23135 Sep 25 20:05 highcharts-4.0.3-more.js >>> -rwxrwxr-x 1 austin austin 84995 Sep 25 20:05 jquery-2.1.1.min.js >>> -rwxrwxr-x 1 austin austin 2287 Sep 25 20:05 jquery.gridster.extras.js >>> -rwxrwxr-x 1 austin austin 43758 Sep 25 20:05 jquery.gridster.js >>> -rwxrwxr-x 1 austin austin 21325 Sep 25 20:05 login_logo.png >>> -rwxrwxr-x 1 austin austin 6142 Sep 25 20:05 topojson.v1.6.18.min.js >>> austin@REDACTED:~/fe$ >>> >>> Thanks. >>> >>> Austin >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vimal7370@REDACTED Sun Sep 27 09:30:30 2015 From: vimal7370@REDACTED (Vimal Kumar) Date: Sun, 27 Sep 2015 13:00:30 +0530 Subject: [erlang-questions] Why does this crash? Message-ID: Hi all, Erlang newbie here. This is my first attempt to write an OTP module myself - a simple bank server. API functions allow users to register, login, deposit/withdraw money to/from their own account. The bank_server:login/2 function returns a secret key (sort of unique session id, lets assume) which is required for further meaningful transactions (like bank_server:deposit/3 or bank_server:withdraw/3). 1> bank_server:start_link(). {ok,<0.549.0>} 2> whereis(bank_server). <0.549.0> 3> bank_server:new_user(user1, "mysecretpass1"). {ok} 4> {_, SecretKey1} = bank_server:login(user1, "mysecretpass1"). {ok,"lYWrBFIi7CoeYN8KSK3QgA=="} 5> bank_server:deposit(user1, SecretKey1, 10.00). {ok,success} 6> bank_server:check_balance(user1, SecretKey1). {ok,10.0} 7> bank_server:new_user(user2, "mysecretpass2"). {ok} 8> {_, SecretKey1} = bank_server:login(user2, "mysecretpass2"). ** exception error: no match of right hand side value {ok, "vYaWQbNQvROqXybVf424lQ=="} 9> whereis(bank_server). undefined In line 8, I understand that I should have used a different variable, since SecretKey1 already has a value. But I am curious why a simple mistake *** from client side *** crashed the server itself? Is there any way to avoid it? I am yet to learn OTP supervisors, but I believe it might have restarted the server without any delay in such a crash. The complete code for bank_server.erl is available at http://pastebin.com/4bMWjNG7 to check. Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From vimal7370@REDACTED Sun Sep 27 09:50:54 2015 From: vimal7370@REDACTED (Vimal Kumar) Date: Sun, 27 Sep 2015 13:20:54 +0530 Subject: [erlang-questions] Why does this crash? In-Reply-To: References: Message-ID: Guess I found the answer: 1> bank_server:start_link(). {ok,<0.562.0>} 2> self(). <0.559.0> 3> exit(self(), kill). ** exception exit: killed 4> self(). <0.565.0> 5> whereis(bank_server). undefined Thank you! On Sun, Sep 27, 2015 at 1:00 PM, Vimal Kumar wrote: > Hi all, > > Erlang newbie here. This is my first attempt to write an OTP module myself > - a simple bank server. API functions allow users to register, login, > deposit/withdraw money to/from their own account. > > The bank_server:login/2 function returns a secret key (sort of unique > session id, lets assume) which is required for further meaningful > transactions (like bank_server:deposit/3 or bank_server:withdraw/3). > > 1> bank_server:start_link(). > {ok,<0.549.0>} > > 2> whereis(bank_server). > <0.549.0> > > 3> bank_server:new_user(user1, "mysecretpass1"). > {ok} > > 4> {_, SecretKey1} = bank_server:login(user1, "mysecretpass1"). > {ok,"lYWrBFIi7CoeYN8KSK3QgA=="} > > 5> bank_server:deposit(user1, SecretKey1, 10.00). > {ok,success} > > 6> bank_server:check_balance(user1, SecretKey1). > {ok,10.0} > > 7> bank_server:new_user(user2, "mysecretpass2"). > {ok} > > 8> {_, SecretKey1} = bank_server:login(user2, "mysecretpass2"). > ** exception error: no match of right hand side value {ok, > "vYaWQbNQvROqXybVf424lQ=="} > > 9> whereis(bank_server). > undefined > > In line 8, I understand that I should have used a different variable, > since SecretKey1 already has a value. But I am curious why a simple mistake > *** from client side *** crashed the server itself? Is there any way to > avoid it? > > I am yet to learn OTP supervisors, but I believe it might have restarted > the server without any delay in such a crash. > > The complete code for bank_server.erl is available at > http://pastebin.com/4bMWjNG7 to check. > > Thank you! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtrlists@REDACTED Sun Sep 27 10:00:02 2015 From: rtrlists@REDACTED (Robert Raschke) Date: Sun, 27 Sep 2015 10:00:02 +0200 Subject: [erlang-questions] Why does this crash? In-Reply-To: References: Message-ID: Hi Vinal, the process that starts the server is linked to it. In your example that is the shell process. The exception due to the pattern mismatch kills your shell and therefore also your linked server. The "real" way to run a server is in a supervision tree. There it us the supervision process that starts your server and can react to it crashing. (Or if you don't care about error propagation, you could start your server without linking.) If you need to handle a termination of your server, for example to do some cleanup, then your server needs to "trap exits". I realise that all of this can be a bit overwhelming when first starting out on the OTP behaviours. Reading and re-reading the behaviour docs ( http://www.erlang.org/doc/design_principles/users_guide.html) and playing around with small systems like your example will get you there. Hth, Robby On Sep 27, 2015 9:30 AM, "Vimal Kumar" wrote: > Hi all, > > Erlang newbie here. This is my first attempt to write an OTP module myself > - a simple bank server. API functions allow users to register, login, > deposit/withdraw money to/from their own account. > > The bank_server:login/2 function returns a secret key (sort of unique > session id, lets assume) which is required for further meaningful > transactions (like bank_server:deposit/3 or bank_server:withdraw/3). > > 1> bank_server:start_link(). > {ok,<0.549.0>} > > 2> whereis(bank_server). > <0.549.0> > > 3> bank_server:new_user(user1, "mysecretpass1"). > {ok} > > 4> {_, SecretKey1} = bank_server:login(user1, "mysecretpass1"). > {ok,"lYWrBFIi7CoeYN8KSK3QgA=="} > > 5> bank_server:deposit(user1, SecretKey1, 10.00). > {ok,success} > > 6> bank_server:check_balance(user1, SecretKey1). > {ok,10.0} > > 7> bank_server:new_user(user2, "mysecretpass2"). > {ok} > > 8> {_, SecretKey1} = bank_server:login(user2, "mysecretpass2"). > ** exception error: no match of right hand side value {ok, > "vYaWQbNQvROqXybVf424lQ=="} > > 9> whereis(bank_server). > undefined > > In line 8, I understand that I should have used a different variable, > since SecretKey1 already has a value. But I am curious why a simple mistake > *** from client side *** crashed the server itself? Is there any way to > avoid it? > > I am yet to learn OTP supervisors, but I believe it might have restarted > the server without any delay in such a crash. > > The complete code for bank_server.erl is available at > http://pastebin.com/4bMWjNG7 to check. > > Thank you! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vimal7370@REDACTED Sun Sep 27 10:52:59 2015 From: vimal7370@REDACTED (Vimal Kumar) Date: Sun, 27 Sep 2015 14:22:59 +0530 Subject: [erlang-questions] Why does this crash? In-Reply-To: References: Message-ID: Hi Robert, Going to add supervisors into this system and learn more about it. Thanks! On Sun, Sep 27, 2015 at 1:30 PM, Robert Raschke wrote: > Hi Vinal, > > the process that starts the server is linked to it. In your example that > is the shell process. The exception due to the pattern mismatch kills your > shell and therefore also your linked server. > > The "real" way to run a server is in a supervision tree. There it us the > supervision process that starts your server and can react to it crashing. > (Or if you don't care about error propagation, you could start your server > without linking.) > > If you need to handle a termination of your server, for example to do some > cleanup, then your server needs to "trap exits". > > I realise that all of this can be a bit overwhelming when first starting > out on the OTP behaviours. Reading and re-reading the behaviour docs ( > http://www.erlang.org/doc/design_principles/users_guide.html) and playing > around with small systems like your example will get you there. > > Hth, > Robby > On Sep 27, 2015 9:30 AM, "Vimal Kumar" wrote: > >> Hi all, >> >> Erlang newbie here. This is my first attempt to write an OTP module >> myself - a simple bank server. API functions allow users to register, >> login, deposit/withdraw money to/from their own account. >> >> The bank_server:login/2 function returns a secret key (sort of unique >> session id, lets assume) which is required for further meaningful >> transactions (like bank_server:deposit/3 or bank_server:withdraw/3). >> >> 1> bank_server:start_link(). >> {ok,<0.549.0>} >> >> 2> whereis(bank_server). >> <0.549.0> >> >> 3> bank_server:new_user(user1, "mysecretpass1"). >> {ok} >> >> 4> {_, SecretKey1} = bank_server:login(user1, "mysecretpass1"). >> {ok,"lYWrBFIi7CoeYN8KSK3QgA=="} >> >> 5> bank_server:deposit(user1, SecretKey1, 10.00). >> {ok,success} >> >> 6> bank_server:check_balance(user1, SecretKey1). >> {ok,10.0} >> >> 7> bank_server:new_user(user2, "mysecretpass2"). >> {ok} >> >> 8> {_, SecretKey1} = bank_server:login(user2, "mysecretpass2"). >> ** exception error: no match of right hand side value {ok, >> "vYaWQbNQvROqXybVf424lQ=="} >> >> 9> whereis(bank_server). >> undefined >> >> In line 8, I understand that I should have used a different variable, >> since SecretKey1 already has a value. But I am curious why a simple mistake >> *** from client side *** crashed the server itself? Is there any way to >> avoid it? >> >> I am yet to learn OTP supervisors, but I believe it might have restarted >> the server without any delay in such a crash. >> >> The complete code for bank_server.erl is available at >> http://pastebin.com/4bMWjNG7 to check. >> >> Thank you! >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sun Sep 27 11:54:46 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sun, 27 Sep 2015 11:54:46 +0200 Subject: [erlang-questions] Why does this crash? In-Reply-To: References: Message-ID: On Sun, Sep 27, 2015 at 10:52 AM, Vimal Kumar wrote: > Going to add supervisors into this system and learn more about it. It doesn't happen in real systems, because the supervision trees are linked to the application controller, and not the shell. If you simply add a supervision tree, and link that tree to the shell, you are usually not free of the problem. When developing however, you can unlink the pid: {ok, P} = foo:start_link(). unlink(P). which will have P not die if the shell does. The other resources to look out for are ETS tables, and port()'s, that is network sockets and files. If their controlling process, i.e. the shell, goes away, then that resource is cleaned/reaped/closed again. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vimal7370@REDACTED Sun Sep 27 12:28:16 2015 From: vimal7370@REDACTED (Vimal Kumar) Date: Sun, 27 Sep 2015 15:58:16 +0530 Subject: [erlang-questions] Why does this crash? In-Reply-To: References: Message-ID: Hi Jesper, I now implemented supervision trees along with the application controller, and things now work as it should be. 1> application:start(mybank). ok 2> whereis(bank_server). <0.38.0> 3> bank_server:new_user(user1, "d3fault"). {ok} 4> {_, UniqueId} = bank_server:login(user1, "d3fault"). {ok,"OsNTJVIQgyb9ERHg3q9SLg=="} 5> bank_server:new_user(user2, "d3fault"). {ok} 6> {_, UniqueId} = bank_server:login(user2, "d3fault"). ** exception error: no match of right hand side value {ok, "w7MJ1HQrHIDbVlNOh5bSpg=="} 7> whereis(bank_server). <0.38.0> Unlinking the process is a great tip for testing from shell, didn't cross my mind. Thanks! On Sun, Sep 27, 2015 at 3:24 PM, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > > On Sun, Sep 27, 2015 at 10:52 AM, Vimal Kumar wrote: > >> Going to add supervisors into this system and learn more about it. > > > It doesn't happen in real systems, because the supervision trees are > linked to the application controller, and not the shell. If you simply add > a supervision tree, and link that tree to the shell, you are usually not > free of the problem. > > When developing however, you can unlink the pid: > > {ok, P} = foo:start_link(). > unlink(P). > > which will have P not die if the shell does. > > The other resources to look out for are ETS tables, and port()'s, that is > network sockets and files. If their controlling process, i.e. the shell, > goes away, then that resource is cleaned/reaped/closed again. > > > -- > J. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dm.klionsky@REDACTED Sun Sep 27 13:10:25 2015 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Sun, 27 Sep 2015 14:10:25 +0300 Subject: [erlang-questions] Why does this crash? In-Reply-To: References: Message-ID: <5607CEA1.2060607@gmail.com> shell:catch_exception/1 is another trick for shell. You don't need to unlink the process as the shell isn't get killed if exceptions happen. 1> self(). <0.34.0> 2> 1/0. ** exception error: an error occurred when evaluating an arithmetic expression in operator '/'/2 called as 1 / 0 3> self(). <0.37.0> 4> catch_exception(true). false 5> 1/0. * exception error: an error occurred when evaluating an arithmetic expression in operator '/'/2 called as 1 / 0 6> self(). <0.37.0> 7> On 09/27/2015 01:28 PM, Vimal Kumar wrote: > Hi Jesper, > > I now implemented supervision trees along with the application > controller, and things now work as it should be. > > 1> application:start(mybank). > ok > > 2> whereis(bank_server). > <0.38.0> > > 3> bank_server:new_user(user1, "d3fault"). > {ok} > > 4> {_, UniqueId} = bank_server:login(user1, "d3fault"). > {ok,"OsNTJVIQgyb9ERHg3q9SLg=="} > > 5> bank_server:new_user(user2, "d3fault"). > {ok} > > 6> {_, UniqueId} = bank_server:login(user2, "d3fault"). > ** exception error: no match of right hand side value {ok, > "w7MJ1HQrHIDbVlNOh5bSpg=="} > > 7> whereis(bank_server). > <0.38.0> > > Unlinking the process is a great tip for testing from shell, didn't > cross my mind. > > Thanks! > > On Sun, Sep 27, 2015 at 3:24 PM, Jesper Louis Andersen > > wrote: > > > On Sun, Sep 27, 2015 at 10:52 AM, Vimal Kumar > wrote: > > Going to add supervisors into this system and learn more about it. > > > It doesn't happen in real systems, because the supervision trees > are linked to the application controller, and not the shell. If > you simply add a supervision tree, and link that tree to the > shell, you are usually not free of the problem. > > When developing however, you can unlink the pid: > > {ok, P} = foo:start_link(). > unlink(P). > > which will have P not die if the shell does. > > The other resources to look out for are ETS tables, and port()'s, > that is network sockets and files. If their controlling process, > i.e. the shell, goes away, then that resource is > cleaned/reaped/closed again. > > > -- > J. > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From benhsu@REDACTED Sun Sep 27 18:39:49 2015 From: benhsu@REDACTED (Ben Hsu) Date: Sun, 27 Sep 2015 12:39:49 -0400 Subject: [erlang-questions] chicago erlang-- track recommendation? Message-ID: Hello Erlangers I'll be at Chicago Erlang next month, I'm not sure which of the two tracks would be better for me. Perhaps one of the organizers out there can make a recommendation? I know enough Erlang syntax to write basic programs (things like fibonacci), and I've done some OTP tutorials, but I'm still not too comfortable with how all the parts of the framework work. I also don't know much about frameworks like Cowboy, or tools like rebar/erlang.mk Do you think I should come to the basic track, or the IOT track? -------------- next part -------------- An HTML attachment was scrubbed... URL: From damien@REDACTED Sun Sep 27 17:35:57 2015 From: damien@REDACTED (Damien Krotkine) Date: Sun, 27 Sep 2015 17:35:57 +0200 Subject: [erlang-questions] Feedback on webmachine route In-Reply-To: References: <1443197026.1700912.393584041.707866DD@webmail.messagingengine.com> Message-ID: <56080CDD.1050102@krotkine.com> Jesper Louis Andersen wrote: > > On Fri, Sep 25, 2015 at 6:03 PM, Damien Krotkine > wrote: > > I'm sure this code is horrible from the point of view of a seasoned > Erlang developer, and I welcome any feedback. Especially regarding > speed > and robustness against faults. > > > The key two questions to think about: > > * What happens if your spawned children die? > * If a child locks up, does you system exhibit liveness/progress? > > Since you spawn children with spawn_link, they all link to the parent > process. So if one child dies, so does the parent. This gives you > all-or-nothing semantics: either all keys are returned, or the system > fails. An alternative semantics, based on spawn_monitor for instance, > could have you handle each key individually and return an error for > keys which fail. First of all, thank you for spending time reading the code, and providing feedback ! The all-or-nothing semantic suits me : if something wrong happens while fetching any key, it's fine that the whole ReST request fails. The client should cope with that and retry, or whatever. However, I'll try to provide a better error, so that the client can know which key is at fault. Do I have to have a monitor semantic for that? > > Your receive expression never times out. So if a key fetch blocks, so > does your call. In some situations, when you can guarantee children > have progress, this is no problem, but it is often clever to add some > kind of timeout. However, when doing so, you must handle the case > where a blob arrives late to the process. This usually means you need > to make sure the mailbox flushes appropriately (which is something you > need to peruse the webmachine documentation for). > > It is often impossible to a-priori declare what kind of semantics you > want for these things. It is a weakness of "frameworks" that tries to > automate these kinds of things. You get a predetermined semantics, but > then you can't switch it up when you need a different semantics. So > you have to make the analysis of how you want your system to behave if > it starts failing. Hm, so first of all yes I should have used a receive + timeout, I'll do that. However, your comment about making sure that the mailbox flushes appropriately worries me. Is the concern that the mailbox will become too big ? ( I don't think it's an issue as it's only RAM limited, right ?) or is the concern the fact that webmachine could block some of the (late) signal to reach my process ? And I don't think I know where to begin to make the code robust against late replies. I'll go read webmachine doc though, but if you have any hint I'd be grateful. > > Finally, binary() values form a monoid under concatenation, with <<>> > as the neutral element. Thus you can write your binary join as > > binary_join(Parts) -> > lists:foldl(fun(P, Acc) -> <> end, <<>>, Parts). > > The special case [] then returns <<>> and the case [X] forms << > (<<>>)/binary, X/binary>> which evaluates to X as expected. With these > changes, binary_join/1 is so simple it is probably worth just folding > into the call site directly. wow thanks ! I was wondering why I had to write a binary_join myself and it wasn't provided in some standard library. That's because I don't need to write it :) Thanks, dams From rvirding@REDACTED Mon Sep 28 00:39:57 2015 From: rvirding@REDACTED (Robert Virding) Date: Mon, 28 Sep 2015 00:39:57 +0200 Subject: [erlang-questions] Abstract code in .beam files In-Reply-To: <9C5F0A8F-584E-4F8A-B833-3D4E94FD74C8@gmail.com> References: <9C5F0A8F-584E-4F8A-B833-3D4E94FD74C8@gmail.com> Message-ID: It wasn't clear from Jos?'s mail that you could add any chunk not just the predefined ones. Not to me anyway. "At EUC, I talked to Bj?rn Gustavsson about officially supporting a core chunk" seemed to me to imply that. I was wrong. Now I will just have to fix it. On 24 September 2015 at 19:12, Anthony Ramine wrote: > Le 23 sept. 2015 ? 17:51, Robert Virding a ?crit : > > > Anyway that was my idea. I see I will have to wait until the Erlang > compiler is fixed to do this. And dialyzer. > > I'm not sure I understand your email, you mostly paraphrased Jos?'s. > > Regards. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ahe.sanath@REDACTED Mon Sep 28 13:42:27 2015 From: ahe.sanath@REDACTED (Sanath Prasanna) Date: Mon, 28 Sep 2015 17:12:27 +0530 Subject: [erlang-questions] Fwd: Unexpected behavior of HTTPC module - tls_connection process take more memory In-Reply-To: References: Message-ID: Hi all, Any update on this? Br, Robert On Fri, Sep 18, 2015 at 2:11 PM, Eranga Udesh wrote: > Hi, > > I am experiencing the same issue in, > > - Erlang version : V6.4 , (Erlang/OTP 17 [erts-6.4] [source] [64-bit]) > - OS version/architecture (32/64 bit) : Red Hat Enterprise Linux > Server release 6.6 (64 bit) > > The issue doesn't come always but after running the system for 3-10 days > can experience the tls_connection instances making high reductions and > consuming high memory as sent by Sanath. Since it's a production system, I > wasn't able to do much experiments, but applied the patches you sent, which > didn't solve this issue. > > There are 2 servers running Erlang VMs making SSL requests to the same > HTTPS host. When this issue happens, it happens in both the VMs. So it > could be triggered by some conditions in SSL or network connection. Even > though I left the VMs to run for a while to see if they can recover, but no > success. However if I restart the VMs, it starts to run as normal. > > So the summary is, > > - It triggers by some condition in remote SSL host or network > connection > - Erlang VMs don't recover itself > - Once restarted, it start working, which implies the issue is local. > i.e. in Erlang VM (tls_connection) > > We will try to recreate it and send you more details. I wonder if anybody > else is experiencing such issue. > > Tks, > - Eranga > > > > > > > On Fri, Sep 18, 2015 at 1:36 PM, Ingela Andin > wrote: > >> Hi! >> >> As we are very busy with the release 18.1 I have not had time to try and >> recreate your problem. What version of OTP and the ssl application are you >> using? >> Can you reproduce the problem with the latest on github? >> >> Regards Ingela Erlang/OTP Team - Ericsson AB >> >> >> On Thu, Sep 17, 2015 at 8:05 PM, Sanath Prasanna >> wrote: >> >>> Hi Ingela, >>> Any update on this?? >>> "Even apply your patch, *still problem is persist.* Any >>> more suggestions to solve this unexpected behavior ? " >>> Br, >>> Robert >>> >>> On Thu, Sep 17, 2015 at 12:39 PM, Sanath Prasanna >>> wrote: >>> >>>> Hi Ingela, >>>> Even apply your patch, *still problem is persist.* Any >>>> more suggestions to solve this unexpected behavior ? >>>> Br, >>>> Robert >>>> >>>> On Tue, Sep 8, 2015 at 1:52 PM, Sanath Prasanna >>>> wrote: >>>> >>>>> Hi Ingela. >>>>> Tx a lot for your help & patch related to that.I'll inform you the >>>>> result after applying & testing patch. >>>>> Br, >>>>> Robert >>>>> >>>>> On Tue, Sep 8, 2015 at 1:31 PM, Ingela Andin >>>>> wrote: >>>>> >>>>>> Hi! >>>>>> >>>>>> It could be an bug that in the ssl application that I just fixed. The >>>>>> default session cache >>>>>> was violating the API, and this in turn made the mechanism for not >>>>>> registering a lot of equivalent >>>>>> sessions in the client fail. >>>>>> >>>>>> Here is the patch: >>>>>> >>>>>> >>>>>> diff --git a/lib/ssl/src/ssl_session.erl b/lib/ssl/src/ssl_session.erl >>>>>> index 1770faf..0d6cc93 100644 >>>>>> --- a/lib/ssl/src/ssl_session.erl >>>>>> +++ b/lib/ssl/src/ssl_session.erl >>>>>> @@ -100,14 +100,14 @@ select_session([], _, _) -> >>>>>> no_session; >>>>>> select_session(Sessions, #ssl_options{ciphers = Ciphers}, OwnCert) -> >>>>>> IsNotResumable = >>>>>> - fun([_Id, Session]) -> >>>>>> + fun(Session) -> >>>>>> not (resumable(Session#session.is_resumable) andalso >>>>>> lists:member(Session#session.cipher_suite, Ciphers) >>>>>> andalso (OwnCert == Session#session.own_certificate)) >>>>>> end, >>>>>> case lists:dropwhile(IsNotResumable, Sessions) of >>>>>> [] -> no_session; >>>>>> - [[Id, _]|_] -> Id >>>>>> + [Session | _] -> Session#session.session_id >>>>>> end. >>>>>> >>>>>> is_resumable(_, _, #ssl_options{reuse_sessions = false}, _, _, _, _) >>>>>> -> >>>>>> diff --git a/lib/ssl/src/ssl_session_cache.erl >>>>>> b/lib/ssl/src/ssl_session_cach >>>>>> e.erl >>>>>> index 11ed310..cfc48cd 100644 >>>>>> --- a/lib/ssl/src/ssl_session_cache.erl >>>>>> +++ b/lib/ssl/src/ssl_session_cache.erl >>>>>> @@ -83,7 +83,7 @@ foldl(Fun, Acc0, Cache) -> >>>>>> >>>>>> %%-------------------------------------------------------------------- >>>>>> select_session(Cache, PartialKey) -> >>>>>> ets:select(Cache, >>>>>> - [{{{PartialKey,'$1'}, '$2'},[],['$$']}]). >>>>>> + [{{{PartialKey,'_'}, '$1'},[],['$1']}]). >>>>>> >>>>>> >>>>>> %%-------------------------------------------------------------------- >>>>>> %%% Internal functions >>>>>> >>>>>> >>>>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>>>> >>>>>> >>>>>> 2015-09-07 5:52 GMT+02:00 Sanath Prasanna : >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I am running HTTP client using httpc Module to send both http and >>>>>>> https requests. normally sending arround 300 request per second without any >>>>>>> issue. however sometimes erlang node become very slow responsive. at that >>>>>>> time server load average is very high and using etop can identify >>>>>>> "tls_connection" process take more memory. when restart the erlang node its >>>>>>> become normal. as per my investigation normal time memory, processors, >>>>>>> loadAverage is not increasing. following is the HTTP request config >>>>>>> >>>>>>> >>>>>>> >>>>>>> httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync, >>>>>>> false}]) >>>>>>> >>>>>>> >>>>>>> >>>>>>> below is the etop output at that time >>>>>>> >>>>>>> >>>>>>> >>>>>>> procs 1134 processes 1504844 >>>>>>> code 9309 >>>>>>> >>>>>>> runq 0 atom 420 >>>>>>> ets 29692 >>>>>>> >>>>>>> >>>>>>> >>>>>>> Pid Name or Initial Func Time Reds Memory MsgQ >>>>>>> Current Function >>>>>>> >>>>>>> >>>>>>> ---------------------------------------------------------------------------------------- >>>>>>> >>>>>>> <5490.26428.14>tls_connection:init/ '-' 733224580768 0 >>>>>>> gen_fsm:loop/7 >>>>>>> >>>>>>> <5490.26429.14>tls_connection:init/ '-' 1328524580768 0 >>>>>>> gen_fsm:loop/7 >>>>>>> >>>>>>> <5490.26430.14>tls_connection:init/ '-' 528924580768 0 >>>>>>> gen_fsm:loop/7 >>>>>>> >>>>>>> <5490.26431.14>tls_connection:init/ '-' 1432224580768 0 >>>>>>> gen_fsm:loop/7 >>>>>>> >>>>>>> <5490.26432.14>tls_connection:init/ '-' 024580768 0 >>>>>>> gen_fsm:loop/7 >>>>>>> >>>>>>> <5490.26433.14>tls_connection:init/ '-' 024580768 0 >>>>>>> gen_fsm:loop/7 >>>>>>> >>>>>>> <5490.26434.14>tls_connection:init/ '-' 024580768 0 >>>>>>> gen_fsm:loop/7 >>>>>>> >>>>>>> <5490.26435.14>tls_connection:init/ '-' 024580768 0 >>>>>>> gen_fsm:loop/7 >>>>>>> >>>>>>> <5490.26436.14>tls_connection:init/ '-' 024580768 0 >>>>>>> gen_fsm:loop/7 >>>>>>> >>>>>>> >>>>>>> can some one help me to solve this issue? >>>>>>> >>>>>>> Br, >>>>>>> >>>>>>> A.H.E. Robert >>>>>>> >>>>>>> _______________________________________________ >>>>>>> erlang-questions mailing list >>>>>>> erlang-questions@REDACTED >>>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuncer.ayaz@REDACTED Mon Sep 28 20:12:33 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Mon, 28 Sep 2015 20:12:33 +0200 Subject: [erlang-questions] [ANNOUNCE] rebar 2.6.1 Message-ID: Fred just cut the rebar bugfix release 2.6.1. Here's the machine-generated changelog: * rebar/514: [Add license to relnotes tool per reviewer request](https://github.com/rebar/rebar/pull/514) * rebar/518: [Generate reserved file for erlc regression test](https://github.com/rebar/rebar/pull/518) * rebar/520: [Fixed version of #451](https://github.com/rebar/rebar/pull/520) * rebar/521: [rebar_utils: fix comment](https://github.com/rebar/rebar/pull/521) * rebar/522: [Fixed version of #395](https://github.com/rebar/rebar/pull/522) * rebar/524: [completion: add libid= to bash and zsh scripts](https://github.com/rebar/rebar/pull/524) * rebar/530: [Allow behaviors defined inside xref_extra_path](https://github.com/rebar/rebar/pull/530) * rebar/535: [Add regression test for covered common test](https://github.com/rebar/rebar/pull/535) * rebar/537: [Fix eunit test on OTP18](https://github.com/rebar/rebar/pull/537) * rebar/538: [Fix #536](https://github.com/rebar/rebar/pull/538) * rebar/539: [Add erl_first_files in eunit_first_files and qc_first_files](https://github.com/rebar/rebar/pull/539) * rebar/540: [Complete port compiler help string](https://github.com/rebar/rebar/pull/540) * rebar/543: [rmemo: use a better R13 check (Reported-by: Stavros Aronis)](https://github.com/rebar/rebar/pull/543) * rebar/545: [Fix #544](https://github.com/rebar/rebar/pull/545) * rebar/546: [Added report and verbose options for LFE compile.](https://github.com/rebar/rebar/pull/546) From sergej.jurecko@REDACTED Mon Sep 28 21:54:32 2015 From: sergej.jurecko@REDACTED (Sergej =?UTF-8?B?SnVyZcSNa28=?=) Date: Mon, 28 Sep 2015 21:54:32 +0200 Subject: [erlang-questions] maps iterator Message-ID: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> Are maps ever going to get an iterator? Sergej -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Mon Sep 28 22:34:29 2015 From: vasdeveloper@REDACTED (Theepan) Date: Tue, 29 Sep 2015 02:04:29 +0530 Subject: [erlang-questions] maps iterator In-Reply-To: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> References: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> Message-ID: What type of an iterator are you expecting? Can't you solve your problem with the existing iterators (fold, map) or facilities to create one (to_list, keys)? Theepan On Tue, Sep 29, 2015 at 1:24 AM, Sergej Jure?ko wrote: > Are maps ever going to get an iterator? > > Sergej > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Mon Sep 28 22:54:56 2015 From: mjtruog@REDACTED (Michael Truog) Date: Mon, 28 Sep 2015 13:54:56 -0700 Subject: [erlang-questions] maps iterator In-Reply-To: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> References: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> Message-ID: <5609A920.7050908@gmail.com> On 09/28/2015 12:54 PM, Sergej Jure?ko wrote: > Are maps ever going to get an iterator? You could use something like this: maps_itera(F, A, Map) when is_function(F, 4) -> Fold = fun(K, V, {Afold, Itr}) -> ItrA = fun(Anew) -> {Anew, Itr} end, case Itr(K, V, Afold, ItrA) of {_, Itr} = Iteration -> Iteration; _ = Last -> exit({maps_itera_last, Last}) end end, try maps:fold(Fold, {A, F}, Map) of {Last, F} -> Last catch exit:{maps_itera_last, Last} -> Last end. Then you define a function: F = fun(K, V, A, Itr) -> % if you want to iterate, call Itr(Anew) % if you want to stop, just have the last evaluation be Anew end > > Sergej > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From wallentin.dahlberg@REDACTED Mon Sep 28 23:45:13 2015 From: wallentin.dahlberg@REDACTED (=?UTF-8?Q?Bj=C3=B6rn=2DEgil_Dahlberg?=) Date: Mon, 28 Sep 2015 23:45:13 +0200 Subject: [erlang-questions] maps iterator In-Reply-To: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> References: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> Message-ID: Yes. It's on the todo list. I prototyped something in May but iterating over the list of maps:to_list/1 was faster than using the iterator (though less memory intensive) so I scrapped it. I didn't spend an over abundance of time on it though. I will revisit it when there's room in the backlog. // Bj?rn-Egil 2015-09-28 21:54 GMT+02:00 Sergej Jure?ko : > Are maps ever going to get an iterator? > > Sergej > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Tue Sep 29 00:43:05 2015 From: zxq9@REDACTED (zxq9) Date: Tue, 29 Sep 2015 07:43:05 +0900 Subject: [erlang-questions] maps iterator In-Reply-To: References: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> Message-ID: <1649152.uzgB99HqlL@changa> On 2015?9?28? ??? 23:45:13 Bj?rn-Egil Dahlberg wrote: > Yes. It's on the todo list. > > I prototyped something in May but iterating over the list of maps:to_list/1 > was faster than using the iterator (though less memory intensive) so I > scrapped it. I didn't spend an over abundance of time on it though. I will > revisit it when there's room in the backlog. What would be the use of having an iterator in the face of existing list operations (maps:filter/2, maps:fold/3, maps:map/2), and fast list conversions such as maps:keys/1, maps:values/1, maps:to_list/1, maps:from_list/1? >From what I can tell "iterators" for maps already go well beyond what I had expected maps to have pre-built for them, considering their intended use. If we're missing something its maps:partition/2 to complement maps:with/2 and maps:without/2. -Craig From mrtndimitrov@REDACTED Tue Sep 29 10:06:48 2015 From: mrtndimitrov@REDACTED (Martin Koroudjiev) Date: Tue, 29 Sep 2015 11:06:48 +0300 Subject: [erlang-questions] variable exported from case in OTP 18 Message-ID: <560A4698.4070609@gmail.com> Hello, I am confused why this code generates warning with the warn_export_vars option: -module(test). -export([test/1]). test(Mode) -> case Mode of r -> {ok, FP} = file:open("warn.erl", [read]); w -> {ok, FP} = file:open("warn.erl", [write]) end, file:close(FP). compile:file("test.erl", [report, warn_export_vars]). test.erl:10: Warning: variable 'FP' exported from 'case' (line 6) {ok,test} Why this is better than this code: test(Mode) -> {ok, FP} = case Mode of r -> file:open("warn.erl", [read]); w -> file:open("warn.erl", [write]) end, file:close(FP). which produces no warnings? Thanks in advance. Regards, Martin From sergej.jurecko@REDACTED Tue Sep 29 10:12:55 2015 From: sergej.jurecko@REDACTED (=?UTF-8?Q?Sergej_Jure=C4=8Dko?=) Date: Tue, 29 Sep 2015 10:12:55 +0200 Subject: [erlang-questions] maps iterator In-Reply-To: References: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> Message-ID: Something like gb_trees:iterator_from If you're traversing the entire map then to_list is fine (which fold,map use). But if you wish to only traverse a subset of k/v pairs in a large map it is quite wasteful. Sergej On Mon, Sep 28, 2015 at 10:34 PM, Theepan wrote: > What type of an iterator are you expecting? Can't you solve your problem > with the existing iterators (fold, map) or facilities to create one > (to_list, keys)? > > Theepan > > On Tue, Sep 29, 2015 at 1:24 AM, Sergej Jure?ko > wrote: > >> Are maps ever going to get an iterator? >> >> Sergej >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Tue Sep 29 10:32:48 2015 From: zxq9@REDACTED (zxq9) Date: Tue, 29 Sep 2015 17:32:48 +0900 Subject: [erlang-questions] maps iterator In-Reply-To: References: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> Message-ID: <5158132.fz0KSQOc4W@burrito> On Tuesday 29 September 2015 10:12:55 Sergej Jure?ko wrote: > Something like gb_trees:iterator_from > If you're traversing the entire map then to_list is fine (which fold,map > use). But if you wish to only traverse a subset of k/v pairs in a large map > it is quite wasteful. Map keys are unordered. GB tree keys are ordered. If you are using maps then something like iterator_from already does not fit. But maybe you have some subset of keys you want to traverse? That is what maps:with/2 and maps:without/2 are for. (I have no idea how efficient with/2 or without/2 are in gigantic maps in R18 -- but I imagine these are the functions that will be optimized to death eventually if anything is, so it is what I would write code against today. Unless you actually do just need gb_trees...) In some cases having maps:partition/2 would be nice, but there are no cases I can think of where an equivalent to iterator_from makes sense without either requiring a filter over the entire map's values to figure out what subset you want, or already having a sublist of keys you want to include or exclude from your traversal. What am I missing? -Craig From sergej.jurecko@REDACTED Tue Sep 29 10:36:00 2015 From: sergej.jurecko@REDACTED (=?UTF-8?Q?Sergej_Jure=C4=8Dko?=) Date: Tue, 29 Sep 2015 10:36:00 +0200 Subject: [erlang-questions] maps iterator In-Reply-To: <5158132.fz0KSQOc4W@burrito> References: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> <5158132.fz0KSQOc4W@burrito> Message-ID: Oh ok. I was under the impression maps have sorted keys. Sergej On Tue, Sep 29, 2015 at 10:32 AM, zxq9 wrote: > On Tuesday 29 September 2015 10:12:55 Sergej Jure?ko wrote: > > Something like gb_trees:iterator_from > > If you're traversing the entire map then to_list is fine (which fold,map > > use). But if you wish to only traverse a subset of k/v pairs in a large > map > > it is quite wasteful. > > Map keys are unordered. GB tree keys are ordered. > > If you are using maps then something like iterator_from already does not > fit. But maybe you have some subset of keys you want to traverse? That is > what maps:with/2 and maps:without/2 are for. (I have no idea how efficient > with/2 or without/2 are in gigantic maps in R18 -- but I imagine these are > the functions that will be optimized to death eventually if anything is, so > it is what I would write code against today. Unless you actually do just > need gb_trees...) > > In some cases having maps:partition/2 would be nice, but there are no > cases I can think of where an equivalent to iterator_from makes sense > without either requiring a filter over the entire map's values to figure > out what subset you want, or already having a sublist of keys you want to > include or exclude from your traversal. > > What am I missing? > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Tue Sep 29 12:07:30 2015 From: vasdeveloper@REDACTED (Theepan) Date: Tue, 29 Sep 2015 15:37:30 +0530 Subject: [erlang-questions] variable exported from case in OTP 18 In-Reply-To: <560A4698.4070609@gmail.com> References: <560A4698.4070609@gmail.com> Message-ID: The difference is implicit vs explicit. When it is explicit compiler assumes you are aware of what you are doing and you control the return value of each case clause. Theepan On Tue, Sep 29, 2015 at 1:36 PM, Martin Koroudjiev wrote: > Hello, > > I am confused why this code generates warning with the warn_export_vars > option: > > -module(test). > > -export([test/1]). > > test(Mode) -> > case Mode of > r -> {ok, FP} = file:open("warn.erl", [read]); > w -> {ok, FP} = file:open("warn.erl", [write]) > end, > file:close(FP). > > compile:file("test.erl", [report, warn_export_vars]). > test.erl:10: Warning: variable 'FP' exported from 'case' (line 6) > {ok,test} > > Why this is better than this code: > > test(Mode) -> > {ok, FP} = > case Mode of > r -> file:open("warn.erl", [read]); > w -> file:open("warn.erl", [write]) > end, > file:close(FP). > > which produces no warnings? > > Thanks in advance. > > Regards, > Martin > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Sep 29 13:13:04 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 29 Sep 2015 13:13:04 +0200 Subject: [erlang-questions] Feedback on webmachine route In-Reply-To: <56080CDD.1050102@krotkine.com> References: <1443197026.1700912.393584041.707866DD@webmail.messagingengine.com> <56080CDD.1050102@krotkine.com> Message-ID: On Sun, Sep 27, 2015 at 5:35 PM, Damien Krotkine wrote: > wow thanks ! I was wondering why I had to write a binary_join myself and > it wasn't provided in some standard library. That's because I don't need to > write it :) Perhaps there is an even easier solution, where you just provide an iolist() to webmachine instead. since a list of binaries is already an iolist(), we can in some cases just pass that around and the network socket will understand how to handle that. Alternatively, call iolist_to_binary(Parts). -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Tue Sep 29 13:42:19 2015 From: zxq9@REDACTED (zxq9) Date: Tue, 29 Sep 2015 20:42:19 +0900 Subject: [erlang-questions] variable exported from case in OTP 18 In-Reply-To: <560A4698.4070609@gmail.com> References: <560A4698.4070609@gmail.com> Message-ID: <4558184.1nhmif8cUn@changa> On 2015?9?29? ??? 11:06:48 Martin Koroudjiev wrote: > Hello, > > I am confused why this code generates warning with the warn_export_vars > option: > > -module(test). > > -export([test/1]). > > test(Mode) -> > case Mode of > r -> {ok, FP} = file:open("warn.erl", [read]); > w -> {ok, FP} = file:open("warn.erl", [write]) > end, > file:close(FP). > > compile:file("test.erl", [report, warn_export_vars]). > test.erl:10: Warning: variable 'FP' exported from 'case' (line 6) > {ok,test} > > Why this is better than this code: > > test(Mode) -> > {ok, FP} = > case Mode of > r -> file:open("warn.erl", [read]); > w -> file:open("warn.erl", [write]) > end, > file:close(FP). > > which produces no warnings? `case`, `if`, comprehensions, etc. are assumed to introduce a new scope of their own (whether they *actually* do or not). From this view you are accessing a variable defined within a different conceptual scope, even if it actually is accessible in this particular case. The reason this has been made into a warning (and many tools compile with warnings-as-errors set) is that it is possible to not assign a variable you access outside the case in every branch of it and still get a clean compile: foo() -> case bar() of {bing, Spam} -> eat(Spam); {bong, Eggs} -> eat(Eggs) end, puke(Spam). VS foo() -> Food = case bar() of {bing, Spam} -> Spam; {bong, Eggs} -> Eggs end, ok = eat(Food), puke(Food). Hey, look, one is more likely to be self-documenting now... This has been discussed several times (here, for example: http://erlang.org/pipermail/erlang-questions/2014-March/078017.html), and because of the weirdness of case scope being one way (its a scope semantically and conceptually, but its not really) and list comprehensions being another (it really is a separate scope, but that's not how some other languages work) this has been made into a warning. I remember the warning/error discussion happening, but can't find the notes for it (I had thought the "this is always a warning" thing came up with R16 or 17...?). :-( Anyway, what I remember of it was the danger of not assiging a variable in every branch, that being a silly thing to even worry with since cases are sorta-kinda their own scope, and case statements have a return value anyway that should be used as per the way everything else in Erlang works. -Craig From zxq9@REDACTED Tue Sep 29 13:58:41 2015 From: zxq9@REDACTED (zxq9) Date: Tue, 29 Sep 2015 20:58:41 +0900 Subject: [erlang-questions] variable exported from case in OTP 18 In-Reply-To: <4558184.1nhmif8cUn@changa> References: <560A4698.4070609@gmail.com> <4558184.1nhmif8cUn@changa> Message-ID: <17396374.bC6hWqK8dy@changa> On 2015?9?29? ??? 20:42:19 zxq9 wrote: > On 2015?9?29? ??? 11:06:48 Martin Koroudjiev wrote: > > Hello, > > > > I am confused why this code generates warning with the warn_export_vars > > option: > > This has been discussed several times (here, for example: http://erlang.org/pipermail/erlang-questions/2014-March/078017.html), and because of the weirdness of case scope being one way (its a scope semantically and conceptually, but its not really) and list comprehensions being another (it really is a separate scope, but that's not how some other languages work) this has been made into a warning. Martin, By the way -- something I should have mentioned is that this is almost never a problem in practice because the normal way to deal with lots of declarations within a `case` (usually because you have a large chain of case statements within a single function, so lots of bindings lay around in scope or collide) is to break most of this stuff out into separate functions. For example, your original code is something I don't think anyone would write: > test(Mode) -> > case Mode of > r -> {ok, FP} = file:open("warn.erl", [read]); > w -> {ok, FP} = file:open("warn.erl", [write]) > end, > file:close(FP). Would be more like: test(Mode) -> {ok, FD} = file:open("warn.erl", [mode(Mode)]), ok = do_stuff(FD), file:close(FD). mode(r) -> read; mode(w) -> write. This bothered me initially because it is hard to imagine this being more natural any other way -- and its pretty weird to need 'r' or 'w' instead of 'read' or 'write' to begin with. But contrived examples are difficult to compare to real situations sometimes. -Craig From daniel.goertzen@REDACTED Tue Sep 29 16:55:02 2015 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Tue, 29 Sep 2015 14:55:02 +0000 Subject: [erlang-questions] NIF Callback Pids Message-ID: I love the Erlang NIF API, but one thing I?ve missed is a nice way to send messages into NIF code. Native Processes could provide that, but for a variety of reasons it doesn?t look like that will be implemented any time soon. As a stopgap I would like to pitch the idea of ?NIF Callback Pids?. The idea is that there would be a new NIF API that would create a new PID and bind it to a user NIF function. When anything is sent to that pid, the user NIF function is executed with the message term as a parameter. It would look something like... ErlNifPid enif_make_callback_pid(ErlNifEnv *, ErlNifSendCallback *cb, void *data); ... and then the callback would look like ... void my_callback(ErlNifEnv *env, ErlNifCBEvent event, ERL_NIF_TERM msg, void *data) { if(event == Message) { ... handle message, parse and post to thread, etc } else if(event == Destroy) { ... free private data } } This would give NIF modules the power to emulate processes in a very flexible way. A typical application would parse the message term and post a C struct to a thread. A single thread could possibly represent many callback pids. Or for certain applications, a thread may not be needed at all. So, I?ve been reading VM code to see how this might be implemented, but before I get too far I should ask? 1. Has anyone explored this already? 2. Is the idea fundamentally broken is some way that would prevent it from ever working? 3. Does anyone other than me think this would be useful? Thanks, Dan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasdeveloper@REDACTED Tue Sep 29 19:36:47 2015 From: vasdeveloper@REDACTED (Theepan) Date: Tue, 29 Sep 2015 23:06:47 +0530 Subject: [erlang-questions] NIF Callback Pids In-Reply-To: References: Message-ID: NIF provides for implementing 'functions with header in Erlang, and body in C'. This improves performance significantly against Port Drivers. I am not sure why you wanted to send messages to a NIF. NIFs are not processes or ports, that you can send a message to. Only function arguments are passed from Erlang to C. Also, NIFs are most suitable for synchronous returns. Theepan On Tue, Sep 29, 2015 at 8:25 PM, Daniel Goertzen wrote: > I love the Erlang NIF API, but one thing I?ve missed is a nice way to send > messages into NIF code. Native Processes could provide that, but for a > variety of reasons it doesn?t look like that will be implemented any time > soon. As a stopgap I would like to pitch the idea of ?NIF Callback Pids?. > The idea is that there would be a new NIF API that would create a new PID > and bind it to a user NIF function. When anything is sent to that pid, the > user NIF function is executed with the message term as a parameter. It > would look something like... ErlNifPid enif_make_callback_pid(ErlNifEnv *, > ErlNifSendCallback *cb, void *data); ... and then the callback would look > like ... void my_callback(ErlNifEnv *env, ErlNifCBEvent event, ERL_NIF_TERM > msg, void *data) { if(event == Message) { ... handle message, parse and > post to thread, etc } else if(event == Destroy) { ... free private data } } > This would give NIF modules the power to emulate processes in a very > flexible way. A typical application would parse the message term and post a > C struct to a thread. A single thread could possibly represent many > callback pids. Or for certain applications, a thread may not be needed at > all. So, I?ve been reading VM code to see how this might be implemented, > but before I get too far I should ask? 1. Has anyone explored this already? > 2. Is the idea fundamentally broken is some way that would prevent it from > ever working? 3. Does anyone other than me think this would be useful? > > > Thanks, > > Dan. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Tue Sep 29 20:49:29 2015 From: roger@REDACTED (Roger Lipscombe) Date: Tue, 29 Sep 2015 19:49:29 +0100 Subject: [erlang-questions] NIF Callback Pids In-Reply-To: References: Message-ID: On 29 September 2015 at 15:55, Daniel Goertzen wrote: > I love the Erlang NIF API, but one thing I?ve missed is a nice way to send > messages into NIF code. You can do this by simply having an Erlang process that calls the NIF when it receives a message: init(Args) -> This = my_nif:init(Args), State = #state { this = This }, {ok, State}. handle_info(Info, #state { this = This } = State) -> my_nif:info(This, Info), {noreply, State}. ...etc. It works pretty well for us. Is there any reason you don't want to do that? From daniel.goertzen@REDACTED Tue Sep 29 21:43:31 2015 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Tue, 29 Sep 2015 19:43:31 +0000 Subject: [erlang-questions] NIF Callback Pids In-Reply-To: References: Message-ID: I recall doing that about a year ago and having a lot of trouble supporting exits from the Erlang side and exits from the thread side. Reviewing this code again today reveals that I was just confused about a few things and missed some easy fixes. :/ I guess the only thing callback pids could offer is higher messaging performance to NIF land by virtue of skipping the message copy to the proxy process and also the serialization imposed by the proxy process. A high performance logger might find this useful. On Tue, Sep 29, 2015 at 1:49 PM Roger Lipscombe wrote: > On 29 September 2015 at 15:55, Daniel Goertzen > wrote: > > I love the Erlang NIF API, but one thing I?ve missed is a nice way to > send > > messages into NIF code. > > You can do this by simply having an Erlang process that calls the NIF > when it receives a message: > > init(Args) -> > This = my_nif:init(Args), > State = #state { this = This }, > {ok, State}. > > handle_info(Info, #state { this = This } = State) -> > my_nif:info(This, Info), > {noreply, State}. > > ...etc. > > It works pretty well for us. Is there any reason you don't want to do > that? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvirding@REDACTED Wed Sep 30 01:06:22 2015 From: rvirding@REDACTED (Robert Virding) Date: Wed, 30 Sep 2015 01:06:22 +0200 Subject: [erlang-questions] maps iterator In-Reply-To: References: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> <5158132.fz0KSQOc4W@burrito> Message-ID: Seeing we are wishing here I would like a function which when you call it with a key returns the next key in the map. This so I can step through the map one key at a time. M = #{a => 1, b => 2, e => 5}. maps:first_key(M) ==> a maps:next(M, a) ==> {b,2} maps:next(M, b) ==> {e,5} The order is irrelevant. I think the next function might as well return the next key and its value seeing there is no copying and very little extra work and you are probably going to get the value anyway. Yes, I know I can get a list of all the keys but then I need to keep the list somewhere and if it is a big map (which is now feasible) then creating a big list seems a waste. Without this I can't use maps in a place where they would really fit, in luerl my Lua implementation. Robert On 29 September 2015 at 10:36, Sergej Jure?ko wrote: > Oh ok. I was under the impression maps have sorted keys. > > > Sergej > > On Tue, Sep 29, 2015 at 10:32 AM, zxq9 wrote: > >> On Tuesday 29 September 2015 10:12:55 Sergej Jure?ko wrote: >> > Something like gb_trees:iterator_from >> > If you're traversing the entire map then to_list is fine (which fold,map >> > use). But if you wish to only traverse a subset of k/v pairs in a large >> map >> > it is quite wasteful. >> >> Map keys are unordered. GB tree keys are ordered. >> >> If you are using maps then something like iterator_from already does not >> fit. But maybe you have some subset of keys you want to traverse? That is >> what maps:with/2 and maps:without/2 are for. (I have no idea how efficient >> with/2 or without/2 are in gigantic maps in R18 -- but I imagine these are >> the functions that will be optimized to death eventually if anything is, so >> it is what I would write code against today. Unless you actually do just >> need gb_trees...) >> >> In some cases having maps:partition/2 would be nice, but there are no >> cases I can think of where an equivalent to iterator_from makes sense >> without either requiring a filter over the entire map's values to figure >> out what subset you want, or already having a sublist of keys you want to >> include or exclude from your traversal. >> >> What am I missing? >> >> -Craig >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Sep 30 01:27:43 2015 From: zxq9@REDACTED (zxq9) Date: Wed, 30 Sep 2015 08:27:43 +0900 Subject: [erlang-questions] maps iterator In-Reply-To: References: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> Message-ID: <1646757.1AKmNSFFZB@changa> On 2015?9?30? ??? 01:06:22 you wrote: > Seeing we are wishing here I would like a function which when you call it > with a key returns the next key in the map. This so I can step through the > map one key at a time. > > M = #{a => 1, b => 2, e => 5}. > maps:first_key(M) ==> a > maps:next(M, a) ==> {b,2} > maps:next(M, b) ==> {e,5} > > The order is irrelevant. I think the next function might as well return the > next key and its value seeing there is no copying and very little extra > work and you are probably going to get the value anyway. Yes, I know I can > get a list of all the keys but then I need to keep the list somewhere and > if it is a big map (which is now feasible) then creating a big list seems a > waste. > > Without this I can't use maps in a place where they would really fit, in > luerl my Lua implementation. To iterate through the map you would have to loop over it. How does the map key generator track its place in the key list without maintaining a list, being like ETS, or being in a separate process? Assuming that what you mean isn't already maps:map/2 or maybe a maps:foreach/2? I'm trying to imagine what the calling code would look like where it isn't like maps:map/2, maps:fold/2 or a hypothetical maps:foreach/2, and maps don't become ETS... and drawing a blank. -Craig From vasdeveloper@REDACTED Wed Sep 30 01:38:52 2015 From: vasdeveloper@REDACTED (Theepan) Date: Wed, 30 Sep 2015 05:08:52 +0530 Subject: [erlang-questions] maps iterator In-Reply-To: References: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> <5158132.fz0KSQOc4W@burrito> Message-ID: Fair enough.. But if size is a concern, shouldn't we look at a different data structure, like gb_trees/ets? Theepan On Wed, Sep 30, 2015 at 4:36 AM, Robert Virding wrote: > Seeing we are wishing here I would like a function which when you call it > with a key returns the next key in the map. This so I can step through the > map one key at a time. > > M = #{a => 1, b => 2, e => 5}. > maps:first_key(M) ==> a > maps:next(M, a) ==> {b,2} > maps:next(M, b) ==> {e,5} > > The order is irrelevant. I think the next function might as well return > the next key and its value seeing there is no copying and very little extra > work and you are probably going to get the value anyway. Yes, I know I can > get a list of all the keys but then I need to keep the list somewhere and > if it is a big map (which is now feasible) then creating a big list seems a > waste. > > Without this I can't use maps in a place where they would really fit, in > luerl my Lua implementation. > > Robert > > > On 29 September 2015 at 10:36, Sergej Jure?ko > wrote: > >> Oh ok. I was under the impression maps have sorted keys. >> >> >> Sergej >> >> On Tue, Sep 29, 2015 at 10:32 AM, zxq9 wrote: >> >>> On Tuesday 29 September 2015 10:12:55 Sergej Jure?ko wrote: >>> > Something like gb_trees:iterator_from >>> > If you're traversing the entire map then to_list is fine (which >>> fold,map >>> > use). But if you wish to only traverse a subset of k/v pairs in a >>> large map >>> > it is quite wasteful. >>> >>> Map keys are unordered. GB tree keys are ordered. >>> >>> If you are using maps then something like iterator_from already does not >>> fit. But maybe you have some subset of keys you want to traverse? That is >>> what maps:with/2 and maps:without/2 are for. (I have no idea how efficient >>> with/2 or without/2 are in gigantic maps in R18 -- but I imagine these are >>> the functions that will be optimized to death eventually if anything is, so >>> it is what I would write code against today. Unless you actually do just >>> need gb_trees...) >>> >>> In some cases having maps:partition/2 would be nice, but there are no >>> cases I can think of where an equivalent to iterator_from makes sense >>> without either requiring a filter over the entire map's values to figure >>> out what subset you want, or already having a sublist of keys you want to >>> include or exclude from your traversal. >>> >>> What am I missing? >>> >>> -Craig >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wallentin.dahlberg@REDACTED Wed Sep 30 02:05:11 2015 From: wallentin.dahlberg@REDACTED (=?UTF-8?Q?Bj=C3=B6rn=2DEgil_Dahlberg?=) Date: Wed, 30 Sep 2015 02:05:11 +0200 Subject: [erlang-questions] maps iterator In-Reply-To: <1646757.1AKmNSFFZB@changa> References: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> <1646757.1AKmNSFFZB@changa> Message-ID: 2015-09-30 1:27 GMT+02:00 zxq9 : > To iterate through the map you would have to loop over it. How does the > map key generator track its place in the key list without maintaining a > list, being like ETS, or being in a separate process? Assuming that what > you mean isn't already maps:map/2 or maybe a maps:foreach/2? > Easy, you keep track of it using an iterator context .. similar to a binary match context. Don't worry about it. Keep in mind that all of those iterations in maps:map/2, maps:fold/3 and maps:foreach/2 are implemented using maps:to_list/1. This works but it's wasteful. Instead we reimplement those using an iterator. Less waste. 5 words less waste per pair if we implement it as an instruction otherwise 2 words. I would prefer the instruction approach. The compiler can always reassemble the return value to an 2-tuple if desired. The iterator in gb_trees is not really fast either. It's faster to iterate over gb_trees:to_list/1 in most cases but it's there and it's useful if you want to save memory. I'm trying to imagine what the calling code would look like where it isn't > like maps:map/2, maps:fold/2 or a hypothetical maps:foreach/2, and maps > don't become ETS... and drawing a blank. > I think most cases can be handled by those functions. That's why they are there. // Bj?rn-Egil -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Sep 30 03:26:55 2015 From: zxq9@REDACTED (zxq9) Date: Wed, 30 Sep 2015 10:26:55 +0900 Subject: [erlang-questions] maps iterator In-Reply-To: References: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> <1646757.1AKmNSFFZB@changa> Message-ID: <4720289.mxDPu6jLHt@burrito> This touches on something that sort of bugs me... I replied inline On Wednesday 30 September 2015 02:05:11 you wrote: > 2015-09-30 1:27 GMT+02:00 zxq9 : > > > To iterate through the map you would have to loop over it. How does the > > map key generator track its place in the key list without maintaining a > > list, being like ETS, or being in a separate process? Assuming that what > > you mean isn't already maps:map/2 or maybe a maps:foreach/2? > > > > Easy, you keep track of it using an iterator context .. similar to a binary > match context. Don't worry about it. Oh, but this sort of discussion quickly becomes an "efficiency" nitpick, so we are compelled to worry about it. OK, that's out of my system. I hate efficiency nitpicking without an actual case. I am actually just curious how "iterator context" is maintained without the code becoming either highly mysterious (as in overuse of the process dictionary) or a different semantic label for an list operation over maps that merely indicates "this version of the function foo/2 is better" (so why have foo/2?). > Keep in mind that all of those iterations in maps:map/2, maps:fold/3 and > maps:foreach/2 are implemented using maps:to_list/1. This works but it's > wasteful. Instead we reimplement those using an iterator. Less waste. 5 > words less waste per pair if we implement it as an instruction otherwise 2 > words. I would prefer the instruction approach. The compiler can always > reassemble the return value to an 2-tuple if desired. > > The iterator in gb_trees is not really fast either. It's faster to iterate > over gb_trees:to_list/1 in most cases but it's there and it's useful if you > want to save memory. That sounds more like a case for "let's improve the implementation underlying the existing iterator semantics in the API" than "we need some totally new thing represented by a new API". That said, if maps, dicts, proplists, gb_trees, and the like all had a common semantics (instead of occasional semantic overlap) making a conscious decision about structural tradeoffs would be a bit more straightforward. (Some have unique traits, and in those places the semantics don't overlap of course.) > > I'm trying to imagine what the calling code would look like where it isn't > > like maps:map/2, maps:fold/2 or a hypothetical maps:foreach/2, and maps > > don't become ETS... and drawing a blank. > > > > I think most cases can be handled by those functions. That's why they are > there. Indeed, had you not told me that they are currently implemented by iterating over maps:to_list/1 I wouldn't have noticed, because huge maps are not a problem with which I must deal very often. I tend to have lots of smallish (hundreds of elements) short-lived maps, and a few very small (tens of elements) long-lived ones -- and lots of code that still just uses dicts, sets, trees or ETS. But APIs are, at least to me, all about semantics, and we already have a fairly well defined semantics surrounding collection-type data structures, even if each data type's API tends to leave out this or that part of it. If I hit tab in the terminal after typing `maps:` or `dicts:` in the terminal and foreach/2 appears as a possibility I would fully expect it to work the same way lists:foreach/2 does -- regardless how it is implemented. Introducing something different, like maps:iterate/2 would confuse me and I would run off and read the docs. If, as so often happens, I discover that the definition was something like this: maps:iterate(Fun, Map) -> ok Works exactly like maps:foreach/2, but implemented more efficiently. (maps:foreach/2 is currently implemented as: `lists:foreach(F, maps:values(M))` and will be deprecated in the next release) I will be annoyed. This sort of thing happens all over the stdlib. If a future implementation of an existing function will be superior in every way (faster in execution *and* require less space) nothing is gained by adding a new name for the superior implementation -- other than generating stacks of future consultancy hours worth of work someday. So... that's two issues: 1- Nearly every collection data type lends itself to some similar form of iteration semantics but the APIs for the different structures, while demonstrating overlap, do not overlap in the same places nor conform to any common rule of "this type of iteration will always be included in a collection API". I think they do all have a Foo:to_list/1 type function, but if traversal is a common need (it is, ordered or not) then doing Foo:to_list/1 in front of a call to lists:TraveralFunction is confusing when often there is a shortcut in *some* APIs, but not others, that does the same thing as Foo:TraversalFunction. 2- The underlying implementations should be made faster instead of introducing new semantics just to indicate a superior implementation. Is it so crazy to expect that something like [{M, F} || M <- CollectionModules, F <- CommonCollectionOperations] would yield a list of valid {Module, Function} pairs, regardless how they are implemented? They could all be calls to M:F(O, M:to_list(C)) to start with for all I care, but I would expect the semantics not to change and the underlying implementations to improve with time. Or... is this just totally insane? -Craig From charles@REDACTED Wed Sep 30 01:59:31 2015 From: charles@REDACTED (Charles Weitzer) Date: Tue, 29 Sep 2015 23:59:31 +0000 Subject: [erlang-questions] Senior Software Engineer Needed for Machine Learning Startup Message-ID: Voleon Capital Management LP is a startup quantitative hedge fund located in Berkeley, California. We would like to hire a senior software engineer as soon as possible. Voleon's founders previously worked together at one of the most successful quantitative hedge funds in the world. Our CEO has a PhD in Computer Science from Stanford and has been CEO and founder of a successful Internet infrastructure startup. Our Chief Investment Officer has a PhD in Statistics from Berkeley. Voleon's team includes PhD's from leading departments in statistics, computer science, and mathematics. We have made several unpublished advances in the field of machine learning and in other areas as well. Here is our formal job description: ********************************************************** * Senior Software Engineer * Technology-driven investment firm employing cutting-edge statistical machine learning techniques seeks an exceptionally capable software engineer. You will architect and implement new production trading systems, machine learning infrastructure, data integration pipelines, and large-scale storage systems. The firm researches and deploys systematic trading strategies designed to generate attractive returns without being dependent on the performance of the overall market. Join a team of under 30 people that includes a Berkeley statistics professor as well as over ten PhD's from Berkeley, Chicago, CMU, Princeton, Stanford, and UCLA, led by the founder and CEO of a successful Internet infrastructure technology firm. The firm's offices are walking distance from BART and the UC Berkeley campus in downtown Berkeley, California. We have a casual and collegial office environment, weekly catered lunches, and competitive benefits packages. We seek candidates with a proven track record of writing correct, well-designed software, solving hard problems, and delivering complex projects on time. You should preferably have experience designing and implementing fault-tolerant distributed systems. Experience with building large-scale data infrastructure, stream processing systems, or latency-sensitive programs is a bonus. We are growing rapidly. Willingness to take initiative and a gritty determination to productize are essential. Required experience: - experience with functional programming languages such as Erlang, Haskell, etc. - developing with C/C++/Python/Go in a Linux environment with a focus on performance, concurrency, and correctness. - working in TCP/IP networking, multi-threading, and server development. - working with common Internet protocols (IP, TCP/UDP, SSL/TLS, HTTP, SNMP, etc.). - architecting and designing highly available systems. - architecting and designing large-scale data management infrastructure. - working in large codebases and building modular, manageable code. Preferred experience: - debugging and performance profiling, including the use of tools such as strace, valgrind, gdb, tcpdump, etc. - working with build and test automation tools. - working with well-defined change management processes. - diagnosing RDBMS performance problems, exploiting indexing, using EXPLAIN PLAN, optimizing at the code layer, etc. - working with messaging queues (RabbitMQ, Redis, etc.) as well as distributed caching systems. Interest in financial applications is essential, but experience in finance is not a primary factor in our hiring. Benefits and compensation are highly competitive. ********************************************************** The above job description is just a starting point in terms of possible duties and seniority. We can be very flexible for the right person. If you are interested, please apply with your full and complete CV: http://voleon.com/apply/ The Voleon Group is an Equal Opportunity employer. Applicants are considered without regard to race, color, religion, creed, national origin, age, sex, gender, marital status, sexual orientation and identity, genetic information, veteran status, citizenship, or any other factors prohibited by local, state, or federal law. -------------- next part -------------- An HTML attachment was scrubbed... URL: From denc716@REDACTED Wed Sep 30 05:42:58 2015 From: denc716@REDACTED (derek) Date: Tue, 29 Sep 2015 20:42:58 -0700 Subject: [erlang-questions] trying to make official docker images for erlang Message-ID: Hi to Erlang users, this is effort trying to make official docker images for erlang otp community, please comment if you like to run it with docker: https://github.com/docker-library/official-images/pull/1075 Nowadays, docker is the popular way to run many applications, have a look on docker hub, many popular programming languages and applications have an official image there, beginners can easily pull a docker image to start playing without hassling their host Linux, https://hub.docker.com/_/python/ https://hub.docker.com/_/golang/ https://hub.docker.com/_/ruby/ ... But there is no official one for erlang yet, let's try to make it happen, https://hub.docker.com/_/erlang/ I have researched some existing effort like these, looks like many ones are already on this way just haven't communicated thru @erlang.org yet - https://hub.docker.com/r/correl/erlang/ this might be the earliest currently have the most stars by `docker search erlang`, provided erlang-otp-17.5 and rebar and relx in a single image, and compiling each one from source code, I haven't tried it, but presumably would be close to 1GB image; - https://hub.docker.com/r/unbalancedparentheses/erlang this one support all versions of erlang R6B-0, R7, R8, ... R16, up to 17.4 from https://github.com/unbalancedparentheses/docker-erlang/blob/master/17.4/install.sh it also provided erlang & rebar & relx all compiled from source code - https://hub.docker.com/r/msaraiva/erlang/ this one is providing erlang-17.5 and 18, on top of Alpine Linux docker image, it's very slim, as small as 16.78 MB, while erlang lib is broken into very small packages, like most OS distro does, broken into erlang-compiler, erlang-dialyzer, erlang-otp, erlang-snmp, and etc. its base erlang:18 image has 5 packages under /usr/lib/erlang/lib/... (compared a full erlang-otp has 52 lib packages) http://dl-4.alpinelinux.org/alpine/edge/main/x86_64/ (search erlang) this one also provided elixir images in different Dockerfile, presumably also slim - https://hub.docker.com/r/voidlock/erlang is very similar to my way, support erlang R16 thru 18.1; I would not start my project if I know this earlier, but it installed update-locale for UTF-8 I'm not sure for what, is that required by erlang runtime ? - https://github.com/synctree/docker-erlang/blob/master/R17/Dockerfile this one installs erlang solutions precompiled deb on top of debian:8 (or debian:jessie) image. $ docker search erlang NAME DESCRIPTION STARS OFFICIAL AUTOMATED correl/erlang Erlang/OTP for Docker 14 [OK] ... While I just started from scratch, before above PR to official images got merged, you can try it with cloning this repo, and build images locally, https://github.com/c0b/docker-erlang-otp So there are two ways to make images: 1) build from source code, from standard debian:8 image, start with apt-get install build-essential and gcc and some lib...-dev and download erlang-otp source code and build, this usually ends up with a fat image, close to 1GB; 2) install from some binary erlang packages, like the one from erlang-solutions, could end up with smaller image; packages from most OS distributions also provided erlang but relatively not up to date like debian https://packages.debian.org/source/sid/erlang fedora centos similar Here from my repo I mainted one each for latest 4 erlang releases, (R15, R16, 17, 18), each with different variant, following the best practices from docker official images guideline, should end up with full feature while relatively slim images: https://github.com/docker-library/official-images 1. the standard variant erlang:18, erlang:17, erlang:R16, erlang:R15 builds from source code, on top of https://hub.docker.com/_/buildpack-deps/ :jessie, it covered gcc compiler and some popular -dev packages, for port driver written in C; while it doesn't have java compiler so out of the standard erlang-otp provided 52 packages under /usr/lib/erlang/lib/... from this one odbc / jinterface / wxwidgets won't work, I assume to run GUI programs in docker is not popular, so here we can save space; jinterface is similar, the java dependencies are too fat, I assume demand is low; 2. the -onbuild variant for each erlang version, to utilize ONBUILD instruction from Dockerfile, those are for starters 3. -esl variant is to pull erlang-solutions deb package to install on top of debian:jessie, results in relatively slim image, but I am trying to avoid wxwidgets / jinterface dependencies, reasons same as above. All these images are almost full featured Erlang-OTP images (except wxwidgets & jinterface), you can run it like this once build locally (or pull over docker hub if above PR can be merged), $ docker run -it --rm erlang:18.1 Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] Eshell V7.1 (abort with ^G) 1> uptime(). # the new uptime() shell command since OTP 18 3 seconds ok 2> application:which_applications(). [{stdlib,"ERTS CXC 138 10","2.6"}, {kernel,"ERTS CXC 138 10","4.1"}] 3> User switch command --> q root@REDACTED:/# ls /usr/local/lib/erlang/lib/ asn1-4.0 cosProperty-1.2 edoc-0.7.17 gs-1.6 observer-2.1 public_key-1.0.1 stdlib-2.6 xmerl-1.3.8 common_test-1.11 cosTime-1.2 eldap-1.2 hipe-3.13 orber-3.8 reltool-0.7 syntax_tools-1.7 compiler-6.0.1 cosTransactions-1.3 erl_docgen-0.4 ic-4.4 os_mon-2.4 runtime_tools-1.9.1 test_server-3.9 cosEvent-2.2 crypto-3.6.1 erl_interface-3.8 inets-6.0.1 ose-1.1 sasl-2.6 tools-2.8.1 cosEventDomain-1.2 debugger-4.1.1 erts-7.1 kernel-4.1 otp_mibs-1.1 snmp-5.2 typer-0.9.9 cosFileTransfer-1.2 dialyzer-2.8.1 et-1.5.1 megaco-3.18 parsetools-2.1 ssh-4.1 webtool-0.9 cosNotification-1.2 diameter-1.11 eunit-2.2.11 mnesia-4.13.1 percept-0.8.11 ssl-7.1 wx-1.5 root@REDACTED:/# ls /usr/local/lib/erlang/lib/ | wc -l 50 Size: $ docker images |grep ^erlang erlang 18.1-esl 138c797adec7 5 days ago 286.9 MB erlang 18.1 27ad0fc44644 5 days ago 741.5 MB erlang R16B03-1 e0deec5e1e72 6 days ago 740.2 MB erlang 18.0.3 52d4a7a4a281 6 days ago 743.7 MB Comments are welcome. Thanks, From shailen.werk@REDACTED Wed Sep 30 07:07:55 2015 From: shailen.werk@REDACTED (=?UTF-8?B?U2hhaWxlbiwg4oiC4oirIA==?=) Date: Tue, 29 Sep 2015 22:07:55 -0700 Subject: [erlang-questions] Erlang app fails to start, crash dumps on an undef Message-ID: I'm writing an erlang app (that's run well before); I use relx/erlang.mk When I attempt a _rel/bin/ console, I get this: {"Kernel pid terminated",application_controller,"{application_start_failure,lager,{bad_return,{{lager_app,start,[normal,[]]},{'EXIT',{{compile_forms,error},[{glc_code,compile_forms,2,[{file,\"src/glc_code.erl\"},{line,509}]},{glc_code,compile,2,[{file,\"src/glc_code.erl\"},{line,51}]},{glc,compile,4,[{file,\"src/glc.erl\"},{line,205}]},{lager_util,trace_filter,2,[{file,\"src/lager_util.erl\"},{line,374}]},{lager_app,start,2,[{file,\"src/lager_app.erl\"},{line,221}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,272}]}]}}}}}"} (not sure what the 'bad_return' is, as this app used to start earlier) My lager.app.src has both syntax_tools and compiler included (alongwith goldrush, stdlib and kernel) for applications. However, when I start lager from a vanilla erl prompt, lager starts after I've started all of its dependencies. Can I get some help with this? I'm not sure what I'm not seeing/missing. Thanks very much in anticipation! :-), Shailen # # _ _/' '_/_ _/ _ # (//)//())(/(/(//)/_) # _ # \/_) # _ # (_ _ _ _ _ _ _/'_ _ / _ # / / (-(-/ (/(//( (/(_) -------------- next part -------------- An HTML attachment was scrubbed... URL: From knutin@REDACTED Wed Sep 30 11:05:06 2015 From: knutin@REDACTED (Knut Nesheim) Date: Wed, 30 Sep 2015 09:05:06 +0000 Subject: [erlang-questions] Erlang tracing In-Reply-To: References: Message-ID: On Mon, 21 Sep 2015 at 10:52 Lukas Larsson wrote: > > To set the scope of the discussion, when I say tracing I include; erlang > tracing, dtrace/systemtap, trace outputs (stdout/file/IP), filtering > through match specs, sequence tracing, tool integration (dbg, fprof, > redbug, recon to mention some) and probably more. > I've spent many hours trying to build "always on" instrumentation tools, that you can even run in busy production systems. The biggest challenge in Erlang compared to other languages / virtual machines is that it's more or less impossible to follow the execution of a request due to the "use processes for everything" designs many systems adopt. seq_trace has the potential to help with this without requiring changes to code, but falls on it's face as soon as a process talks to a port driver, which happens all the time in most systems. If after calling gen_tcp:send/receive the seq_trace token is still around, it would be possible to build such an instrumentation tool. If the seq_trace token could be transferred as part of a message to another node, that would be incredibly useful. To give some context, New Relic provides agent libraries for many languages and relies on thread locals to trace typically a HTTP request as it calls different libraries, different databases, etc. This agent periodically sends aggregated data back to New Relic, which has a nice web app where you can view what is going on inside your systems. Regards Knut -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Wed Sep 30 13:10:26 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 30 Sep 2015 13:10:26 +0200 Subject: [erlang-questions] maps iterator In-Reply-To: <4720289.mxDPu6jLHt@burrito> References: <37615BC3-A1BE-4FE3-BCBF-B3837157A4F0@gmail.com> <1646757.1AKmNSFFZB@changa> <4720289.mxDPu6jLHt@burrito> Message-ID: On Wed, Sep 30, 2015 at 3:26 AM, zxq9 wrote: > I am actually just curious how "iterator context" is maintained without > the code becoming either highly mysterious (as in overuse of the process > dictionary) or a different semantic label for an list operation over maps > that merely indicates "this version of the function foo/2 is better" (so > why have foo/2?). You run a "zipper[0]" in the C layer. A map is a tree internally. A very wide tree. If the iterator references the top of the tree, it won't be garbage collected, so I have a stable snapshot of the tree. Now, if I keep track of the "path not chosen" in the tree, inside the iterator structure, I obtain a "cursor" on where I am in the tree and can thus traverse the tree as in next/1, prev/1 and friends. The only data I have to store is "where I am" as the tree data themselves are inside the snapshot. The goal here is to use this to implement a variant of map/2 which doesn't blit the whole map in memory before traversal. While fast, it has quite bad memory characteristics. [0] https://en.wikibooks.org/wiki/Haskell/Zippers -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.wobben@REDACTED Wed Sep 30 17:42:20 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Wed, 30 Sep 2015 17:42:20 +0200 Subject: [erlang-questions] ** exception error: no function clause matching test_calculate:validate(["1", "+", "1"], []) (test_calculate.erl, line 11) Message-ID: <560C02DC.4090800@home.nl> Hello, I try to make a little parser which parse calculations. So so far I have this code : -module(test_calculate). -export([validate/1]). scan(String) -> validate(string:tokens(String, " ")). validate(String) -> validate(String, []). validate([Head | Tail], Validated_list) when Head >= $0 , Head =< $9 -> validate(Tail, [Head | Validated_list]); validate([Head | Tail], Validated_list) when Head =:= $+ -> validate(Tail, [Head | Validated_list]); validate([], Validated_list) -> Validated_list. test() -> ["1","+","1"] = scan("1 + 1"), ["10", "+", "1"] = scan("10 + 1"), ["1", "+", "10"] = scan("1 + 10"), io:format("All tests are green ~n"). but as soon as I try : test_calculate:validate(["1","+", "1"]) I see the above error Someone a tip where things are going the wrong way ? Roelof From hugo@REDACTED Wed Sep 30 18:01:18 2015 From: hugo@REDACTED (Hugo Mills) Date: Wed, 30 Sep 2015 16:01:18 +0000 Subject: [erlang-questions] ** exception error: no function clause matching test_calculate:validate(["1", "+", "1"], []) (test_calculate.erl, line 11) In-Reply-To: <560C02DC.4090800@home.nl> References: <560C02DC.4090800@home.nl> Message-ID: <20150930160118.GH25907@carfax.org.uk> On Wed, Sep 30, 2015 at 05:42:20PM +0200, Roelof Wobben wrote: > Hello, > > I try to make a little parser which parse calculations. > > So so far I have this code : > > -module(test_calculate). > > -export([validate/1]). > > scan(String) -> > validate(string:tokens(String, " ")). string:tokens/2 expects its first parameter to be a string (i.e. a list of integers representing the character codes of the elements of the string). [snip] > test_calculate:validate(["1","+", "1"]) What is this parameter? Is it what string:tokens/2 is expecting? Brief exercise for you: how does erlang represent each of the following? Write each one as a term without string notation, and as a string (if it can be written as a string). [53] "5" $5 [$5] Hugo. > I see the above error (What error?) > Someone a tip where things are going the wrong way ? > > Roelof > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Hugo Mills | 2 + 2 = 5, for sufficiently large values of 2. hugo@REDACTED carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 | -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From s@REDACTED Wed Sep 30 19:02:00 2015 From: s@REDACTED (Stefan Schmiedl) Date: Wed, 30 Sep 2015 19:02:00 +0200 Subject: [erlang-questions] ** exception error: no function clause matching test_calculate:validate(["1", "+", "1"], []) (test_calculate.erl, line 11) In-Reply-To: <560C02DC.4090800@home.nl> References: <560C02DC.4090800@home.nl> Message-ID: <20150930170200.GC31023@g128.fritz.box> Roelof Wobben (30.09. 17:42): > Hello, > > I try to make a little parser which parse calculations. > > So so far I have this code : > > -module(test_calculate). > > -export([validate/1]). > > scan(String) -> > validate(string:tokens(String, " ")). > > validate(String) -> > validate(String, []). > > validate([Head | Tail], Validated_list) when Head >= $0 , Head =< $9 -> > validate(Tail, [Head | Validated_list]); > > validate([Head | Tail], Validated_list) when Head =:= $+ -> > validate(Tail, [Head | Validated_list]); > > validate([], Validated_list) -> > Validated_list. > > test() -> > ["1","+","1"] = scan("1 + 1"), > ["10", "+", "1"] = scan("10 + 1"), > ["1", "+", "10"] = scan("1 + 10"), > io:format("All tests are green ~n"). > > but as soon as I try : > > test_calculate:validate(["1","+", "1"]) line 11 points probably to the first "validate" implementation. So "validate" is missing a clause matching the input. What is the input? It's a non-empty list, so the first two clauses are still possible. Both are conditions for head. What is the head of the list? The string "1". First clause compares this string, i.e. list of integers, against a single int, fail. Second clause compares this list of integers against a single int, fail. So you still need to parse the tokenized strings into "real" values. s. > > I see the above error > > Someone a tip where things are going the wrong way ? > > Roelof > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ruan.jonker@REDACTED Wed Sep 30 18:19:00 2015 From: ruan.jonker@REDACTED (Ruan Jonker) Date: Wed, 30 Sep 2015 18:19:00 +0200 Subject: [erlang-questions] ** exception error: no function clause matching test_calculate:validate(["1", "+", "1"], []) (test_calculate.erl, line 11) In-Reply-To: <560C02DC.4090800@home.nl> References: <560C02DC.4090800@home.nl> Message-ID: Hi, Your validate function seems to expect a list of chars, but it is in fact getting a list of lists (result of string:tokens). So what's happening is that in the first clause you are comparing a list to an integer, and in the second you are asserting the type =:= as well (which will not match). Rather have a look @ Sean Cribbs' neotoma (github) than trying to roll your own. R On 30 Sep 2015 5:50 pm, "Roelof Wobben" wrote: > Hello, > > I try to make a little parser which parse calculations. > > So so far I have this code : > > -module(test_calculate). > > -export([validate/1]). > > scan(String) -> > validate(string:tokens(String, " ")). > > validate(String) -> > validate(String, []). > > validate([Head | Tail], Validated_list) when Head >= $0 , Head =< $9 -> > validate(Tail, [Head | Validated_list]); > > validate([Head | Tail], Validated_list) when Head =:= $+ -> > validate(Tail, [Head | Validated_list]); > > validate([], Validated_list) -> > Validated_list. > > test() -> > ["1","+","1"] = scan("1 + 1"), > ["10", "+", "1"] = scan("10 + 1"), > ["1", "+", "10"] = scan("1 + 10"), > io:format("All tests are green ~n"). > > but as soon as I try : > > test_calculate:validate(["1","+", "1"]) > > I see the above error > > Someone a tip where things are going the wrong way ? > > Roelof > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.wobben@REDACTED Wed Sep 30 21:33:13 2015 From: r.wobben@REDACTED (Roelof Wobben) Date: Wed, 30 Sep 2015 21:33:13 +0200 Subject: [erlang-questions] ** exception error: no function clause matching test_calculate:validate(["1", "+", "1"], []) (test_calculate.erl, line 11) In-Reply-To: <20150930160118.GH25907@carfax.org.uk> References: <560C02DC.4090800@home.nl> <20150930160118.GH25907@carfax.org.uk> Message-ID: <560C38F9.5@home.nl> Op 30-9-2015 om 18:01 schreef Hugo Mills: > On Wed, Sep 30, 2015 at 05:42:20PM +0200, Roelof Wobben wrote: >> Hello, >> >> I try to make a little parser which parse calculations. >> >> So so far I have this code : >> >> -module(test_calculate). >> >> -export([validate/1]). >> >> scan(String) -> >> validate(string:tokens(String, " ")). > string:tokens/2 expects its first parameter to be a string (i.e. a > list of integers representing the character codes of the elements of > the string). > > [snip] Correct, that is why string:tokens is called with as parameter "1 + 1 ". >> test_calculate:validate(["1","+", "1"]) > What is this parameter? Is it what string:tokens/2 is expecting? This is the outcome of the string:token with as argument "1 + 1 ". > Brief exercise for you: how does erlang represent each of the > following? Write each one as a term without string notation, and as a > string (if it can be written as a string). > > [53] > "5" > $5 > [$5] > > Hugo. [53] "5" "5" "5" $5 "53" [$5] "5" From denc716@REDACTED Wed Sep 30 23:47:39 2015 From: denc716@REDACTED (derek) Date: Wed, 30 Sep 2015 14:47:39 -0700 Subject: [erlang-questions] trying to make official docker images for erlang In-Reply-To: References: Message-ID: Please click Reply to all to make comments available thru @erlang.org ML, http://erlang.org/pipermail/erlang-questions/2015-September/thread.html#86239 And this PR is requesting comments from language upstream, so please also comment there if you could; thanks https://github.com/docker-library/official-images/pull/1075 On Wed, Sep 30, 2015 at 4:28 AM, Jesper Louis Andersen wrote: > > On Wed, Sep 30, 2015 at 5:42 AM, derek wrote: >> >> this is effort trying to make official docker images for erlang otp >> community, please comment if you like to run it with docker: > > > Nice idea, and really needed. I'm not too familiar with docker, so I can't > help you out in the details of things, but it seems like a worthwhile thing > to have! > > > -- > J.