From rj@REDACTED Thu Dec 1 12:37:37 2016 From: rj@REDACTED (Richard Jones) Date: Thu, 1 Dec 2016 11:37:37 +0000 Subject: [erlang-questions] Unimplemented {buffer, integer()} option in ssl module - bug? Message-ID: The {buffer, integer()} option from gen_tcp is useful for breaking up incoming data into manageable sizes, especially when in {packet, line} mode. The docs suggest (kinda) that the gen_tcp options should also work when connecting via the ssl module, but this isn't always the case - the buffer option is silently ignored by ssl:connect. Is this a bug? Tested on erl 17, 18, 19 Examples follow using netcat/socat as a server. Note that gen_tcp breaks the incoming data into 3 messages, ssl delivers as one large message. Non-SSL example: ---------------- $ echo "12345678901234567890123" | nc -l 1234 1> {ok, _} = gen_tcp:connect("localhost", 1234, [ binary, {packet, line}, {buffer, 10}, {active, true}]). {ok,#Port<0.1180>} 2> flush(). Shell got {tcp,#Port<0.1180>,<<"1234567890">>} Shell got {tcp,#Port<0.1180>,<<"1234567890">>} Shell got {tcp,#Port<0.1180>,<<"123\n">>} Shell got {tcp_closed,#Port<0.1180>} ok SSL example: ------------ $ echo "12345678901234567890123" | socat STDIO OPENSSL-LISTEN:1234,reuseaddr,cert=/some.pem,cafile=/some.crt,verify=0 1> {ok, _} = ssl:connect("localhost", 1234, [ binary, {packet, line}, {buffer, 10}, {active, true}, {verify, verify_none}]). {ok,{sslsocket,{gen_tcp,#Port<0.1179>,tls_connection, undefined}, <0.54.0>}} 2> flush(). Shell got {ssl,{sslsocket,{gen_tcp,#Port<0.1179>,tls_connection,undefined}, <0.54.0>}, <<"12345678901234567890123\n">>} Shell got {ssl_closed, {sslsocket, {gen_tcp,#Port<0.1179>,tls_connection,undefined}, <0.54.0>}} ok From khalfella@REDACTED Thu Dec 1 12:31:41 2016 From: khalfella@REDACTED (Mohamed Khalfella) Date: Thu, 1 Dec 2016 06:31:41 -0500 Subject: [erlang-questions] How to enhance runtime and reduce memory usage Message-ID: Hi, I am new to Erlang. I was solving one of the exercises in hackerrank.com and I faced an issue where my solution is terminated because it takes long to process the test case. Attached to this email the problem definition. You can access my solution as below. My questions are: How to pin point code that excessively eats memory? How to pin point code that consumes CPU time? http://pastebin.com/NaeHihYx -- BR, Mohamed A. Khalfella -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: The Tree Of Life _ Functional Programming Question _ HackerRank.pdf Type: application/pdf Size: 370701 bytes Desc: not available URL: From bengt.kleberg@REDACTED Thu Dec 1 13:16:45 2016 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Thu, 1 Dec 2016 13:16:45 +0100 Subject: [erlang-questions] How to enhance runtime and reduce memory usage In-Reply-To: References: Message-ID: Greetings, For finding CPU time consumption you can start here: http://erlang.org/doc/efficiency_guide/profiling.html bengt On 12/01/2016 12:31 PM, Mohamed Khalfella wrote: > Hi, > > I am new to Erlang. I was solving one of the exercises in > hackerrank.com and I faced an issue where my > solution is terminated because it takes long to process the test case. > Attached to this email the problem definition. You can access my > solution as below. My questions are: > > How to pin point code that excessively eats memory? > How to pin point code that consumes CPU time? > > http://pastebin.com/NaeHihYx > > -- > BR, > Mohamed A. Khalfella > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From khalfella@REDACTED Thu Dec 1 13:40:46 2016 From: khalfella@REDACTED (Mohamed Khalfella) Date: Thu, 1 Dec 2016 07:40:46 -0500 Subject: [erlang-questions] How to enhance runtime and reduce memory usage In-Reply-To: References: Message-ID: Hi Bengt, Thanks for the link. I am reading that page now. I fixed the code. It passes the test now. However, I just guessed it. Next time I have to be lucky to guess it again, or better find a systematic way to understand CPU and memory consumption of erlang programs. Below is a link for the corrected code. http://pastebin.com/2h2iF32A On Thu, Dec 1, 2016 at 7:16 AM, Bengt Kleberg wrote: > Greetings, > > For finding CPU time consumption you can start here: > http://erlang.org/doc/efficiency_guide/profiling.html > > > bengt > > > On 12/01/2016 12:31 PM, Mohamed Khalfella wrote: > > Hi, > > I am new to Erlang. I was solving one of the exercises in hackerrank.com > and I faced an issue where my solution is terminated because it takes long > to process the test case. Attached to this email the problem definition. > You can access my solution as below. My questions are: > > How to pin point code that excessively eats memory? > How to pin point code that consumes CPU time? > > http://pastebin.com/NaeHihYx > > -- > BR, > Mohamed A. Khalfella > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- BR, Mohamed A. Khalfella -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Thu Dec 1 18:11:21 2016 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 1 Dec 2016 18:11:21 +0100 Subject: [erlang-questions] wildcarded resource counters in Gproc Message-ID: There's a new PR on Gproc, adding support for limited wildcard matching in resource counters. https://github.com/uwiger/gproc/pull/128 I don't know how many people use resource counters ('rc'), but thought I'd ask. The operation of registering (or unregistering) a resource now makes one extra update_counter() protected by a try ... catch. My assumption is that this extra overhead will be insignificant to just about all use cases, but ... does anyone out there want to disagree? Briefly, when registering a resource ('r') object, gproc would try to perform an update_counter on the corresponding 'rc' object - an operation that might fail if the 'rc' object doesn't exist. Now, if the name of the resource is a tuple, it also tries to update an 'rc' object with the last element of the name tuple replaced with '\\_'. This make it possible to count resources whose names differ in only the last tuple element. For non-tuple resource names, it works just like before. BR, Ulf W -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Thu Dec 1 19:11:25 2016 From: g@REDACTED (Garrett Smith) Date: Thu, 1 Dec 2016 12:11:25 -0600 Subject: [erlang-questions] wildcarded resource counters in Gproc In-Reply-To: References: Message-ID: This doesn't feel right to me - it strikes me as edge case support that's impacting what's otherwise a very simple feature. If this is generally useful, would it make sense to implement it as a new function call, which made the additional update explicit? On Thu, Dec 1, 2016 at 11:11 AM, Ulf Wiger wrote: > There's a new PR on Gproc, adding support for limited wildcard matching in > resource counters. > > https://github.com/uwiger/gproc/pull/128 > > I don't know how many people use resource counters ('rc'), but thought I'd > ask. > > The operation of registering (or unregistering) a resource now makes one > extra update_counter() protected by a try ... catch. My assumption is that > this extra overhead will be insignificant to just about all use cases, but > ... does anyone out there want to disagree? > > Briefly, when registering a resource ('r') object, gproc would try to > perform an update_counter on the corresponding 'rc' object - an operation > that might fail if the 'rc' object doesn't exist. Now, if the name of the > resource is a tuple, it also tries to update an 'rc' object with the last > element of the name tuple replaced with '\\_'. This make it possible to > count resources whose names differ in only the last tuple element. For > non-tuple resource names, it works just like before. > > BR, > Ulf W > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From g@REDACTED Thu Dec 1 19:18:28 2016 From: g@REDACTED (Garrett Smith) Date: Thu, 1 Dec 2016 12:18:28 -0600 Subject: [erlang-questions] wildcarded resource counters in Gproc In-Reply-To: References: Message-ID: And another tack - is there a path here to a generalized interface for alternative schemes? E.g. passing along a function to do some work on the table? Maybe not for the faint of heart, but this would reduce the temptation to modify the interface for these case. On Thu, Dec 1, 2016 at 12:11 PM, Garrett Smith wrote: > This doesn't feel right to me - it strikes me as edge case support > that's impacting what's otherwise a very simple feature. > > If this is generally useful, would it make sense to implement it as a > new function call, which made the additional update explicit? > > On Thu, Dec 1, 2016 at 11:11 AM, Ulf Wiger wrote: >> There's a new PR on Gproc, adding support for limited wildcard matching in >> resource counters. >> >> https://github.com/uwiger/gproc/pull/128 >> >> I don't know how many people use resource counters ('rc'), but thought I'd >> ask. >> >> The operation of registering (or unregistering) a resource now makes one >> extra update_counter() protected by a try ... catch. My assumption is that >> this extra overhead will be insignificant to just about all use cases, but >> ... does anyone out there want to disagree? >> >> Briefly, when registering a resource ('r') object, gproc would try to >> perform an update_counter on the corresponding 'rc' object - an operation >> that might fail if the 'rc' object doesn't exist. Now, if the name of the >> resource is a tuple, it also tries to update an 'rc' object with the last >> element of the name tuple replaced with '\\_'. This make it possible to >> count resources whose names differ in only the last tuple element. For >> non-tuple resource names, it works just like before. >> >> BR, >> Ulf W >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> From ulf@REDACTED Thu Dec 1 22:14:30 2016 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 1 Dec 2016 22:14:30 +0100 Subject: [erlang-questions] wildcarded resource counters in Gproc In-Reply-To: References: Message-ID: Hi Garrett, Thanks for playing. ;-) In the case I have in mind, a given process may need to register multiple instances of a resource, where the key is basically {ServiceName, Route}. The resource count should keep track of instances of ServiceName, and especially trigger when the count reaches 0 (service no longer available). While one could add an extra function that creates a resource, also updating a wildcarded resource count, this opens up for possible bugs where the wrong function is used to create a resource, throwing the count out of sync. The proposed change was the most lightweight way I could think of to extend the 'rc' type, essentially adding one more layer of abstraction to it. But if there is another way, efficient and elegant, I'm wide open to suggestions. BR, Ulf W 2016-12-01 19:11 GMT+01:00 Garrett Smith : > This doesn't feel right to me - it strikes me as edge case support > that's impacting what's otherwise a very simple feature. > > If this is generally useful, would it make sense to implement it as a > new function call, which made the additional update explicit? > > On Thu, Dec 1, 2016 at 11:11 AM, Ulf Wiger wrote: > > There's a new PR on Gproc, adding support for limited wildcard matching > in > > resource counters. > > > > https://github.com/uwiger/gproc/pull/128 > > > > I don't know how many people use resource counters ('rc'), but thought > I'd > > ask. > > > > The operation of registering (or unregistering) a resource now makes one > > extra update_counter() protected by a try ... catch. My assumption is > that > > this extra overhead will be insignificant to just about all use cases, > but > > ... does anyone out there want to disagree? > > > > Briefly, when registering a resource ('r') object, gproc would try to > > perform an update_counter on the corresponding 'rc' object - an operation > > that might fail if the 'rc' object doesn't exist. Now, if the name of the > > resource is a tuple, it also tries to update an 'rc' object with the last > > element of the name tuple replaced with '\\_'. This make it possible to > > count resources whose names differ in only the last tuple element. For > > non-tuple resource names, it works just like before. > > > > BR, > > Ulf W > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Thu Dec 1 22:19:33 2016 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 1 Dec 2016 22:19:33 +0100 Subject: [erlang-questions] wildcarded resource counters in Gproc In-Reply-To: References: Message-ID: 2016-12-01 19:18 GMT+01:00 Garrett Smith : > And another tack - is there a path here to a generalized interface for > alternative schemes? E.g. passing along a function to do some work on > the table? Maybe not for the faint of heart, but this would reduce the > temptation to modify the interface for these case. One complication that comes to mind is the need for symmetry - at least in the case of aggregated counters and resource counters: they need to increment on one end and correspondingly decrement on the other. There is of course also the question of how much functionality can be crammed into gproc before people simply give up. :) But again, open to suggestions and PRs. BR, Ulf W -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Thu Dec 1 22:22:49 2016 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Thu, 1 Dec 2016 22:22:49 +0100 Subject: [erlang-questions] [erlang-patches] add compiler checked atoms In-Reply-To: References: <4F2EF17F.3020006@gmail.com> <4F3A2D95.8030305@gmail.com> Message-ID: Wow, time flies... has it been that long? Anyway, I found this old branch and decided to rework it like Kenneth suggested. Here's the PR: https://github.com/erlang/otp/pull/1265 /Richard 2012-02-14 12:24 GMT+01:00 Kenneth Lundin : > I think atoms introduced in both -type and -spec should be respected in > this feature. If you want to > guard yourself against a misspelled atom in one -spec you can use -type > but you are not forced to. > > And if you introduce an atom in a -spec and never use it in the code then > you could maybe issue a warning for that too. > > /Kenneth > > On Tue, Feb 14, 2012 at 10:47 AM, Richard Carlsson < > carlsson.richard@REDACTED> wrote: > >> On 02/14/2012 09:44 AM, Kenneth Lundin wrote: >> >>> Hi, >>> I don't think this is a good way to introduce declared atoms. >>> We already have the -type and -spec notation for definition of >>> types and function signatures which are then used by Dialyzer for type >>> checking. >>> I don't think we should introduce yet another notation that is not >>> harmonized with -type and -spec. >>> It is already possible to declare atoms in -type and -spec. Why not use >>> this already present notation and add >>> optional checking of atoms against declared atoms inside -type and -spec >>> instead. >>> With the suggested -atom declaration we will probably get the same atoms >>> declared 2 times with different notations and I think that >>> will clutter down the code with redundant information to an >>> unacceptable degree. >>> >> >> Yes, that is probably a better idea: that any atom occurring in a -type >> and/or -spec declaration is implicitly said to be known, and others are >> not. This would be an incentive for people to define types for things like >> the set of messages to a server or the set of atoms allowed as flags to a >> function (or error codes returned from a function). >> >> The question is, should atoms occurring in -spec declarations be taken as >> implicit "exists"-declarations, or should it only be those in -type? If >> it's only -type that counts, you could get checking of the atoms in -spec >> declarations as well, so you don't spell an atom wrong in the spec for one >> of 3 versions of a function and suddenly that atom is also implicitly >> allowed. I think -type only is the right way. >> >> I'll see if I can change my patch to do this instead. >> >> /Richard >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Thu Dec 1 22:45:27 2016 From: g@REDACTED (Garrett Smith) Date: Thu, 1 Dec 2016 15:45:27 -0600 Subject: [erlang-questions] wildcarded resource counters in Gproc In-Reply-To: References: Message-ID: I need to set aside some time - and compared to some of the recent events in global politics, this is not a super big deal. My line of inquiry would be a second type of counter where the key contained parts - and incrementing this type of counter added additional wild card entries up the path - so any level of wild carding is possible. I can't really make any informed statement without playing around. This just strikes me as a new type of counter. On Thu, Dec 1, 2016 at 3:19 PM, Ulf Wiger wrote: > > > 2016-12-01 19:18 GMT+01:00 Garrett Smith : >> >> And another tack - is there a path here to a generalized interface for >> alternative schemes? E.g. passing along a function to do some work on >> the table? Maybe not for the faint of heart, but this would reduce the >> temptation to modify the interface for these case. > > > One complication that comes to mind is the need for symmetry - at least in > the case of aggregated counters and resource counters: they need to > increment on one end and correspondingly decrement on the other. > > There is of course also the question of how much functionality can be > crammed into gproc before people simply give up. :) > > But again, open to suggestions and PRs. > > BR, > Ulf W > From nbartley@REDACTED Fri Dec 2 04:25:49 2016 From: nbartley@REDACTED (nato) Date: Thu, 1 Dec 2016 19:25:49 -0800 Subject: [erlang-questions] cover in a target release Message-ID: Is there a parent app/lib that I need to include so that I can have access to the `cover' module for a a target release? Not unlike having dbg via including runtime_tools ... I can't seem to figure it out via the documentation for cover. From kennethlakin@REDACTED Fri Dec 2 05:10:28 2016 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Thu, 1 Dec 2016 20:10:28 -0800 Subject: [erlang-questions] cover in a target release In-Reply-To: References: Message-ID: <4853d556-0953-6954-7c5f-8c8ec756764f@gmail.com> On 12/01/2016 07:25 PM, nato wrote: > Is there a parent app/lib that I need to include so that I can have > access to the `cover' module for a a target release? Have you tried the 'tools' application? The User's Guide in the cover man page is a link to the 'tools' User's Guide . Also, the man page index seems to indicate that cover belongs to the tools app. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ingela.andin@REDACTED Fri Dec 2 11:28:05 2016 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 2 Dec 2016 11:28:05 +0100 Subject: [erlang-questions] Unimplemented {buffer, integer()} option in ssl module - bug? In-Reply-To: References: Message-ID: 2016-12-01 12:37 GMT+01:00 Richard Jones : > The {buffer, integer()} option from gen_tcp is useful for breaking up > incoming data into manageable sizes, especially when in {packet, line} > mode. > > The docs suggest (kinda) that the gen_tcp options should also work > when connecting via the ssl module, but this isn' always the case - > the buffer option is silently ignored by ssl:connect > Actually it is not ignored it just is not has no noticeable affect to you. Is this a bug? Tested on erl 17, 18, 19 > > For this option to work as expected for you it needs to be emulated by the ssl process as are packet related options. So it is a missing feature. Regards Ingela Erlang/OTP- team > Examples follow using netcat/socat as a server. Note that gen_tcp > breaks the incoming data into 3 messages, ssl delivers as one large > message. > > Non-SSL example: > ---------------- > > $ echo "12345678901234567890123" | nc -l 1234 > > 1> {ok, _} = gen_tcp:connect("localhost", 1234, [ > binary, > {packet, line}, > {buffer, 10}, > {active, true}]). > > {ok,#Port<0.1180>} > > 2> flush(). > > Shell got {tcp,#Port<0.1180>,<<"1234567890">>} > Shell got {tcp,#Port<0.1180>,<<"1234567890">>} > Shell got {tcp,#Port<0.1180>,<<"123\n">>} > Shell got {tcp_closed,#Port<0.1180>} > ok > > SSL example: > ------------ > > $ echo "12345678901234567890123" | socat STDIO > OPENSSL-LISTEN:1234,reuseaddr,cert=/some.pem,cafile=/some.crt,verify=0 > > 1> {ok, _} = ssl:connect("localhost", 1234, [ > binary, > {packet, line}, > {buffer, 10}, > {active, true}, > {verify, verify_none}]). > > {ok,{sslsocket,{gen_tcp,#Port<0.1179>,tls_connection, > undefined}, > <0.54.0>}} > > 2> flush(). > > Shell got {ssl,{sslsocket,{gen_tcp,#Port<0.1179>,tls_connection, > undefined}, > <0.54.0>}, > <<"12345678901234567890123\n">>} > Shell got {ssl_closed, > {sslsocket, > {gen_tcp,#Port<0.1179>,tls_connection,undefined}, > <0.54.0>}} > ok > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex0player@REDACTED Fri Dec 2 11:43:41 2016 From: alex0player@REDACTED (Alex S.) Date: Fri, 2 Dec 2016 13:43:41 +0300 Subject: [erlang-questions] Unimplemented {buffer, integer()} option in ssl module - bug? In-Reply-To: References: Message-ID: > 2 ???. 2016 ?., ? 13:28, Ingela Andin ???????(?): > > > > 2016-12-01 12:37 GMT+01:00 Richard Jones >: > The {buffer, integer()} option from gen_tcp is useful for breaking up > incoming data into manageable sizes, especially when in {packet, line} > mode. > > The docs suggest (kinda) that the gen_tcp options should also work > when connecting via the ssl module, but this isn' > always the case - > the buffer option is silently ignored by ssl:connect > > Actually it is not ignored it just is not has no noticeable affect to you. > > Is this > a bug? Tested on erl 17, 18, 19 > > > For this option to work as expected for you it needs to be emulated by the ssl process as are packet related options. So it is a missing feature. > > Regards Ingela Erlang/OTP- team I would argue that the combination of these two effects can also be described as ?buffer_size is pushed a level lower? (where it is obviously not useful to the user of ssl module), and I would additionally argue that erroneously so. The option needs to be either explicitly not supported (with a note in the documentation), or do the right thing, not do some invisible thing that has no positive effect. -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Fri Dec 2 12:24:49 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 2 Dec 2016 14:24:49 +0300 Subject: [erlang-questions] Are any code dependencies in erlang from glibc? Message-ID: We need to make erlang package for ancient ubuntu 10. Are there any dependencies from new features in glibc older than 2.4? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Fri Dec 2 14:57:48 2016 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 2 Dec 2016 14:57:48 +0100 Subject: [erlang-questions] Unimplemented {buffer, integer()} option in ssl module - bug? In-Reply-To: References: Message-ID: Hi! 2016-12-02 11:43 GMT+01:00 Alex S. : > > 2 ???. 2016 ?., ? 13:28, Ingela Andin ???????(?): > > > > 2016-12-01 12:37 GMT+01:00 Richard Jones : > >> The {buffer, integer()} option from gen_tcp is useful for breaking up >> incoming data into manageable sizes, especially when in {packet, line} >> mode. >> >> The docs suggest (kinda) that the gen_tcp options should also work >> when connecting via the ssl module, but this isn' > > always the case - >> the buffer option is silently ignored by ssl:connect >> > > Actually it is not ignored it just is not has no noticeable affect to you. > > > Is this > > a bug? Tested on erl 17, 18, 19 >> >> > For this option to work as expected for you it needs to be emulated by the > ssl process as are packet related options. So it is a missing feature. > > Regards Ingela Erlang/OTP- team > > I would argue that the combination of these two effects can also be > described as ?buffer_size is pushed a level lower? (where it is obviously > not useful to the user of ssl module), and I would additionally argue that > erroneously so. The option needs to be either explicitly not supported > (with a note in the documentation), or do the right thing, not do some > invisible thing that has no positive effect. > I am not disagreeing with you on that. I think we can consider it a bug that this options is let through, and a missing feature that it is not emulated. The reasoning is that ssl shall not do option checking on gen_tcp options that may be used and does note affect ssl. But some options must be emulated. This option was clearly missed. Anyone wanting this option is welcome to make a PR. Regards Ingela Erlang/OTP- team -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Fri Dec 2 16:21:56 2016 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Fri, 2 Dec 2016 16:21:56 +0100 Subject: [erlang-questions] Do you wanna play with enif_select? Message-ID: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> https://github.com/erlang/otp/pull/1264 This PR is introducing a new feature enif_select() as a way for NIFs to ask for asynchronous notification when a file descriptor (a socket for example) is ready for (more) reading or writing. The API is not carved in stone yet and we would like some feedback. So please, take enif_select for a spin and share your thoughts. /Sverker, Erlang/OTP @ Ericsson From vans_163@REDACTED Fri Dec 2 16:47:36 2016 From: vans_163@REDACTED (Vans S) Date: Fri, 2 Dec 2016 15:47:36 +0000 (UTC) Subject: [erlang-questions] Do you wanna play with enif_select? In-Reply-To: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> References: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> Message-ID: <114754091.6293987.1480693656506@mail.yahoo.com> The title intrigued me. On line 148 of lib/kernel/test/netmarks_SUITE_data/sverk_tcp.c + cret = connect(sock, ai->ai_addr, ai->ai_addrlen); Any idea what a good way would be to make this a non-blocking connect. We obv want to avoid callback spaghetti in erlang, but we also want to avoid blocking in the nifs? Im thinking a try_connect select recursion func on the erlang side? Or would blocking on the connect there in the NIF be fine now and not throw off the scheduler? I noticed some files commited that are called after benchmark suites. Any results of how performance fairs compared to the built in net-io lib? On Friday, December 2, 2016 10:22 AM, Sverker Eriksson wrote: https://github.com/erlang/otp/pull/1264 This PR is introducing a new feature enif_select() as a way for NIFs to ask for asynchronous notification when a file descriptor (a socket for example) is ready for (more) reading or writing. The API is not carved in stone yet and we would like some feedback. So please, take enif_select for a spin and share your thoughts. /Sverker, Erlang/OTP @ Ericsson _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From hachreak@REDACTED Fri Dec 2 13:46:48 2016 From: hachreak@REDACTED (Leo) Date: Fri, 2 Dec 2016 13:46:48 +0100 Subject: [erlang-questions] Are any code dependencies in erlang from glibc? In-Reply-To: References: Message-ID: 2016-12-02 12:24 GMT+01:00 Max Lapshin : > > > We need to make erlang package for ancient ubuntu 10. > > Are there any dependencies from new features in glibc older than 2.4? > I don't know if this answer to your question, but maybe it can be useful to know.. XD Have you watched HydrOS presentation? https://youtu.be/9XqHiumZ02E?t=10m13s If I understood correctly, they were able to compile/run erlang implementing the minimum set of functions for the standard library: 154 function definitions stubs, 24 implemented functions. Leo -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Fri Dec 2 17:34:10 2016 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Fri, 2 Dec 2016 17:34:10 +0100 Subject: [erlang-questions] Do you wanna play with enif_select? In-Reply-To: <114754091.6293987.1480693656506@mail.yahoo.com> References: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> <114754091.6293987.1480693656506@mail.yahoo.com> Message-ID: <2f0c4320-7517-f1b7-e9af-093d499b537c@ericsson.com> The connect in sverk_tcp is blocking (for no reason) but the send and receive ops are non-blocking using enif_select() when the socket buffers gets full/empty. To fix the connect move up SET_NONBLOCKING(sock) before connect() then catch the EINPROGRESS from connect and call enif_select(.., ERL_NIF_SELECT_WRITE, ..) return to Erlang and wait for read_output message and then call some continue_connect_nif for example. I did some runs on the loopback and it looked promising, almost twice the throughput if I remember correctly. But, sverk_tcp is a very stripped down implementation without all the features that gen_tcp provides, so I wouldn't get overly excited about that. You can compare yourself with ts:run(kernel, netmarks_SUITE, sverk_tcp). vs ts:run(kernel, netmarks_SUITE, gen_tcp). /Sverker On 12/02/2016 04:47 PM, Vans S wrote: > The title intrigued me. > > On line 148 of lib/kernel/test/netmarks_SUITE_data/sverk_tcp.c > > + cret = connect(sock, ai->ai_addr, ai->ai_addrlen); > > Any idea what a good way would be to make this a non-blocking connect. > We obv want to avoid callback spaghetti in erlang, but we also want to avoid blocking > in the nifs? > > Im thinking a try_connect select recursion func on the erlang side? > > Or would blocking on the connect there in the NIF be fine now and not throw off the scheduler? > > I noticed some files commited that are called after benchmark suites. Any results of how performance fairs compared > to the built in net-io lib? > > > > On Friday, December 2, 2016 10:22 AM, Sverker Eriksson wrote: > https://github.com/erlang/otp/pull/1264 > > > This PR is introducing a new feature enif_select() as a way for > NIFs to ask for asynchronous notification when a file descriptor > (a socket for example) is ready for (more) reading or writing. > > The API is not carved in stone yet and we would like some feedback. > So please, take enif_select for a spin and share your thoughts. > > /Sverker, Erlang/OTP @ Ericsson > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From vans_163@REDACTED Fri Dec 2 17:39:35 2016 From: vans_163@REDACTED (Vans S) Date: Fri, 2 Dec 2016 16:39:35 +0000 (UTC) Subject: [erlang-questions] Do you wanna play with enif_select? In-Reply-To: <2f0c4320-7517-f1b7-e9af-093d499b537c@ericsson.com> References: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> <114754091.6293987.1480693656506@mail.yahoo.com> <2f0c4320-7517-f1b7-e9af-093d499b537c@ericsson.com> Message-ID: <2038423209.6354545.1480696775618@mail.yahoo.com> Interesting, so erlang gets notified when the connect completes (via read_output msg), that is a great solution. I will try this eventually and let know. Maybe it will fix my issue https://github.com/vans163/stargate/issues/1 On Friday, December 2, 2016 11:34 AM, Sverker Eriksson wrote: The connect in sverk_tcp is blocking (for no reason) but the send and receive ops are non-blocking using enif_select() when the socket buffers gets full/empty. To fix the connect move up SET_NONBLOCKING(sock) before connect() then catch the EINPROGRESS from connect and call enif_select(.., ERL_NIF_SELECT_WRITE, ..) return to Erlang and wait for read_output message and then call some continue_connect_nif for example. I did some runs on the loopback and it looked promising, almost twice the throughput if I remember correctly. But, sverk_tcp is a very stripped down implementation without all the features that gen_tcp provides, so I wouldn't get overly excited about that. You can compare yourself with ts:run(kernel, netmarks_SUITE, sverk_tcp). vs ts:run(kernel, netmarks_SUITE, gen_tcp). /Sverker On 12/02/2016 04:47 PM, Vans S wrote: > The title intrigued me. > > On line 148 of lib/kernel/test/netmarks_SUITE_data/sverk_tcp.c > > + cret = connect(sock, ai->ai_addr, ai->ai_addrlen); > > Any idea what a good way would be to make this a non-blocking connect. > We obv want to avoid callback spaghetti in erlang, but we also want to avoid blocking > in the nifs? > > Im thinking a try_connect select recursion func on the erlang side? > > Or would blocking on the connect there in the NIF be fine now and not throw off the scheduler? > > I noticed some files commited that are called after benchmark suites. Any results of how performance fairs compared > to the built in net-io lib? > > > > On Friday, December 2, 2016 10:22 AM, Sverker Eriksson wrote: > https://github.com/erlang/otp/pull/1264 > > > This PR is introducing a new feature enif_select() as a way for > NIFs to ask for asynchronous notification when a file descriptor > (a socket for example) is ready for (more) reading or writing. > > The API is not carved in stone yet and we would like some feedback. > So please, take enif_select for a spin and share your thoughts. > > /Sverker, Erlang/OTP @ Ericsson > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From max.lapshin@REDACTED Fri Dec 2 18:30:31 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 2 Dec 2016 20:30:31 +0300 Subject: [erlang-questions] Do you wanna play with enif_select? In-Reply-To: <2f0c4320-7517-f1b7-e9af-093d499b537c@ericsson.com> References: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> <114754091.6293987.1480693656506@mail.yahoo.com> <2f0c4320-7517-f1b7-e9af-093d499b537c@ericsson.com> Message-ID: > almost twice the throughput if I remember correctly. you mean that new implementation outperforms driver? Driver is calling C callback without scheduling to erlang and enif is sending message and it takes less CPU cycles? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Fri Dec 2 20:29:54 2016 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Fri, 2 Dec 2016 20:29:54 +0100 Subject: [erlang-questions] Do you wanna play with enif_select? In-Reply-To: References: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> <114754091.6293987.1480693656506@mail.yahoo.com> <2f0c4320-7517-f1b7-e9af-093d499b537c@ericsson.com> Message-ID: <7416345b-1ee0-2042-10ba-aff215bcf92b@ericsson.com> On 12/02/2016 06:30 PM, Max Lapshin wrote: >> almost twice the throughput if I remember correctly. > you mean that new implementation outperforms driver? Driver is calling C > callback without scheduling to erlang and enif is sending message and it > takes less CPU cycles? > Well, ports are actually scheduled to execute the C callbacks. But I don't think that matters in my benchmark runs, because the number of calls to enif_select and messages sent are tiny compared to the number of successful read/write calls. So "twice the throughput" is probably more an effect of NIF call overhead vs port call overhead plus the efficiency of my toy client NIF code vs gen_tcp down to inet_drv.c with all their features. /Sverker From max.lapshin@REDACTED Fri Dec 2 20:34:54 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 2 Dec 2016 22:34:54 +0300 Subject: [erlang-questions] Do you wanna play with enif_select? In-Reply-To: <7416345b-1ee0-2042-10ba-aff215bcf92b@ericsson.com> References: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> <114754091.6293987.1480693656506@mail.yahoo.com> <2f0c4320-7517-f1b7-e9af-093d499b537c@ericsson.com> <7416345b-1ee0-2042-10ba-aff215bcf92b@ericsson.com> Message-ID: Well, we can try your sverk_tcp on our test installation with 1gbps of input. But I'm a bit afraid of blocking connect. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vans_163@REDACTED Fri Dec 2 21:22:27 2016 From: vans_163@REDACTED (Vans S) Date: Fri, 2 Dec 2016 20:22:27 +0000 (UTC) Subject: [erlang-questions] Do you wanna play with enif_select? In-Reply-To: References: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> <114754091.6293987.1480693656506@mail.yahoo.com> <2f0c4320-7517-f1b7-e9af-093d499b537c@ericsson.com> <7416345b-1ee0-2042-10ba-aff215bcf92b@ericsson.com> Message-ID: <1092195091.6583500.1480710147242@mail.yahoo.com> > Well, we can try your sverk_tcp on our test installation with ?1gbps of input. > But I'm a bit afraid of blocking connect.? I would be interested in this. ?Also the connect does not block if you apply the changes Sverker mentioned: """ To fix the connect move up SET_NONBLOCKING(sock) before connect() then catch the EINPROGRESS from connect and call enif_select(.., ERL_NIF_SELECT_WRITE, ..) return to Erlang and wait for read_output message and then call some continue_connect_nif for example. I did some runs on the loopback and it looked promising, almost twice the throughput if I remember correctly. But, sverk_tcp is a very stripped down implementation without all the features that gen_tcp provides, so I wouldn't get overly excited about that. """ On Friday, December 2, 2016 2:35 PM, Max Lapshin wrote: Well, we can try your sverk_tcp on our test installation with ?1gbps of input. But I'm a bit afraid of blocking connect.? _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaotixpro@REDACTED Fri Dec 2 23:34:39 2016 From: kaotixpro@REDACTED (Edgar H) Date: Fri, 2 Dec 2016 23:34:39 +0100 Subject: [erlang-questions] Erlang for an audio streaming server project? Message-ID: Hello! I'm currently developing a personal project for the university which consists in a Java desktop software which should stream live audio via HTTP so anybody can listen through the server's IP. I've been looking for Java libraries that do this but it's almost impossible to find one that is actually maintained nowadays. Then I found Erland and saw the possibility of using it together with Java. Does Erlang solve the typical problems you would have in Java having to code all the streaming process with the formats and such? I was thinking about taking this Java + Erlang path as it seems the most reasonable approach instead of using old libraries and also learn an interesting new language in the process. And also, I would be very pleased if you could provide me (if you know some) a starting point related to streaming with Erlang so I can start looking at it after learning the syntax of the language. Thanks in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From vans_163@REDACTED Sat Dec 3 03:59:38 2016 From: vans_163@REDACTED (Vans S) Date: Sat, 3 Dec 2016 02:59:38 +0000 (UTC) Subject: [erlang-questions] Erlang for an audio streaming server project? In-Reply-To: References: Message-ID: <565649226.6785684.1480733978447@mail.yahoo.com> > Does Erlang solve the typical problems you would have in Java Yes. > having to code all the streaming process with the formats and such? This is not a problem solely to java. > And also, I would be very pleased if you could provide me (if you know some) a starting point related to streaming with Erlang Streaming media can range from being elementary to being very complex. If you are looking to stream over HTTP, first figure out the method, you have Chunked-Encoding, websockets or other. Then find a webserver that supports that, ensure the frontend can decode the stream and its in the correct format. Decide if you need real time streaming with low latency. Once you have settled on the formats and methods, the rest is an elementary network service. On Friday, December 2, 2016 6:05 PM, Edgar H wrote: Hello! I'm currently developing a personal project for the university which consists in a Java desktop software which should stream live audio via HTTP so anybody can listen through the server's IP. I've been looking for Java libraries that do this but it's almost impossible to find one that is actually maintained nowadays. Then I found Erland and saw the possibility of using it together with Java. Does Erlang solve the typical problems you would have in Java having to code all the streaming process with the formats and such? I was thinking about taking this Java + Erlang path as it seems the most reasonable approach instead of using old libraries and also learn an interesting new language in the process. And also, I would be very pleased if you could provide me (if you know some) a starting point related to streaming with Erlang so I can start looking at it after learning the syntax of the language. Thanks in advance! _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From me@REDACTED Sun Dec 4 02:47:47 2016 From: me@REDACTED (Matthew Shapiro) Date: Sat, 3 Dec 2016 20:47:47 -0500 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance Message-ID: I posted this question in the Elixir forums a day or so ago, but I wanted to put it here as well to gain visibility by people who have more experience with the internals of Erlang, since my question is related more to the Erlang libraries rather than Elixir itself. ## Summary I am trying to create a media streaming server in Elixir, with an initial focus on RTMP publishing and playback. I chose Elixir/Erlang because it seemed like a perfect candidate but I seem to be having trouble. The testing setup is 3 applications, 1 RTMP publisher (3rd party OBS studio), 1 RTMP viewer (VLC), and my Elixir server. Both the publisher and viewer connect to my elixir server over localhost, the publisher sends the elixir server video and audio data and each packet gets relayed off to the viewer, all over TCP. The publisher is currently set to send 2500kbps, and network traffic shows it pretty close to this. When running the test I notice the video is stuttering a lot. VLC debug messages show it's receiving frames inconsistently and trying to compensate for it. After getting help from people in IRC and looking through observer, I think I have pretty much pinpointed the issue to the `:gen_tcp.send()` calls being slow, so slow in fact I have observed up to 5-10 seconds just to push out an individual send call. Since i know Erlang is heavily used in switches I can't believe that this performance I"m getting is normal. Lowering my video's bitrate to 500kbps does show smoother playback but I can still tell there is an issue. For reference, the code I have so far is up at https://github.com/KallDrexx/mmids-temp. Note that this is a temporary repository, I plan to split each of hte apps up into their own repositories, slap an MIT license on them, then upload them to hex once I have this thing stabilized. Based on diagnostics I coded a 2500kbps video is averaging 200-250 messages per second going from the publisher to the viewing client. # What is the architecture? The general architecture I have right now is that when any type of client connects I utilize `ranch` to spawn a `gen_server`. This server receives TCP binary (using `active_once` and `raw` flags), attempts to deserialize any RTMP messages contained in it, react to messages that can/should be reacted to, and respond with any responses back to the client. This all occurs within a single `gen_server` and no other processes are involved. For demonstration purposes when a viewing client requests playback I use `pg2` to subscribe to a specific channel for audio and video data. Publishing clients that are publishing a/v data on that same stream key push that data to all subscribed clients. The viewing clients then receive the a/v data, serialize them into RTMP messages, serialize them into binary, then send them off across the network pipe. # What have I tried? First I tried utilizing `:os.system_time(:milli_seconds)` to determine how long any audio/video data packet took from deserializing from the publisher to right before binary serialization of the client. I noticed that it would start out extremely fast and then pauses would occur (long 5-10 second pauses) and then batches of packets would get processed, then another pause, etc... Then I was reminded about observer, and I loaded it and saw the following graph: https://dl.dropboxusercontent.com/u/6753359/observer1.PNG. The I/O graph told me that while inbound traffic was smooth, outbound was being staggered. I then opened the process for the server managing the viewing client. I noticed the message queue length was constantly increasing, never decreasing, and the process was constantly stuck in the `prim_inet:send/3` function. In doing some Googling I came across [this thread]( http://erlang.2086793.n4.nabble.com/why-is-gen-tcp-send-slow-td2106954.html) talking about slow `send()` performance, and while it didn't have a definite fix it did mention batching up the binary for the send() call so I wasn't calling it 200 times every second. The first thing I tried was to utilize a timer. Instead of calling `send()` every message I put the binary in an iodata queue held in the gen_server's state. I then added `:timer.send_interval(100, :send_queue)` to my initialization thinking I could send data once every 100ms. This did not give any better results outside of managing the message queue better. What I noticed with observer and this timer was odd in that I would keep pressing the refresh hotkey and I would see my queue keep growing for up 5-10 seconds, and then go down to zero again. This repeated over and over, and every refresh it was still stuck on `prim_inet:send/3`. This seems to me that send is just taking a ridiculous amount of time. Changing the timer interval up or down did not really help noticably. The next thing I tried was to stop the interval and send every X times I try to send a message, allowing me to batch messages together but make smaller batches then the interval method caused. This didn't help by a noticeable amount either, and was worse for managing the message queue. Finally I tried tweaking the watermark values (even having them go up to 64k) but I could not stop prim_inet:send/3 from causing my process to wait upwards of 15 seconds. # So what now? I'm not quite sure how to proceed from here. I can't believe that sending data via TCP is really that bad for a VM that I hear so many low latency and soft-realtime praise for. At the end of the day when the final final system is built I am hoping to get 50 inputs sending data to 150 outputs (based on current performance I've seen from other third party products), each connection (in and out) dealing with around 3Mbps of audio/video data. So it's a bit disconcerning that I can't even get 1 in 1 out working reliably. Does anyone have any advice on where I go from here? -------------- next part -------------- An HTML attachment was scrubbed... URL: From felixgallo@REDACTED Sun Dec 4 03:36:09 2016 From: felixgallo@REDACTED (felixgallo@REDACTED) Date: Sat, 3 Dec 2016 18:36:09 -0800 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: References: Message-ID: Another avenue you could check would be your operating system, nic driver, and the physical layer. For example, I traced issues with symptoms much like yours to buffer problems inside a consumer wifi access point. It's possible there is a misconfiguration or limitation somewhere between you and the receiver. Anecdotally, Erlang is commonly used to ship many hundreds of concurrent video streams from a single box, so your failure at 1 stream is not expected. F. > On Dec 3, 2016, at 5:47 PM, Matthew Shapiro wrote: > > I posted this question in the Elixir forums a day or so ago, but I wanted to put it here as well to gain visibility by people who have more experience with the internals of Erlang, since my question is related more to the Erlang libraries rather than Elixir itself. > > ## Summary > > I am trying to create a media streaming server in Elixir, with an initial focus on RTMP publishing and playback. I chose Elixir/Erlang because it seemed like a perfect candidate but I seem to be having trouble. > > The testing setup is 3 applications, 1 RTMP publisher (3rd party OBS studio), 1 RTMP viewer (VLC), and my Elixir server. Both the publisher and viewer connect to my elixir server over localhost, the publisher sends the elixir server video and audio data and each packet gets relayed off to the viewer, all over TCP. The publisher is currently set to send 2500kbps, and network traffic shows it pretty close to this. > > When running the test I notice the video is stuttering a lot. VLC debug messages show it's receiving frames inconsistently and trying to compensate for it. > > After getting help from people in IRC and looking through observer, I think I have pretty much pinpointed the issue to the `:gen_tcp.send()` calls being slow, so slow in fact I have observed up to 5-10 seconds just to push out an individual send call. > > Since i know Erlang is heavily used in switches I can't believe that this performance I"m getting is normal. Lowering my video's bitrate to 500kbps does show smoother playback but I can still tell there is an issue. > > For reference, the code I have so far is up at https://github.com/KallDrexx/mmids-temp. Note that this is a temporary repository, I plan to split each of hte apps up into their own repositories, slap an MIT license on them, then upload them to hex once I have this thing stabilized. > > Based on diagnostics I coded a 2500kbps video is averaging 200-250 messages per second going from the publisher to the viewing client. > > # What is the architecture? > > The general architecture I have right now is that when any type of client connects I utilize `ranch` to spawn a `gen_server`. This server receives TCP binary (using `active_once` and `raw` flags), attempts to deserialize any RTMP messages contained in it, react to messages that can/should be reacted to, and respond with any responses back to the client. This all occurs within a single `gen_server` and no other processes are involved. > > For demonstration purposes when a viewing client requests playback I use `pg2` to subscribe to a specific channel for audio and video data. Publishing clients that are publishing a/v data on that same stream key push that data to all subscribed clients. The viewing clients then receive the a/v data, serialize them into RTMP messages, serialize them into binary, then send them off across the network pipe. > > # What have I tried? > > First I tried utilizing `:os.system_time(:milli_seconds)` to determine how long any audio/video data packet took from deserializing from the publisher to right before binary serialization of the client. I noticed that it would start out extremely fast and then pauses would occur (long 5-10 second pauses) and then batches of packets would get processed, then another pause, etc... > > Then I was reminded about observer, and I loaded it and saw the following graph: https://dl.dropboxusercontent.com/u/6753359/observer1.PNG. The I/O graph told me that while inbound traffic was smooth, outbound was being staggered. > > I then opened the process for the server managing the viewing client. I noticed the message queue length was constantly increasing, never decreasing, and the process was constantly stuck in the `prim_inet:send/3` function. > > In doing some Googling I came across [this thread](http://erlang.2086793.n4.nabble.com/why-is-gen-tcp-send-slow-td2106954.html) talking about slow `send()` performance, and while it didn't have a definite fix it did mention batching up the binary for the send() call so I wasn't calling it 200 times every second. > > The first thing I tried was to utilize a timer. Instead of calling `send()` every message I put the binary in an iodata queue held in the gen_server's state. I then added `:timer.send_interval(100, :send_queue)` to my initialization thinking I could send data once every 100ms. > > This did not give any better results outside of managing the message queue better. What I noticed with observer and this timer was odd in that I would keep pressing the refresh hotkey and I would see my queue keep growing for up 5-10 seconds, and then go down to zero again. This repeated over and over, and every refresh it was still stuck on `prim_inet:send/3`. This seems to me that send is just taking a ridiculous amount of time. Changing the timer interval up or down did not really help noticably. > > The next thing I tried was to stop the interval and send every X times I try to send a message, allowing me to batch messages together but make smaller batches then the interval method caused. This didn't help by a noticeable amount either, and was worse for managing the message queue. > > Finally I tried tweaking the watermark values (even having them go up to 64k) but I could not stop prim_inet:send/3 from causing my process to wait upwards of 15 seconds. > > # So what now? > > I'm not quite sure how to proceed from here. I can't believe that sending data via TCP is really that bad for a VM that I hear so many low latency and soft-realtime praise for. > > At the end of the day when the final final system is built I am hoping to get 50 inputs sending data to 150 outputs (based on current performance I've seen from other third party products), each connection (in and out) dealing with around 3Mbps of audio/video data. So it's a bit disconcerning that I can't even get 1 in 1 out working reliably. > > Does anyone have any advice on where I go from here? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From me@REDACTED Sun Dec 4 03:45:25 2016 From: me@REDACTED (Matthew Shapiro) Date: Sat, 3 Dec 2016 21:45:25 -0500 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: References: Message-ID: Well this is on windows 10, and all applications are running locally on 127.0.0.1. I suppose I could throw this on a linux box somewhere and test to see if localhost is just broken. On Sat, Dec 3, 2016 at 9:36 PM, wrote: > Another avenue you could check would be your operating system, nic driver, > and the physical layer. For example, I traced issues with symptoms much > like yours to buffer problems inside a consumer wifi access point. It's > possible there is a misconfiguration or limitation somewhere between you > and the receiver. > > Anecdotally, Erlang is commonly used to ship many hundreds of concurrent > video streams from a single box, so your failure at 1 stream is not > expected. > > F. > > On Dec 3, 2016, at 5:47 PM, Matthew Shapiro wrote: > > I posted this question in the Elixir forums a day or so ago, but I wanted > to put it here as well to gain visibility by people who have more > experience with the internals of Erlang, since my question is related more > to the Erlang libraries rather than Elixir itself. > > ## Summary > > I am trying to create a media streaming server in Elixir, with an initial > focus on RTMP publishing and playback. I chose Elixir/Erlang because it > seemed like a perfect candidate but I seem to be having trouble. > > The testing setup is 3 applications, 1 RTMP publisher (3rd party OBS > studio), 1 RTMP viewer (VLC), and my Elixir server. Both the publisher and > viewer connect to my elixir server over localhost, the publisher sends the > elixir server video and audio data and each packet gets relayed off to the > viewer, all over TCP. The publisher is currently set to send 2500kbps, and > network traffic shows it pretty close to this. > > When running the test I notice the video is stuttering a lot. VLC debug > messages show it's receiving frames inconsistently and trying to compensate > for it. > > After getting help from people in IRC and looking through observer, I > think I have pretty much pinpointed the issue to the `:gen_tcp.send()` > calls being slow, so slow in fact I have observed up to 5-10 seconds just > to push out an individual send call. > > Since i know Erlang is heavily used in switches I can't believe that this > performance I"m getting is normal. Lowering my video's bitrate to 500kbps > does show smoother playback but I can still tell there is an issue. > > For reference, the code I have so far is up at > https://github.com/KallDrexx/mmids-temp. Note that this is a temporary > repository, I plan to split each of hte apps up into their own > repositories, slap an MIT license on them, then upload them to hex once I > have this thing stabilized. > > Based on diagnostics I coded a 2500kbps video is averaging 200-250 > messages per second going from the publisher to the viewing client. > > # What is the architecture? > > The general architecture I have right now is that when any type of client > connects I utilize `ranch` to spawn a `gen_server`. This server receives > TCP binary (using `active_once` and `raw` flags), attempts to deserialize > any RTMP messages contained in it, react to messages that can/should be > reacted to, and respond with any responses back to the client. This all > occurs within a single `gen_server` and no other processes are involved. > > For demonstration purposes when a viewing client requests playback I use > `pg2` to subscribe to a specific channel for audio and video data. > Publishing clients that are publishing a/v data on that same stream key > push that data to all subscribed clients. The viewing clients then receive > the a/v data, serialize them into RTMP messages, serialize them into > binary, then send them off across the network pipe. > > # What have I tried? > > First I tried utilizing `:os.system_time(:milli_seconds)` to determine > how long any audio/video data packet took from deserializing from the > publisher to right before binary serialization of the client. I noticed > that it would start out extremely fast and then pauses would occur (long > 5-10 second pauses) and then batches of packets would get processed, then > another pause, etc... > > Then I was reminded about observer, and I loaded it and saw the following > graph: https://dl.dropboxusercontent.com/u/6753359/observer1.PNG. The > I/O graph told me that while inbound traffic was smooth, outbound was being > staggered. > > I then opened the process for the server managing the viewing client. I > noticed the message queue length was constantly increasing, never > decreasing, and the process was constantly stuck in the `prim_inet:send/3` > function. > > In doing some Googling I came across [this thread](http://erlang.2086793. > n4.nabble.com/why-is-gen-tcp-send-slow-td2106954.html) talking about slow > `send()` performance, and while it didn't have a definite fix it did > mention batching up the binary for the send() call so I wasn't calling it > 200 times every second. > > The first thing I tried was to utilize a timer. Instead of calling > `send()` every message I put the binary in an iodata queue held in the > gen_server's state. I then added `:timer.send_interval(100, :send_queue)` > to my initialization thinking I could send data once every 100ms. > > This did not give any better results outside of managing the message queue > better. What I noticed with observer and this timer was odd in that I > would keep pressing the refresh hotkey and I would see my queue keep > growing for up 5-10 seconds, and then go down to zero again. This repeated > over and over, and every refresh it was still stuck on `prim_inet:send/3`. > This seems to me that send is just taking a ridiculous amount of time. > Changing the timer interval up or down did not really help noticably. > > The next thing I tried was to stop the interval and send every X times I > try to send a message, allowing me to batch messages together but make > smaller batches then the interval method caused. This didn't help by a > noticeable amount either, and was worse for managing the message queue. > > Finally I tried tweaking the watermark values (even having them go up to > 64k) but I could not stop prim_inet:send/3 from causing my process to wait > upwards of 15 seconds. > > # So what now? > > I'm not quite sure how to proceed from here. I can't believe that sending > data via TCP is really that bad for a VM that I hear so many low latency > and soft-realtime praise for. > > At the end of the day when the final final system is built I am hoping to > get 50 inputs sending data to 150 outputs (based on current performance > I've seen from other third party products), each connection (in and out) > dealing with around 3Mbps of audio/video data. So it's a bit disconcerning > that I can't even get 1 in 1 out working reliably. > > Does anyone have any advice on where I go from here? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me@REDACTED Sun Dec 4 04:42:14 2016 From: me@REDACTED (Matthew Shapiro) Date: Sat, 3 Dec 2016 22:42:14 -0500 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: References: Message-ID: On a whim (sorry probably should have tried this earlier) I used ffmpeg as my playback client instead of VLC. For some reason when using ffmpeg there's zero backlog shown in observer (even with send each packet immediately) and the I/O chart in the load tab has the Output line exactly tracking the input graph (which is what I would expect for a 1 in 1 out connection. So I guess VLC is doing something funky on localhost connections that is making prim_inet.send/3 to go into waiting. On Sat, Dec 3, 2016 at 9:45 PM, Matthew Shapiro wrote: > Well this is on windows 10, and all applications are running locally on > 127.0.0.1. I suppose I could throw this on a linux box somewhere and test > to see if localhost is just broken. > > On Sat, Dec 3, 2016 at 9:36 PM, wrote: > >> Another avenue you could check would be your operating system, nic >> driver, and the physical layer. For example, I traced issues with symptoms >> much like yours to buffer problems inside a consumer wifi access point. >> It's possible there is a misconfiguration or limitation somewhere between >> you and the receiver. >> >> Anecdotally, Erlang is commonly used to ship many hundreds of concurrent >> video streams from a single box, so your failure at 1 stream is not >> expected. >> >> F. >> >> On Dec 3, 2016, at 5:47 PM, Matthew Shapiro wrote: >> >> I posted this question in the Elixir forums a day or so ago, but I wanted >> to put it here as well to gain visibility by people who have more >> experience with the internals of Erlang, since my question is related more >> to the Erlang libraries rather than Elixir itself. >> >> ## Summary >> >> I am trying to create a media streaming server in Elixir, with an initial >> focus on RTMP publishing and playback. I chose Elixir/Erlang because it >> seemed like a perfect candidate but I seem to be having trouble. >> >> The testing setup is 3 applications, 1 RTMP publisher (3rd party OBS >> studio), 1 RTMP viewer (VLC), and my Elixir server. Both the publisher and >> viewer connect to my elixir server over localhost, the publisher sends the >> elixir server video and audio data and each packet gets relayed off to the >> viewer, all over TCP. The publisher is currently set to send 2500kbps, and >> network traffic shows it pretty close to this. >> >> When running the test I notice the video is stuttering a lot. VLC debug >> messages show it's receiving frames inconsistently and trying to compensate >> for it. >> >> After getting help from people in IRC and looking through observer, I >> think I have pretty much pinpointed the issue to the `:gen_tcp.send()` >> calls being slow, so slow in fact I have observed up to 5-10 seconds just >> to push out an individual send call. >> >> Since i know Erlang is heavily used in switches I can't believe that this >> performance I"m getting is normal. Lowering my video's bitrate to 500kbps >> does show smoother playback but I can still tell there is an issue. >> >> For reference, the code I have so far is up at >> https://github.com/KallDrexx/mmids-temp. Note that this is a temporary >> repository, I plan to split each of hte apps up into their own >> repositories, slap an MIT license on them, then upload them to hex once I >> have this thing stabilized. >> >> Based on diagnostics I coded a 2500kbps video is averaging 200-250 >> messages per second going from the publisher to the viewing client. >> >> # What is the architecture? >> >> The general architecture I have right now is that when any type of client >> connects I utilize `ranch` to spawn a `gen_server`. This server receives >> TCP binary (using `active_once` and `raw` flags), attempts to deserialize >> any RTMP messages contained in it, react to messages that can/should be >> reacted to, and respond with any responses back to the client. This all >> occurs within a single `gen_server` and no other processes are involved. >> >> For demonstration purposes when a viewing client requests playback I use >> `pg2` to subscribe to a specific channel for audio and video data. >> Publishing clients that are publishing a/v data on that same stream key >> push that data to all subscribed clients. The viewing clients then receive >> the a/v data, serialize them into RTMP messages, serialize them into >> binary, then send them off across the network pipe. >> >> # What have I tried? >> >> First I tried utilizing `:os.system_time(:milli_seconds)` to determine >> how long any audio/video data packet took from deserializing from the >> publisher to right before binary serialization of the client. I noticed >> that it would start out extremely fast and then pauses would occur (long >> 5-10 second pauses) and then batches of packets would get processed, then >> another pause, etc... >> >> Then I was reminded about observer, and I loaded it and saw the following >> graph: https://dl.dropboxusercontent.com/u/6753359/observer1.PNG. The >> I/O graph told me that while inbound traffic was smooth, outbound was being >> staggered. >> >> I then opened the process for the server managing the viewing client. I >> noticed the message queue length was constantly increasing, never >> decreasing, and the process was constantly stuck in the `prim_inet:send/3` >> function. >> >> In doing some Googling I came across [this thread](http://erlang.2086793. >> n4.nabble.com/why-is-gen-tcp-send-slow-td2106954.html) talking about >> slow `send()` performance, and while it didn't have a definite fix it did >> mention batching up the binary for the send() call so I wasn't calling it >> 200 times every second. >> >> The first thing I tried was to utilize a timer. Instead of calling >> `send()` every message I put the binary in an iodata queue held in the >> gen_server's state. I then added `:timer.send_interval(100, :send_queue)` >> to my initialization thinking I could send data once every 100ms. >> >> This did not give any better results outside of managing the message >> queue better. What I noticed with observer and this timer was odd in that >> I would keep pressing the refresh hotkey and I would see my queue keep >> growing for up 5-10 seconds, and then go down to zero again. This repeated >> over and over, and every refresh it was still stuck on `prim_inet:send/3`. >> This seems to me that send is just taking a ridiculous amount of time. >> Changing the timer interval up or down did not really help noticably. >> >> The next thing I tried was to stop the interval and send every X times I >> try to send a message, allowing me to batch messages together but make >> smaller batches then the interval method caused. This didn't help by a >> noticeable amount either, and was worse for managing the message queue. >> >> Finally I tried tweaking the watermark values (even having them go up to >> 64k) but I could not stop prim_inet:send/3 from causing my process to wait >> upwards of 15 seconds. >> >> # So what now? >> >> I'm not quite sure how to proceed from here. I can't believe that >> sending data via TCP is really that bad for a VM that I hear so many low >> latency and soft-realtime praise for. >> >> At the end of the day when the final final system is built I am hoping to >> get 50 inputs sending data to 150 outputs (based on current performance >> I've seen from other third party products), each connection (in and out) >> dealing with around 3Mbps of audio/video data. So it's a bit disconcerning >> that I can't even get 1 in 1 out working reliably. >> >> Does anyone have any advice on where I go from here? >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From felixgallo@REDACTED Sun Dec 4 05:20:59 2016 From: felixgallo@REDACTED (felixgallo@REDACTED) Date: Sat, 3 Dec 2016 20:20:59 -0800 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: References: Message-ID: Try using your external ip instead of 127.0.0.1 to validate that theory. Also maybe run sender and receiver on separate machines as a test. I've had unusual problems with windows localhost before as well so your idea is in the realm of possibility. Linux boxes have better network debugging tools so moving over there if the problem continues could be fruitful. F. > On Dec 3, 2016, at 7:42 PM, Matthew Shapiro wrote: > > On a whim (sorry probably should have tried this earlier) I used ffmpeg as my playback client instead of VLC. For some reason when using ffmpeg there's zero backlog shown in observer (even with send each packet immediately) and the I/O chart in the load tab has the Output line exactly tracking the input graph (which is what I would expect for a 1 in 1 out connection. > > So I guess VLC is doing something funky on localhost connections that is making prim_inet.send/3 to go into waiting. > > On Sat, Dec 3, 2016 at 9:45 PM, Matthew Shapiro wrote: >> Well this is on windows 10, and all applications are running locally on 127.0.0.1. I suppose I could throw this on a linux box somewhere and test to see if localhost is just broken. >> >>> On Sat, Dec 3, 2016 at 9:36 PM, wrote: >>> Another avenue you could check would be your operating system, nic driver, and the physical layer. For example, I traced issues with symptoms much like yours to buffer problems inside a consumer wifi access point. It's possible there is a misconfiguration or limitation somewhere between you and the receiver. >>> >>> Anecdotally, Erlang is commonly used to ship many hundreds of concurrent video streams from a single box, so your failure at 1 stream is not expected. >>> >>> F. >>> >>>> On Dec 3, 2016, at 5:47 PM, Matthew Shapiro wrote: >>>> >>>> I posted this question in the Elixir forums a day or so ago, but I wanted to put it here as well to gain visibility by people who have more experience with the internals of Erlang, since my question is related more to the Erlang libraries rather than Elixir itself. >>>> >>>> ## Summary >>>> >>>> I am trying to create a media streaming server in Elixir, with an initial focus on RTMP publishing and playback. I chose Elixir/Erlang because it seemed like a perfect candidate but I seem to be having trouble. >>>> >>>> The testing setup is 3 applications, 1 RTMP publisher (3rd party OBS studio), 1 RTMP viewer (VLC), and my Elixir server. Both the publisher and viewer connect to my elixir server over localhost, the publisher sends the elixir server video and audio data and each packet gets relayed off to the viewer, all over TCP. The publisher is currently set to send 2500kbps, and network traffic shows it pretty close to this. >>>> >>>> When running the test I notice the video is stuttering a lot. VLC debug messages show it's receiving frames inconsistently and trying to compensate for it. >>>> >>>> After getting help from people in IRC and looking through observer, I think I have pretty much pinpointed the issue to the `:gen_tcp.send()` calls being slow, so slow in fact I have observed up to 5-10 seconds just to push out an individual send call. >>>> >>>> Since i know Erlang is heavily used in switches I can't believe that this performance I"m getting is normal. Lowering my video's bitrate to 500kbps does show smoother playback but I can still tell there is an issue. >>>> >>>> For reference, the code I have so far is up at https://github.com/KallDrexx/mmids-temp. Note that this is a temporary repository, I plan to split each of hte apps up into their own repositories, slap an MIT license on them, then upload them to hex once I have this thing stabilized. >>>> >>>> Based on diagnostics I coded a 2500kbps video is averaging 200-250 messages per second going from the publisher to the viewing client. >>>> >>>> # What is the architecture? >>>> >>>> The general architecture I have right now is that when any type of client connects I utilize `ranch` to spawn a `gen_server`. This server receives TCP binary (using `active_once` and `raw` flags), attempts to deserialize any RTMP messages contained in it, react to messages that can/should be reacted to, and respond with any responses back to the client. This all occurs within a single `gen_server` and no other processes are involved. >>>> >>>> For demonstration purposes when a viewing client requests playback I use `pg2` to subscribe to a specific channel for audio and video data. Publishing clients that are publishing a/v data on that same stream key push that data to all subscribed clients. The viewing clients then receive the a/v data, serialize them into RTMP messages, serialize them into binary, then send them off across the network pipe. >>>> >>>> # What have I tried? >>>> >>>> First I tried utilizing `:os.system_time(:milli_seconds)` to determine how long any audio/video data packet took from deserializing from the publisher to right before binary serialization of the client. I noticed that it would start out extremely fast and then pauses would occur (long 5-10 second pauses) and then batches of packets would get processed, then another pause, etc... >>>> >>>> Then I was reminded about observer, and I loaded it and saw the following graph: https://dl.dropboxusercontent.com/u/6753359/observer1.PNG. The I/O graph told me that while inbound traffic was smooth, outbound was being staggered. >>>> >>>> I then opened the process for the server managing the viewing client. I noticed the message queue length was constantly increasing, never decreasing, and the process was constantly stuck in the `prim_inet:send/3` function. >>>> >>>> In doing some Googling I came across [this thread](http://erlang.2086793.n4.nabble.com/why-is-gen-tcp-send-slow-td2106954.html) talking about slow `send()` performance, and while it didn't have a definite fix it did mention batching up the binary for the send() call so I wasn't calling it 200 times every second. >>>> >>>> The first thing I tried was to utilize a timer. Instead of calling `send()` every message I put the binary in an iodata queue held in the gen_server's state. I then added `:timer.send_interval(100, :send_queue)` to my initialization thinking I could send data once every 100ms. >>>> >>>> This did not give any better results outside of managing the message queue better. What I noticed with observer and this timer was odd in that I would keep pressing the refresh hotkey and I would see my queue keep growing for up 5-10 seconds, and then go down to zero again. This repeated over and over, and every refresh it was still stuck on `prim_inet:send/3`. This seems to me that send is just taking a ridiculous amount of time. Changing the timer interval up or down did not really help noticably. >>>> >>>> The next thing I tried was to stop the interval and send every X times I try to send a message, allowing me to batch messages together but make smaller batches then the interval method caused. This didn't help by a noticeable amount either, and was worse for managing the message queue. >>>> >>>> Finally I tried tweaking the watermark values (even having them go up to 64k) but I could not stop prim_inet:send/3 from causing my process to wait upwards of 15 seconds. >>>> >>>> # So what now? >>>> >>>> I'm not quite sure how to proceed from here. I can't believe that sending data via TCP is really that bad for a VM that I hear so many low latency and soft-realtime praise for. >>>> >>>> At the end of the day when the final final system is built I am hoping to get 50 inputs sending data to 150 outputs (based on current performance I've seen from other third party products), each connection (in and out) dealing with around 3Mbps of audio/video data. So it's a bit disconcerning that I can't even get 1 in 1 out working reliably. >>>> >>>> Does anyone have any advice on where I go from here? >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Sun Dec 4 08:23:09 2016 From: sergej.jurecko@REDACTED (=?utf-8?Q?Sergej_Jure=C4=8Dko?=) Date: Sun, 4 Dec 2016 08:23:09 +0100 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: References: Message-ID: <504E33D7-4029-4C54-B86E-B7882279170D@gmail.com> On 4 Dec 2016, at 04:42, Matthew Shapiro wrote: > > So I guess VLC is doing something funky on localhost connections that is making prim_inet.send/3 to go into waiting. There is only one thing that could be happening, VLC is not reading data from the socket fast enough, causing your side to block. Unless there is a bug in windows (or any possible virtualization if it is being used), which is possible as we've encountered it once and it resulted in similar behavior. I think your problems have pretty much nothing to do with erlang/elixir. This is a bit off topic, but why RTMP? Pretty much everything is or has moved to HLS/DASH. HLS especially will work on practically every client (all major browsers, all relevant mobiles). regards, Sergej From vans_163@REDACTED Sun Dec 4 17:37:50 2016 From: vans_163@REDACTED (Vans S) Date: Sun, 4 Dec 2016 16:37:50 +0000 (UTC) Subject: [erlang-questions] Binary memory usage problems. Ref holding, how to debug? References: <281547996.7511616.1480869470269.ref@mail.yahoo.com> Message-ID: <281547996.7511616.1480869470269@mail.yahoo.com> So I have an app that decodes very large JSON using jiffy into maps. The size of a single json ends up being 8mb or more. Using 'instrument' module we can see the memory_data(). Afaik 187 TypeNo is binary? Based on the index of binary in MemoryData = {term(), AllocList}, term() [{163,140323985428560,102675368,{0,328,0}}, {162,140324609712208,16582664,{0,328,0}}, {187,140322805108816,8857012,{0,19531,1}}, {187,140324408574032,8853330,{0,31903,1}}, {187,140324348485712,8841158,{0,18560,2}}, {187,140324318306384,8833780,{0,25812,2}}, {187,140324439670864,8828792,{0,25113,2}}, {187,140322376695888,8809693,{0,31250,1}}, {187,140323227144272,8785428,{0,20346,0}}, {187,140324357328976,8778769,{0,17234,2}}, {187,140324374511696,8751684,{0,10006,2}}, {187,140323087188048,8747085,{0,24112,1}}, {187,140324678754384,8728978,{0,14962,0}}, {187,140321762197584,8726561,{0,13901,2}}, {187,140322517651536,8724998,{0,23468,1}}, {187,140321205051472,8711957,{0,23748,2}}, Each of these pids is a temporary decode pid, that dies after decode. As soon as the PID dies, we clean our binary, life is great! BUT if we use a part of this binary, say we do BinaryOver64Bytes = maps:get(description, JsonMap), mnesia:write({just_example, BinaryOver64Bytes , 5} We have a huge problem now. This 8mb binary will NEVER get cleaned. Even with a forced GC. My hypothesis is jiffy optimizes decoding the json and does not copy the terms over (binary:copy/1) when it creates the output map. This means when we store a part of this binary permanently say into mnesia, we have the whole binary that never gets cleaned. The solution is obvious, binary:copy/1 the terms that are permanently stored from the 8mb binary. My question is, does erlang provide anything to see what is holding a ref to which binary? Also another question is should erlang make it so easy to shoot yourself in the foot here. Would it work to optimize the shared binary garbage collection that it will consider which chunk of a binary is used, and discard the rest that is unused? This is a common question/problem that arises and I already lost count of how many Erlang users start chats on this topic. IMO It should not be this easy to screw up, especially with a language like erlang. From max.lapshin@REDACTED Sun Dec 4 20:20:45 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Sun, 4 Dec 2016 22:20:45 +0300 Subject: [erlang-questions] Erlang for an audio streaming server project? In-Reply-To: References: Message-ID: You will have very interesting deployment pain if you take java with you. It is not clear what for to waste time on it? There are 2-3 codecs and 3-4 transport payloads for audio. Just write all this in erlang, it is not hard. -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Sun Dec 4 20:24:38 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Sun, 4 Dec 2016 22:24:38 +0300 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: <504E33D7-4029-4C54-B86E-B7882279170D@gmail.com> References: <504E33D7-4029-4C54-B86E-B7882279170D@gmail.com> Message-ID: The first question is your architecture. If you want to make gen_tcp:send to different sockets from one process, then you have wrong architecture and need to change all this. Flussonic has rtmp server inside and after some performance optimizations (all are high-level, algorithmic) it can send 1-3 gbit/s via rtmp. But it is the top. RTMP is a dead protocol, you are just wasting your time on it because till the end of 2017 it will be disabled in almost all browsers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From me@REDACTED Mon Dec 5 00:22:05 2016 From: me@REDACTED (Matthew Shapiro) Date: Sun, 4 Dec 2016 18:22:05 -0500 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: References: <504E33D7-4029-4C54-B86E-B7882279170D@gmail.com> Message-ID: In regards to your architecture response, Each process is responsible to sending and receiving to only one TCP socket as each process represents a single TCP connection. In terms of RTMP, I run a streaming service company so I'm pretty aware of whats going on. RTMP may be dead for viewing but it most certainly is not a dead protocol for live video ingestion and distribution, and thus there are still very valid reasons for RTMP support in a media server. I have much grander plans for this than just RTMP ingest/playback but it's a pointless effort to start working on other things like creating an HLS feed from an RTMP feed if I can't even get regular RTMP playback to work properly yet. On Sun, Dec 4, 2016 at 2:24 PM, Max Lapshin wrote: > The first question is your architecture. > > If you want to make gen_tcp:send to different sockets from one process, > then you have wrong architecture and need to change all this. > > Flussonic has rtmp server inside and after some performance optimizations > (all are high-level, algorithmic) it can send 1-3 gbit/s via rtmp. But it > is the top. RTMP is a dead protocol, you are just wasting your time on it > because till the end of 2017 it will be disabled in almost all browsers. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me@REDACTED Mon Dec 5 01:29:55 2016 From: me@REDACTED (Matthew Shapiro) Date: Sun, 4 Dec 2016 19:29:55 -0500 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: <504E33D7-4029-4C54-B86E-B7882279170D@gmail.com> References: <504E33D7-4029-4C54-B86E-B7882279170D@gmail.com> Message-ID: Yeah I think I can safely say this is something to do with how VLC is functioning. Even remotely I'm getting stuck in send, though not as bad, and looking at what's going on in the difference between VLC and Ffmpeg I believe my timestamps are being unexpected and both are reacting weirdly too it, with VLC trying to slow the feed down for whatever reason and that may be causing it to be slow to do a TCP read. Shot in the dark but at least I know now that it's not an BEAM issue (and my time on this hasn't been to waste). So sorry for the thread when Erlang/Beam isn't at fault in the end :) On Sun, Dec 4, 2016 at 2:23 AM, Sergej Jure?ko wrote: > On 4 Dec 2016, at 04:42, Matthew Shapiro wrote: > > > > So I guess VLC is doing something funky on localhost connections that is > making prim_inet.send/3 to go into waiting. > > There is only one thing that could be happening, VLC is not reading data > from the socket fast enough, causing your side to block. > > Unless there is a bug in windows (or any possible virtualization if it is > being used), which is possible as we've encountered it once and it resulted > in similar behavior. > > I think your problems have pretty much nothing to do with erlang/elixir. > > This is a bit off topic, but why RTMP? Pretty much everything is or has > moved to HLS/DASH. HLS especially will work on practically every client > (all major browsers, all relevant mobiles). > > regards, > Sergej -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Mon Dec 5 01:54:09 2016 From: zxq9@REDACTED (zxq9) Date: Mon, 05 Dec 2016 09:54:09 +0900 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: References: <504E33D7-4029-4C54-B86E-B7882279170D@gmail.com> Message-ID: <6387484.4fr15ba5RA@changa> On 2016?12?4? ??? 19:29:55 Matthew Shapiro wrote: > Yeah I think I can safely say this is something to do with how VLC is > functioning. Even remotely I'm getting stuck in send, though not as bad, > and looking at what's going on in the difference between VLC and Ffmpeg I > believe my timestamps are being unexpected and both are reacting weirdly > too it, with VLC trying to slow the feed down for whatever reason and that > may be causing it to be slow to do a TCP read. Shot in the dark but at > least I know now that it's not an BEAM issue (and my time on this hasn't > been to waste). > > So sorry for the thread when Erlang/Beam isn't at fault in the end :) It is of peripheral interest to those of us here in the peanut gallery, though -- everyone loves VLC, right? If you wouldn't mind I'd love to know the resolution whenever you get things works out. Have fun! -Craig From mononcqc@REDACTED Mon Dec 5 03:13:56 2016 From: mononcqc@REDACTED (Fred Hebert) Date: Sun, 4 Dec 2016 21:13:56 -0500 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: References: <504E33D7-4029-4C54-B86E-B7882279170D@gmail.com> Message-ID: <20161205021355.GI684@ferdmbp.local> On 12/04, Matthew Shapiro wrote: >Yeah I think I can safely say this is something to do with how VLC is >functioning. Even remotely I'm getting stuck in send, though not as bad, >and looking at what's going on in the difference between VLC and Ffmpeg I >believe my timestamps are being unexpected and both are reacting weirdly >too it, with VLC trying to slow the feed down for whatever reason and that >may be causing it to be slow to do a TCP read. Shot in the dark but at >least I know now that it's not an BEAM issue (and my time on this hasn't >been to waste). > >So sorry for the thread when Erlang/Beam isn't at fault in the end :) > It is and it is not. One of the interesting things with the TCP stack is that it sends a signal through ports, and if the send queue is full, the whole thing blocks; it however blocks with live data in there, and there's no good way to give it up. The reason is blocks that the TCP work is done over a port program, meaning message passing takes place: Erlang Process TCP port program | | |----------- Data ------------->| | | |<--- {inet_reply,S,Status} ----| This is roughly what takes place within the thing. This is done through calling erlang:port_command/2-3. The thing though is that if the buffer of the port program is full, you get descheduled until there's place. There's a cheat though, one we at Heroku have used in Logplex. The trick is to bypass the stack in critical 'must never block' circumstances and call port_command yourself, but by passing in the option list '[nosuspend]'. When you do this and the port program representing the TCP socket is busy, you'll get a 'false' return value indicating it failed because it was full (you get 'true' when it works): case erlang:port_command(Socket, Data, [nosuspend]) of false -> handle_busy(...); true -> receive {inet_reply, Socket, Status} -> % Status == send return Status end end This will let you do things like figure out the consumer side (in this case VLC) is too slow, and possibly give up sending a few frames and send newer ones later. Control flow is fun! Regards, Fred. From zxq9@REDACTED Mon Dec 5 03:22:59 2016 From: zxq9@REDACTED (zxq9) Date: Mon, 05 Dec 2016 11:22:59 +0900 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: <20161205021355.GI684@ferdmbp.local> References: <20161205021355.GI684@ferdmbp.local> Message-ID: <1534158.vnu8HodBQW@changa> On 2016?12?4? ??? 21:13:56 Fred Hebert wrote: > It is and it is not. One of the interesting things with the TCP stack is > that it sends a signal through ports, and if the send queue is full, the > whole thing blocks; it however blocks with live data in there, and > there's no good way to give it up. > > The reason is blocks that the TCP work is done over a port program, > meaning message passing takes place: > > Erlang Process TCP port program > | | > |----------- Data ------------->| > | | > |<--- {inet_reply,S,Status} ----| > > This is roughly what takes place within the thing. This is done through > calling erlang:port_command/2-3. The thing though is that if the buffer > of the port program is full, you get descheduled until there's place. > > There's a cheat though, one we at Heroku have used in Logplex. The trick > is to bypass the stack in critical 'must never block' circumstances and > call port_command yourself, but by passing in the option list > '[nosuspend]'. > > When you do this and the port program representing the TCP socket is > busy, you'll get a 'false' return value indicating it failed because it > was full (you get 'true' when it works): > > case erlang:port_command(Socket, Data, [nosuspend]) of > false -> > handle_busy(...); > true -> > receive > {inet_reply, Socket, Status} -> % Status == send return > Status > end > end > > This will let you do things like figure out the consumer side (in this > case VLC) is too slow, and possibly give up sending a few frames and > send newer ones later. > > Control flow is fun! ...and once again Fred has, by psychic divination, just supplied the answer to a question I would have asked about one week from now. I appreciate your knack for preemptive answer scheduling. It allows me to remain in lurk mode most of the time. -Craig From quantumpotato@REDACTED Mon Dec 5 02:43:08 2016 From: quantumpotato@REDACTED (qp) Date: Sun, 4 Dec 2016 20:43:08 -0500 Subject: [erlang-questions] Update Pid State on a Timer Message-ID: I am trying to update my process's state on a 3 second timer. -define(INTERVAL, 3000). start_link() -> gen_server:start_link(?MODULE, [], []). action(Pid, Action) -> gen_server:call(Pid, Action). init([]) -> erlang:send_after(?INTERVAL, self(), trigger), {ok, temple:new()}. what I want to do is call this handle_call({fight}, _From, Temple) -> NewTemple = temple:fight(Temple), {reply, NewTemple, NewTemple}; So I try handle_info(trigger, _State) -> land:action(self(), {fight}), erlang:send_after(?INTERVAL, self(), trigger); but I get =ERROR REPORT==== 4-Dec-2016::19:00:35 === ** Generic server <0.400.0> terminating ** Last message in was trigger ** When Server state == {{dict,0,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}}}, []} ** Reason for termination == ** {function_clause,[{land,terminate, [{timeout,{gen_server,call,[<0.400.0>,{fight}]}}, {{dict,0,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[], [],[]}, {{[],[],[],[],[],[],[],[],[],[],[],[],[], [],[],[]}}}, []}], [{file,"src/land.erl"},{line,47}]} What am I missing here? -------------- next part -------------- An HTML attachment was scrubbed... URL: From james@REDACTED Mon Dec 5 06:29:43 2016 From: james@REDACTED (James Aimonetti) Date: Sun, 4 Dec 2016 21:29:43 -0800 Subject: [erlang-questions] Update Pid State on a Timer In-Reply-To: References: Message-ID: <87d1h70wqg.fsf@2600hz.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 http://erlang.org/doc/man/gen_server.html#Module:handle_info-2 Compare the expected return values there to what you return in your clause... qp writes: > I am trying to update my process's state on a 3 second timer. > -define(INTERVAL, 3000). > > start_link() -> > gen_server:start_link(?MODULE, [], []). > > action(Pid, Action) -> > gen_server:call(Pid, Action). > > init([]) -> > erlang:send_after(?INTERVAL, self(), trigger), > {ok, temple:new()}. > > what I want to do is call this > > handle_call({fight}, _From, Temple) -> > NewTemple = temple:fight(Temple), > {reply, NewTemple, NewTemple}; > > So I try > > handle_info(trigger, _State) -> > land:action(self(), {fight}), > erlang:send_after(?INTERVAL, self(), trigger); > > but I get > > =ERROR REPORT==== 4-Dec-2016::19:00:35 === > ** Generic server <0.400.0> terminating > ** Last message in was trigger > ** When Server state == {{dict,0,16,16,8,80,48, > {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], > []}, > {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], > []}}}, > []} > ** Reason for termination == > ** {function_clause,[{land,terminate, > [{timeout,{gen_server,call,[<0.400.0>,{fight}]}}, > {{dict,0,16,16,8,80,48, > {[],[],[],[],[],[],[],[],[],[],[],[],[],[], > [],[]}, > {{[],[],[],[],[],[],[],[],[],[],[],[],[], > [],[],[]}}}, > []}], > [{file,"src/land.erl"},{line,47}]} > > What am I missing here? - -- James Aimonetti Lead Systems Architect "If Dialyzer don't care, I don't care" 2600HzPDX | http://2600hz.com sip:james@REDACTED tel:415.886.7905 irc:mc_ @ freenode -----BEGIN PGP SIGNATURE----- iQEcBAEBCAAGBQJYRPtHAAoJENTKa+JPXCVgE08H/3m3QPHveD/tJjJ9qzirF76g aFFEGGKvAPjUJBTuoTZLTd8cLe9RdB9KaTObLpTwdr+tnXwTg7K1JLHh5qJGakGQ RXH2hkWG56Mn5SCbVTPvQIUxEXM48jsi9WZZUqadez+FALxDxkAH1h07wCfU5KJS ZVQqSh1JIuGBSV6iA8SZhWL9p04axIdwl9TUXFIDQsTLmGfblvuzIr9Cv1GpuaDe tZ0JyyED/3W/2qIlhfxYTwpip5S8WayR3rznauJwDnoTEygkI3rzw6hSaYpo4zU6 t+a2RXb8KMixI06+PRBOXIpRee/OQx7vbjAtRUxYyIYC4krp7f/Gc5f5W8xrJYk= =83kE -----END PGP SIGNATURE----- From zxq9@REDACTED Mon Dec 5 06:30:05 2016 From: zxq9@REDACTED (zxq9) Date: Mon, 05 Dec 2016 14:30:05 +0900 Subject: [erlang-questions] Update Pid State on a Timer In-Reply-To: References: Message-ID: <2672614.hSuflUOdZj@changa> On 2016?12?4? ??? 20:43:08 qp wrote: > I am trying to update my process's state on a 3 second timer. > > -define(INTERVAL, 3000). > > start_link() -> > gen_server:start_link(?MODULE, [], []). > > action(Pid, Action) -> > gen_server:call(Pid, Action). > > init([]) -> > erlang:send_after(?INTERVAL, self(), trigger), > {ok, temple:new()}. > > what I want to do is call this > > handle_call({fight}, _From, Temple) -> > NewTemple = temple:fight(Temple), > {reply, NewTemple, NewTemple}; > > So I try > > handle_info(trigger, _State) -> > land:action(self(), {fight}), > erlang:send_after(?INTERVAL, self(), trigger); > > but I get > > =ERROR REPORT==== 4-Dec-2016::19:00:35 === > ** Generic server <0.400.0> terminating > ** Last message in was trigger > ** When Server state == {{dict,0,16,16,8,80,48, > {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], > []}, > {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], > []}}}, > []} > ** Reason for termination == > ** {function_clause,[{land,terminate, > [{timeout,{gen_server,call,[<0.400.0>,{fight}]}}, > {{dict,0,16,16,8,80,48, > {[],[],[],[],[],[],[],[],[],[],[],[],[],[], > [],[]}, > {{[],[],[],[],[],[],[],[],[],[],[],[],[], > [],[],[]}}}, > []}], > [{file,"src/land.erl"},{line,47}]} > > > What am I missing here? I assume the module name is `land`. If that is the case the handler for 'trigger' will call land:action(self(), {fight}) which puts us inside action/2 above, which tries to make a call to self(). Why would making a call to self() always fail with a timeout? Think about this carefully -- it is the most basic form of deadlock you can create. -Craig From james@REDACTED Mon Dec 5 06:39:17 2016 From: james@REDACTED (James Aimonetti) Date: Sun, 4 Dec 2016 21:39:17 -0800 Subject: [erlang-questions] Update Pid State on a Timer In-Reply-To: <2672614.hSuflUOdZj@changa> References: <2672614.hSuflUOdZj@changa> Message-ID: <87bmwr0wai.fsf@2600hz.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 zxq9 writes: > On 2016?12?4? ??? 20:43:08 qp wrote: >> I am trying to update my process's state on a 3 second timer. >> >> -define(INTERVAL, 3000). >> >> start_link() -> >> gen_server:start_link(?MODULE, [], []). >> >> action(Pid, Action) -> >> gen_server:call(Pid, Action). >> >> init([]) -> >> erlang:send_after(?INTERVAL, self(), trigger), >> {ok, temple:new()}. >> >> what I want to do is call this >> >> handle_call({fight}, _From, Temple) -> >> NewTemple = temple:fight(Temple), >> {reply, NewTemple, NewTemple}; >> >> So I try >> >> handle_info(trigger, _State) -> >> land:action(self(), {fight}), >> erlang:send_after(?INTERVAL, self(), trigger); >> >> but I get >> >> =ERROR REPORT==== 4-Dec-2016::19:00:35 === >> ** Generic server <0.400.0> terminating >> ** Last message in was trigger >> ** When Server state == {{dict,0,16,16,8,80,48, >> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], >> []}, >> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], >> []}}}, >> []} >> ** Reason for termination == >> ** {function_clause,[{land,terminate, >> [{timeout,{gen_server,call,[<0.400.0>,{fight}]}}, >> {{dict,0,16,16,8,80,48, >> {[],[],[],[],[],[],[],[],[],[],[],[],[],[], >> [],[]}, >> {{[],[],[],[],[],[],[],[],[],[],[],[],[], >> [],[],[]}}}, >> []}], >> [{file,"src/land.erl"},{line,47}]} >> >> >> What am I missing here? > > I assume the module name is `land`. > > If that is the case the handler for 'trigger' will call > > land:action(self(), {fight}) > > which puts us inside action/2 above, which tries to make a call to self(). > > Why would making a call to self() always fail with a timeout? Think about > this carefully -- it is the most basic form of deadlock you can create. > > -Craig This is more directly related to the problem; tired eyes on my part just saw the return value - that will get you once you've ruminated on the above reply :) - -- James Aimonetti Lead Systems Architect "If Dialyzer don't care, I don't care" 2600HzPDX | http://2600hz.com sip:james@REDACTED tel:415.886.7905 irc:mc_ @ freenode -----BEGIN PGP SIGNATURE----- iQEcBAEBCAAGBQJYRP2FAAoJENTKa+JPXCVgZPEH/00KZao8ZbtDBpujkvpk+zew SMl75rb28f058xY0jQCpaS67/+02iGEShvZ4Sz9voMyc47ti7PN/2BYDIuLXX8lf Z2XgDxeIJeiasxqZMMDkMKg+Fiep5GbW7cf2H9tyri9xTXgb1mEGry9hkaFFR4/h jX1PLZo8Z1Qg6ec41w5dWwSe47NvdX+uNecxruwl/HLv+54fRnOrh5TvNULrfFpp zkWAojLb7O65Bn4EbtRj8Iw9wLxqaIl0Hyt2UIsPtpod7SSDBIbnfui4SR6j5bQ0 xEINW8MeKrEnTJ/yeyD9faIYUN8I9+868E2VKBC72ZSDLccwfJA5+Q50EU1h8sw= =7jQ3 -----END PGP SIGNATURE----- From zxq9@REDACTED Mon Dec 5 06:49:18 2016 From: zxq9@REDACTED (zxq9) Date: Mon, 05 Dec 2016 14:49:18 +0900 Subject: [erlang-questions] Update Pid State on a Timer In-Reply-To: <87bmwr0wai.fsf@2600hz.com> References: <2672614.hSuflUOdZj@changa> <87bmwr0wai.fsf@2600hz.com> Message-ID: <1527875.IxfcnmflMH@changa> On 2016?12?4? ??? 21:39:17 you wrote: > > This is more directly related to the problem; tired eyes on my part just > saw the return value - that will get you once you've ruminated on the > above reply :) I think it might be helpful for him to just walk procedurally through what he intends to have happen to work out the early wonkiness of the desired steps, and then start getting more messagy with things. New paradigms can be mind melting. As for finding Waldo... this message caught me at one of those moments where I am knee-deep in the final rush through refactoring the inner protocols in a medium-sized (but very busy) system, so I'm especially tuned to spot the deadlock just now. -Craig From max.lapshin@REDACTED Mon Dec 5 08:49:40 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 5 Dec 2016 10:49:40 +0300 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: <1534158.vnu8HodBQW@changa> References: <20161205021355.GI684@ferdmbp.local> <1534158.vnu8HodBQW@changa> Message-ID: Yes, you need also to check stability of your video stream timestamps. They must be aligned with realtime or player will fetch data slower and slower. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Mon Dec 5 09:41:23 2016 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Mon, 5 Dec 2016 09:41:23 +0100 Subject: [erlang-questions] Update Pid State on a Timer In-Reply-To: References: Message-ID: In line 47 you do a gen_server call, possibly to your current process, which causes a deadlock and hence the timeout error. Change it to asynchronous gen_server:cast or a direct function call to break the deadlock or don't call gen_server functions from a gen_server in the same module. 2016-12-05 2:43 GMT+01:00 qp : > > I am trying to update my process's state on a 3 second timer. > > -define(INTERVAL, 3000). > > start_link() -> > gen_server:start_link(?MODULE, [], []). > > action(Pid, Action) -> > gen_server:call(Pid, Action). > > init([]) -> > erlang:send_after(?INTERVAL, self(), trigger), > {ok, temple:new()}. > > what I want to do is call this > > handle_call({fight}, _From, Temple) -> > NewTemple = temple:fight(Temple), > {reply, NewTemple, NewTemple}; > > So I try > > handle_info(trigger, _State) -> > land:action(self(), {fight}), > erlang:send_after(?INTERVAL, self(), trigger); > > but I get > > =ERROR REPORT==== 4-Dec-2016::19:00:35 === > ** Generic server <0.400.0> terminating > ** Last message in was trigger > ** When Server state == {{dict,0,16,16,8,80,48, > {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], > []}, > {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], > []}}}, > []} > ** Reason for termination == > ** {function_clause,[{land,terminate, > [{timeout,{gen_server,call,[<0.400.0>,{fight}]}}, > {{dict,0,16,16,8,80,48, > {[],[],[],[],[],[],[],[],[],[],[],[],[],[], > [],[]}, > {{[],[],[],[],[],[],[],[],[],[],[],[],[], > [],[],[]}}}, > []}], > [{file,"src/land.erl"},{line,47}]} > > > What am I missing here? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me@REDACTED Mon Dec 5 14:10:21 2016 From: me@REDACTED (Matthew Shapiro) Date: Mon, 5 Dec 2016 08:10:21 -0500 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: <20161205021355.GI684@ferdmbp.local> References: <504E33D7-4029-4C54-B86E-B7882279170D@gmail.com> <20161205021355.GI684@ferdmbp.local> Message-ID: That's super interesting information, and once the port's buffer fills up once I can totally see now how it could get into cascading performance issues, since once the message queue is backed up with A/V data messages from the publisher's process every gen_tcp:send/3 call now has to do a selective receive past all of them to find the inet_reply message. Thanks! On Sun, Dec 4, 2016 at 9:13 PM, Fred Hebert wrote: > On 12/04, Matthew Shapiro wrote: > >> Yeah I think I can safely say this is something to do with how VLC is >> functioning. Even remotely I'm getting stuck in send, though not as bad, >> and looking at what's going on in the difference between VLC and Ffmpeg I >> believe my timestamps are being unexpected and both are reacting weirdly >> too it, with VLC trying to slow the feed down for whatever reason and that >> may be causing it to be slow to do a TCP read. Shot in the dark but at >> least I know now that it's not an BEAM issue (and my time on this hasn't >> been to waste). >> >> So sorry for the thread when Erlang/Beam isn't at fault in the end :) >> >> > It is and it is not. One of the interesting things with the TCP stack is > that it sends a signal through ports, and if the send queue is full, the > whole thing blocks; it however blocks with live data in there, and there's > no good way to give it up. > > The reason is blocks that the TCP work is done over a port program, > meaning message passing takes place: > > Erlang Process TCP port program > | | > |----------- Data ------------->| > | | > |<--- {inet_reply,S,Status} ----| > > This is roughly what takes place within the thing. This is done through > calling erlang:port_command/2-3. The thing though is that if the buffer of > the port program is full, you get descheduled until there's place. > > There's a cheat though, one we at Heroku have used in Logplex. The trick > is to bypass the stack in critical 'must never block' circumstances and > call port_command yourself, but by passing in the option list '[nosuspend]'. > > When you do this and the port program representing the TCP socket is busy, > you'll get a 'false' return value indicating it failed because it was full > (you get 'true' when it works): > > case erlang:port_command(Socket, Data, [nosuspend]) of > false -> > handle_busy(...); > true -> > receive > {inet_reply, Socket, Status} -> % Status == send return > Status > end > end > > This will let you do things like figure out the consumer side (in this > case VLC) is too slow, and possibly give up sending a few frames and send > newer ones later. > > Control flow is fun! > > Regards, > Fred. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Mon Dec 5 14:56:17 2016 From: erlang@REDACTED (Stefan Marr) Date: Mon, 5 Dec 2016 14:56:17 +0100 Subject: [erlang-questions] [MoreVMs'17] Call for Contributions: 1st Workshop on Modern Language Runtimes, Ecosystems, and VMs at 2017 Message-ID: <73549EDC-D09F-4AED-B742-2450DF366A33@stefan-marr.de> An HTML attachment was scrubbed... URL: From community-manager@REDACTED Mon Dec 5 16:00:11 2016 From: community-manager@REDACTED (Bruce Yinhe) Date: Mon, 5 Dec 2016 16:00:11 +0100 Subject: [erlang-questions] Erlang.org website source code now on Github Message-ID: Hello all We have just released the source code for erlang.org website on Github. https://github.com/erlang/erlang-org The website is written in Cowboy, ErlyDTL and sumo_db. It is licensed under Apache License 2.0. You can follow the instructions to set up. The erlang.org website is developed and maintained by Industrial Erlang User Group in collaboration with Erlang/OTP team at Ericsson. Happy hacking! Bruce Yinhe | Community Manager Industrial Erlang User Group ErlangCentral.org | @ErlangCentral -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Mon Dec 5 16:07:09 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 5 Dec 2016 18:07:09 +0300 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: References: <504E33D7-4029-4C54-B86E-B7882279170D@gmail.com> <20161205021355.GI684@ferdmbp.local> Message-ID: > once the message queue is backed up with A/V data messages you should not allow this situation. Monitor your output processes, drop frames when more than 100 in queue. Also as far as I remember, erlang has mark in the message queue: selective receive will not step below this mark. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilya.khaprov@REDACTED Mon Dec 5 16:13:45 2016 From: ilya.khaprov@REDACTED (Ilya Khaprov) Date: Mon, 5 Dec 2016 15:13:45 +0000 Subject: [erlang-questions] Erlang.org website source code now on Github In-Reply-To: References: Message-ID: Hi Bruce, Congratulations! Question: why OTP 17.5? Regards, Ilya From: Bruce Yinhe Sent: Monday, December 5, 2016 06:00 PM To: Erlang Questions Subject: [erlang-questions] Erlang.org website source code now on Github Hello all We have just released the source code for erlang.org website on Github. https://github.com/erlang/erlang-org The website is written in Cowboy, ErlyDTL and sumo_db. It is licensed under Apache License 2.0. You can follow the instructions to set up. The erlang.org website is developed and maintained by Industrial Erlang User Group in collaboration with Erlang/OTP team at Ericsson. Happy hacking! Bruce Yinhe | Community Manager Industrial Erlang User Group ErlangCentral.org | @ErlangCentral -------------- next part -------------- An HTML attachment was scrubbed... URL: From marc@REDACTED Mon Dec 5 16:14:02 2016 From: marc@REDACTED (Marc Worrell) Date: Mon, 5 Dec 2016 16:14:02 +0100 Subject: [erlang-questions] [ANN] Zotonic 0.23 released Message-ID: <62F6DF61-787F-4F28-819B-533213F97EF4@worrell.nl> Zotonic is the Erlang Content Management System and framework. Zotonic version 0.23 has been released. This is a maintenance release with mainly bug fixes. For more information about Zotonic, see http://zotonic.com/ Short release notes: ? Added delete interval (and completely disabling deleting) to files stored on S3 (#1493). ? Added search rank weight configuration parameter (#1538). ? Added editing of embedded collections in admin (#1520). ? Added support for custom Google Analytics parameters (#1518). ? Fixed consistency of log messages and log metadata (#1510). ? Fixed ACL checks in admin (#1514). ? Fixed embedding not allowed for non-admins (#1545). ? Fixed initialization error in mod_acl_user_groups (#1505). ? Fixed initial tab on connect modal (#1497). See the full release notes at http://docs.zotonic.com/en/latest/developer-guide/releasenotes/rel_0.23.0.html Kind regards, The Zotonic core team. From marc@REDACTED Mon Dec 5 16:29:04 2016 From: marc@REDACTED (Marc Worrell) Date: Mon, 5 Dec 2016 16:29:04 +0100 Subject: [erlang-questions] Some examples of Zotonic web sites Message-ID: <0ACCD80F-5238-4F10-B3DF-5FEC684D3FE7@worrell.nl> Hi, Maybe nice to see more examples of ?Erlang Powered? web sites. There are too many sites to list in this single email, but a couple of sites illustrate nicely the flexibility of the platform: Joodsmonument - In memory of the Dutch Jewish victims of WW2 https://www.joodsmonument.nl/ LearningStone - Collaboration for learning and courses https://learningstone.com/ Channel.me - Co-browsing for help desks and live chat http://channel.me/ Women on Waves - essential health care for women http://www.womenonwaves.org/ And then there are many, many, more sites. We especially are grateful for Driebit, Maximonster and Channel.me for their continued active development and support of Zotonic. Regards, Marc Worrell and the rest of the team. From me@REDACTED Mon Dec 5 16:36:32 2016 From: me@REDACTED (Matthew Shapiro) Date: Mon, 5 Dec 2016 10:36:32 -0500 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: References: <504E33D7-4029-4C54-B86E-B7882279170D@gmail.com> <20161205021355.GI684@ferdmbp.local> Message-ID: > you should not allow this situation. Yep, understood. I was more describing that because of Fred's response I have a feeling this was one of the symptoms of the performance I was seeing, in that if the port was busy it would compound my performance issues because I was letting the Erlang message queue just keep filling. On Mon, Dec 5, 2016 at 10:07 AM, Max Lapshin wrote: > > once the message queue is backed up with A/V data messages > > you should not allow this situation. > > Monitor your output processes, drop frames when more than 100 in queue. > > Also as far as I remember, erlang has mark in the message queue: > selective receive will not step below this mark. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Mon Dec 5 17:05:25 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 5 Dec 2016 19:05:25 +0300 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: References: <504E33D7-4029-4C54-B86E-B7882279170D@gmail.com> <20161205021355.GI684@ferdmbp.local> Message-ID: How much traffic do you send at the moment? -------------- next part -------------- An HTML attachment was scrubbed... URL: From me@REDACTED Mon Dec 5 18:17:06 2016 From: me@REDACTED (Matthew Shapiro) Date: Mon, 5 Dec 2016 12:17:06 -0500 Subject: [erlang-questions] Troubling gen_tcp.send/3 performance In-Reply-To: References: <504E33D7-4029-4C54-B86E-B7882279170D@gmail.com> <20161205021355.GI684@ferdmbp.local> Message-ID: Right now I"m sending 1 2500kbps video in and out. Unfortunately, there's something (non-Erlang) wrong with something I'm doing. The A/V data that is being relayed must be getting mis-interpreted by VLC somehow and VLC is trying to throttle down the TCP reads, which then causes my Erlang process to get stuck in waiting on a gen_tcp.send() call. So while I need to not do gen_tcp.send() and instead probably do port_commands and drop frames if things are backed up, at the end of the day things shouldn't be backed up like they are for a single 2500kbps video over local host and I am apparently not-understanding something very fundamental to either RTMP or video in general that's causing things to go haywire. At least from this thread I know it's not Erlang and instead something on my side (which truthfully I believed from the beginning, but at least I have proof now :) ). On Mon, Dec 5, 2016 at 11:05 AM, Max Lapshin wrote: > How much traffic do you send at the moment? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrallen1@REDACTED Mon Dec 5 18:42:03 2016 From: mrallen1@REDACTED (Mark Allen) Date: Mon, 5 Dec 2016 17:42:03 +0000 (UTC) Subject: [erlang-questions] [ANN] Basho move Lager to erlang-lager organization on Github In-Reply-To: References: Message-ID: <1673082147.6824986.1480959723671@mail.yahoo.com> Just to follow up on this:? Andrew Thompson and I are going to start hosting a monthly lager issue/PR triage meeting on freenode, in #lager every third Thursday of the month starting 19 January 2017 at 1600 US/Central time, 1400 US/Pacific time, 2200 UTC for about an hour or so. During that time, we will review open pulls, open issues and generally be available via IRC to answer questions. ?If you have questions about lager, or want to help, please join us. In the meantime, I hope to groom the current PRs/issues during the upcoming holidays so that we can enter January with a nice clean start for the new year. Many many thanks to Basho and Doug for making this happen!? Cheers, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From community-manager@REDACTED Mon Dec 5 18:58:26 2016 From: community-manager@REDACTED (Bruce Yinhe) Date: Mon, 5 Dec 2016 18:58:26 +0100 Subject: [erlang-questions] Erlang.org website source code now on Github In-Reply-To: References: Message-ID: Hi Erlang.org had been rewritten from erlang_web to cowboy. OTP 17.5 was used when the website revamp was developed. Regards Bruce Yinhe | Erlang Community Manager Industrial Erlang User Group ErlangCentral.org | @ErlangCentral On Mon, Dec 5, 2016 at 4:13 PM, Ilya Khaprov wrote: > Hi Bruce, > > > > Congratulations! Question: why OTP 17.5? > > > > Regards, > > Ilya > > > > *From: *Bruce Yinhe > *Sent: *Monday, December 5, 2016 06:00 PM > *To: *Erlang Questions > *Subject: *[erlang-questions] Erlang.org website source code now on Github > > > Hello all > > We have just released the source code for erlang.org website on Github. > > https://github.com/erlang/erlang-org > > The website is written in Cowboy, ErlyDTL and sumo_db. It is licensed > under Apache License 2.0. You can follow the instructions to set up. > > The erlang.org website is developed and maintained by Industrial Erlang > User Group in collaboration with Erlang/OTP team at Ericsson. > > Happy hacking! > > Bruce Yinhe | Community Manager > Industrial Erlang User Group > > ErlangCentral.org | @ErlangCentral > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthony@REDACTED Mon Dec 5 15:25:06 2016 From: anthony@REDACTED (=?UTF-8?B?QW50aMOYbnk=?=) Date: Mon, 5 Dec 2016 09:25:06 -0500 Subject: [erlang-questions] Update Pid State on a Timer In-Reply-To: References: Message-ID: Hi, thank you for your advice. I have removed the reference to handleInfo - that was a mistake to have that function in there. I found a timer.beam i my current directory from an older example and that is what caused the error! Fully working timer example is here: -module(counter). -compile(export_all). counter(State) -> receive {From, {add, Number}} -> NewNumber = State+Number, From ! {self(), NewNumber}, counter(NewNumber); {From, {resolve}} -> From ! {self(), State}, counter(0); terminate -> ok end. add(Pid, Number) -> Pid ! {self(), {add, Number}}, receive {Pid, Msg} -> Msg after 3000 -> timeout end. resolve(Pid) -> Pid ! {self(), {resolve}}, {ok, _Tref} = timer:apply_after(1000, ?MODULE, resolve, [Pid]), receive {Pid, Msg} -> Msg after 3000 -> timeout end. start() -> Pid = spawn(?MODULE, counter, [0]), {ok, _Tref} = timer:apply_after(1000, ?MODULE, resolve, [Pid]), Pid. On Mon, Dec 5, 2016 at 3:41 AM, Dmytro Lytovchenko < dmytro.lytovchenko@REDACTED> wrote: > In line 47 you do a gen_server call, possibly to your current process, > which causes a deadlock and hence the timeout error. Change it to > asynchronous gen_server:cast or a direct function call to break the > deadlock or don't call gen_server functions from a gen_server in the same > module. > > 2016-12-05 2:43 GMT+01:00 qp : > >> >> I am trying to update my process's state on a 3 second timer. >> >> -define(INTERVAL, 3000). >> >> start_link() -> >> gen_server:start_link(?MODULE, [], []). >> >> action(Pid, Action) -> >> gen_server:call(Pid, Action). >> >> init([]) -> >> erlang:send_after(?INTERVAL, self(), trigger), >> {ok, temple:new()}. >> >> what I want to do is call this >> >> handle_call({fight}, _From, Temple) -> >> NewTemple = temple:fight(Temple), >> {reply, NewTemple, NewTemple}; >> >> So I try >> >> handle_info(trigger, _State) -> >> land:action(self(), {fight}), >> erlang:send_after(?INTERVAL, self(), trigger); >> >> but I get >> >> =ERROR REPORT==== 4-Dec-2016::19:00:35 === >> ** Generic server <0.400.0> terminating >> ** Last message in was trigger >> ** When Server state == {{dict,0,16,16,8,80,48, >> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], >> []}, >> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], >> []}}}, >> []} >> ** Reason for termination == >> ** {function_clause,[{land,terminate, >> [{timeout,{gen_server,call,[<0.400.0>,{fight}]}}, >> {{dict,0,16,16,8,80,48, >> {[],[],[],[],[],[],[],[],[],[],[],[],[],[], >> [],[]}, >> {{[],[],[],[],[],[],[],[],[],[],[],[],[], >> [],[],[]}}}, >> []}], >> [{file,"src/land.erl"},{line,47}]} >> >> >> What am I missing here? >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue Dec 6 03:15:59 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 6 Dec 2016 05:15:59 +0300 Subject: [erlang-questions] Erlang for an audio streaming server project? In-Reply-To: References: Message-ID: It is very unclearly what you want to do: take raw audio and encode it to mp3/aac (it is insane to do this in java, because it is unsuitable for such task) or you want to take mp3 and stream it via shoutcast. Encode raw audio (where do you take it from?) to mp3 is a task for any C library that is very easily embeddable into erlang. Streaming mp3 to shoutcast is very, very, extremely easy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From quantumpotato@REDACTED Tue Dec 6 00:16:16 2016 From: quantumpotato@REDACTED (qp) Date: Mon, 5 Dec 2016 18:16:16 -0500 Subject: [erlang-questions] Idiomatically handling multiple validation checks Message-ID: Hi, I am new to Erlang and wrote this code for validating that the Name, Action & Target atoms passed in to validRequest are all valid. validRequest(valid, valid, Target) -> case validName(Target) of true -> true; false -> false end; validRequest(valid, Action, Target) -> case validAction(Action) of true -> validRequest(valid, valid, Target); false -> false end; validRequest(Name, Action, Target) -> case validName(Name) of true -> validRequest(valid, Action, Target); false -> false end. I've since refactored into validRequest(Name, Action, Target) -> validName(Name) and validAction(Action) and validName(Target). I'm curious what a more idiomatic way of writing this would be? I've seen a mix of styles using booleans for return values and tuples, with errors listed. I don't think that's needed here but I'm just curious how you would handle multiple validation checks, especially if they were more complicated than this example. Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaotixpro@REDACTED Tue Dec 6 01:42:14 2016 From: kaotixpro@REDACTED (Edgar H) Date: Tue, 6 Dec 2016 01:42:14 +0100 Subject: [erlang-questions] Erlang for an audio streaming server project? In-Reply-To: References: Message-ID: Thanks a lot for the replies guys, really appreciate it! >From what you've told me so far I think have the following options... 1) Stream into Icecast directly from Java and forget about Erlang (I wouldn't want to take this one - It isn't challenging and won't learn anything new, but would get the job done). 2) Send raw data from Java to the mini-audio-streaming-server coded by me (or an existing HTTP server like cowboy) which implements an encoder developed by me or can I blindly send the data pre-encoded from Java using ffmpeg without checking which format is it? (If I know for sure the kind of content I'm streaming, should I just bother about the MIME-types?). 3) (If I understood Max's reply correctly) Develop the codecs I need in Erlang and then work with them to encode the raw data and stream it. At this point, with my current knowledge in codecs (null) and time for the project, it seems like an impossible goal to reach for me... Eric, the codecs I need are mainly mp3 and vorbis. 2016-12-05 16:14 GMT+01:00 Eric des Courtis : > I agree with Max ditch the Java it will just slow you down. What codecs do > you need anyhow? > > On Sun, Dec 4, 2016 at 2:20 PM, Max Lapshin wrote: > >> You will have very interesting deployment pain if you take java with you. >> >> It is not clear what for to waste time on it? >> >> There are 2-3 codecs and 3-4 transport payloads for audio. Just write all >> this in erlang, it is not hard. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From technion@REDACTED Tue Dec 6 06:56:59 2016 From: technion@REDACTED (Technion) Date: Tue, 6 Dec 2016 05:56:59 +0000 Subject: [erlang-questions] Idiomatically handling multiple validation checks In-Reply-To: References: Message-ID: Hi, Someone who's written more could well point out this is heresy but I find verify* functions very convenient to write if I make them an Erlang assertion. eg: isone(A)-> A = 1. istwo(A) -> A = 2. Compare that with a case A of.. for each function. Much more complex verifies still work very well with a binary pattern match. Then I can write a verify like this: verify(Data) -> try isone(Data), istwo(Data) % As many verify functions as needed here of _ -> true catch error:{badmatch, _} -> false end. ________________________________ From: erlang-questions-bounces@REDACTED on behalf of qp Sent: Tuesday, 6 December 2016 10:16:16 AM To: Erlang/OTP discussions Subject: [erlang-questions] Idiomatically handling multiple validation checks Hi, I am new to Erlang and wrote this code for validating that the Name, Action & Target atoms passed in to validRequest are all valid. validRequest(valid, valid, Target) -> case validName(Target) of true -> true; false -> false end; validRequest(valid, Action, Target) -> case validAction(Action) of true -> validRequest(valid, valid, Target); false -> false end; validRequest(Name, Action, Target) -> case validName(Name) of true -> validRequest(valid, Action, Target); false -> false end. I've since refactored into validRequest(Name, Action, Target) -> validName(Name) and validAction(Action) and validName(Target). I'm curious what a more idiomatic way of writing this would be? I've seen a mix of styles using booleans for return values and tuples, with errors listed. I don't think that's needed here but I'm just curious how you would handle multiple validation checks, especially if they were more complicated than this example. Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Tue Dec 6 07:57:03 2016 From: zxq9@REDACTED (zxq9) Date: Tue, 06 Dec 2016 15:57:03 +0900 Subject: [erlang-questions] Idiomatically handling multiple validation checks In-Reply-To: References: Message-ID: <2480782.vzNZox2jzC@changa> On 2016?12?5? ??? 18:16:16 qp wrote: > Hi, I am new to Erlang and wrote this code for validating that the Name, > Action & Target atoms passed in to validRequest are all valid. > > validRequest(valid, valid, Target) -> > case validName(Target) of > true -> true; > false -> false > end; > validRequest(valid, Action, Target) -> > case validAction(Action) of > true -> validRequest(valid, valid, Target); > false -> false > end; > validRequest(Name, Action, Target) -> > case validName(Name) of > true -> validRequest(valid, Action, Target); > false -> false > end. > > I've since refactored into > > validRequest(Name, Action, Target) -> > validName(Name) and validAction(Action) and validName(Target). > > I'm curious what a more idiomatic way of writing this would be? I've seen a > mix of styles using booleans for return values and tuples, with errors > listed. I don't think that's needed here but I'm just curious how you would > handle multiple validation checks, especially if they were more complicated > than this example. Thank you! We don't often actually do many validation checks, more like a circumstance check. Many times you will have a case where you want to take some specific action only if *everything* is good to go and some other action if *anything* is not good: validate_request(valid, valid, valid) -> true; validate_request(_, _, _) -> false. But this is sort of weird, because normally I wouldn't want a boolean, I would want to be doing something already (also... notice that the function name is not wonkyCamelCase -- use underscores between parts of a word in atoms, modules and function names; its sort of a thing around here): validate_request(valid, valid, valid) -> do_whatever(); validate_request(_, _, _) -> {error, invalid!}. But why are we resolving things to the atom 'valid' in the first place? This makes me think something else is going on way back in the code that is unidiomatic to start with. Consider if maybe you have a process, and you want that process to be doing something specific once a certain point is reached. Let's say you have the state of that process stored in a record. Maybe it is parsing a stream of bytes that is supposed to be a chat message and you need to pull out the channel, user, and timestamp before taking some action: -record(message, {channel = none :: none | string(), user = none :: none | string(), timestamp = none :: none | timestamp(), message = none :: none | string()}). The parts after the '::' are the type specification. In this case we are saying that the 'channel' attribute could be either the atom 'none' or a string -- but that it will default to being 'none' if we just create the record without defining the 'channel' attribute. So we are receiving some data in a stream and calling a parse/2 function over it every time we receive another segment. Which part of the message we are in might have to do with what parts of the record are still 'none': parse(Segment, Chat = #message{channel = none}) -> case parse_channel(Segment) of {ok, Channel, Remainder} -> parse(Remainder, Chat#message{channel = Channel}); {not_enough_data} -> {incomplete, Segment, Chat} end; parse(segment, Chat = #message{user = none}) -> case parse_user(Segment) of {ok, User, Remainder} -> parse(Remainder, Chat#message{user = User}); {not_enough_data} -> {incomplete, Segment, Chat} end; parse(Segment, Chat = #message{timestamp = none}) -> % ...you get the idea... That could be naked arguments, it could be in a record, it could be any form where you can match to make a decision. Notice how utterly the same the code above is. It is common to see code sort of like this, but pretty soon you'll figure out how to refactor the kind of thing above into something more concise. But concise code wasn't the subject of your question, idiomatic situation checking was -- and the code above is doing more like what we would be used to seeing. If you handle you data an unidiomatic way, or enter your functions in an unidiomatic way you can often find yourself forced into writing some pretty unidiomatic function bodies -- and that sucks. It happens to me all the time that long after starting a project I realize something fundamental about the nature of the problem I'm solving and in one fell swoop a hundred or more lines of frustrating, unidiomatic code just becomes obsolete. Poof! You'll probably have this experience at least a few times as you play with Erlang a bit more. Speaking of unidiomatic entries to functions... This reminds me of a question on SO a while back that was sort of like this, where I suspected the guy had a bit of an X-Y problem, in that he was asking how to make a function better (un-nest some nested cases, specifically) that really seemed to call for refactoring further back in the program before the call in question is made: http://stackoverflow.com/questions/28673437/erlang-nested-cases/28673997#28673997 Blah blah. I hope I gave you some ideas. It is good that you are not letting yourself just be satisfied with annoyingly unidiomatic code! Have fun! -Craig From zxq9@REDACTED Tue Dec 6 08:00:08 2016 From: zxq9@REDACTED (zxq9) Date: Tue, 06 Dec 2016 16:00:08 +0900 Subject: [erlang-questions] Idiomatically handling multiple validation checks In-Reply-To: <2480782.vzNZox2jzC@changa> References: <2480782.vzNZox2jzC@changa> Message-ID: <5024545.HvMiPNXfGd@changa> On 2016?12?6? ??? 15:57:03 zxq9 wrote: > validate_request(valid, valid, valid) -> > do_whatever(); > validate_request(_, _, _) -> > {error, invalid!}. I have to appologize for the last line here (and any other typos). The naked "!" is not something you would be able to include in an atom name (quoting aside). That was just a typo. -Craig From dmytro.lytovchenko@REDACTED Tue Dec 6 10:13:34 2016 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Tue, 6 Dec 2016 10:13:34 +0100 Subject: [erlang-questions] Idiomatically handling multiple validation checks In-Reply-To: References: Message-ID: For things like web form or input validation and transformation, it is often a good idea to make a chain of nested calls. Imagine your user submitted a form which you store in Data = #{login="", password=""}, you can do something like this: try create_session(check_password(check_login(Data))) catch throw:{validation_error, E} -> my_report_error(E) end,... Here each function (check_login, check_password, create_session) takes result of the previous function, it can be that same Data, or Data paired with some intermediate results if you wish. To signal an error you use erlang:throw({validation_error, "my error"}) and it will be caught in the catch clause. 2016-12-06 0:16 GMT+01:00 qp : > Hi, I am new to Erlang and wrote this code for validating that the Name, > Action & Target atoms passed in to validRequest are all valid. > > validRequest(valid, valid, Target) -> > case validName(Target) of > true -> true; > false -> false > end; > validRequest(valid, Action, Target) -> > case validAction(Action) of > true -> validRequest(valid, valid, Target); > false -> false > end; > validRequest(Name, Action, Target) -> > case validName(Name) of > true -> validRequest(valid, Action, Target); > false -> false > end. > > I've since refactored into > > validRequest(Name, Action, Target) -> > validName(Name) and validAction(Action) and validName(Target). > > I'm curious what a more idiomatic way of writing this would be? I've seen > a mix of styles using booleans for return values and tuples, with errors > listed. I don't think that's needed here but I'm just curious how you would > handle multiple validation checks, especially if they were more complicated > than this example. Thank you! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Tue Dec 6 10:25:10 2016 From: zxq9@REDACTED (zxq9) Date: Tue, 06 Dec 2016 18:25:10 +0900 Subject: [erlang-questions] Idiomatically handling multiple validation checks In-Reply-To: References: Message-ID: <211577939.sQYGRkTBiL@changa> On 2016?12?6? ??? 10:13:34 Dmytro Lytovchenko wrote: > For things like web form or input validation and transformation, it is > often a good idea to make a chain of nested calls. > > Imagine your user submitted a form which you store in Data = #{login="", > password=""}, you can do something like this: > > try > create_session(check_password(check_login(Data))) > catch throw:{validation_error, E} -> my_report_error(E) > end,... > > > Here each function (check_login, check_password, create_session) takes > result of the previous function, it can be that same Data, or Data paired > with some intermediate results if you wish. To signal an error you use > erlang:throw({validation_error, "my error"}) and it will be caught in the > catch clause. That's one way to do it... but most code I've seen that makes heavy use of `throw` has insane structural issues (either actual bugs, or mental grenades with C++ style confusing execution). Basically, what I'm getting at is that `throw` is a keyword oft used in prayer and curse in Mordor. Why not return `{error, Reason}` back up the chain of calls? Then the top-level caller can choose whether to crash on an `{ok, Value}` or `{error, Reason}` assertion match, or engage in their own insanity with `throw` .. `catch` and other nonsense. -Craig PS: Yes, I generally regard use of `throw` as "nonsense" and won't apologize for that characterization of this 99% horrible curseword, no matter how much I may be hurting its feelings. From dmytro.lytovchenko@REDACTED Tue Dec 6 10:34:21 2016 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Tue, 6 Dec 2016 10:34:21 +0100 Subject: [erlang-questions] Idiomatically handling multiple validation checks In-Reply-To: <211577939.sQYGRkTBiL@changa> References: <211577939.sQYGRkTBiL@changa> Message-ID: 2016-12-06 10:25 GMT+01:00 zxq9 : > On 2016?12?6? ??? 10:13:34 Dmytro Lytovchenko wrote: > > For things like web form or input validation and transformation, it is > > often a good idea to make a chain of nested calls. > > > > Imagine your user submitted a form which you store in Data = #{login="", > > password=""}, you can do something like this: > > > > try > > create_session(check_password(check_login(Data))) > > catch throw:{validation_error, E} -> my_report_error(E) > > end,... > > > > > > Here each function (check_login, check_password, create_session) takes > > result of the previous function, it can be that same Data, or Data paired > > with some intermediate results if you wish. To signal an error you use > > erlang:throw({validation_error, "my error"}) and it will be caught in > the > > catch clause. > > That's one way to do it... but most code I've seen that makes heavy use > of `throw` has insane structural issues (either actual bugs, or mental > grenades with C++ style confusing execution). > > Basically, what I'm getting at is that `throw` is a keyword oft used in > prayer and curse in Mordor. > Any best practice can be corrupted and written in a horrible way, it's a matter of self discipline and good reviewing in the team. Best idea is to not allow throw propagate more than 1 logical level up. We chain-call the handlers, and the handlers are allowed to throw, we catch it. Anything that handlers can call and that can throw, should be caught by them and not propagate wildly onto higher levels. > > Why not return `{error, Reason}` back up the chain of calls? Then the > top-level caller can choose whether to crash on an `{ok, Value}` or > `{error, Reason}` assertion match, or engage in their own insanity with > `throw` .. `catch` and other nonsense Absolutely it can be done Haskell/ML way with values wrapped as {ok,_} or {error, _} and returned. This will make beautiful flow and will triple the chain runner function size, or you will have to add an {error, _} clause to each of your handlers. And exceptions still are going to happen, so those have to be caught and wrapped with {error, _} too. On the other hand throw was designed as a way for user to safely interrupt whatever was running and hop out of deep nested calls. Using it wisely and keeping it simple may produce clean short code. For simplicity i kept my example short. -------------- next part -------------- An HTML attachment was scrubbed... URL: From khitai.pang@REDACTED Tue Dec 6 10:34:48 2016 From: khitai.pang@REDACTED (Khitai Pang) Date: Tue, 6 Dec 2016 09:34:48 +0000 Subject: [erlang-questions] processes spawned on remote nodes don't run Message-ID: Hi, I have the following function in user_default.erl: run_on_all_nodes() -> [spawn(Node, fun() -> io:format("This runs on ~p~n", [node()]) end) || Node <- nodes()]. When I run this function, processes can be spawned on remote nodes but it seems that they don't run: (myapp@REDACTED)1> nodes(). [myapp@REDACTED,rabbit@REDACTED,rabbit@REDACTED] (myapp@REDACTED)2> run_on_all_nodes(). [<27616.12336.0>,<27612.15341.39>,<27615.22399.50>] If I manually run the spawn command in the attached shell, it all works fine: (myapp@REDACTED)3> [spawn(Node, fun() -> io:format("This runs on ~p~n", [node()]) end) || Node <- nodes()]. This runs on myapp@REDACTED This runs on rabbit@REDACTED This runs on rabbit@REDACTED [<27616.12339.0>,<27612.15569.39>,<27615.22640.50>] Any idea? Thanks Khitai From roe.adrian@REDACTED Tue Dec 6 10:52:42 2016 From: roe.adrian@REDACTED (Adrian Roe) Date: Tue, 6 Dec 2016 09:52:42 +0000 Subject: [erlang-questions] processes spawned on remote nodes don't run In-Reply-To: References: Message-ID: This confused me for a while as well? To quote http://erlang.org/doc/man/erlang.html group_leader() -> pid() Returns the process identifier of the group leader for the process evaluating the function. Every process is a member of some process group and all groups have a group leader. All I/O from the group is channeled to the group leader. When a new process is spawned, it gets the same group leader as the spawning process. Initially, at system startup, init is both its own group leader and the group leader of all processes. There?s a helpful post here http://erlang.org/pipermail/erlang-questions/2008-April/034573.html that talks about exactly what you are seeing. If you change the io:format("This runs on ~p~n", [node()]) to io:format(user, "This runs on ~p~n", [node()]) I strongly suspect you will see the output you were expecting? Adrian > On 6 Dec 2016, at 09:34, Khitai Pang wrote: > > Hi, > > I have the following function in user_default.erl: > > run_on_all_nodes() -> > [spawn(Node, fun() -> > io:format("This runs on ~p~n", [node()]) > end) || Node <- nodes()]. > > When I run this function, processes can be spawned on remote nodes but > it seems that they don't run: > > (myapp@REDACTED)1> nodes(). > [myapp@REDACTED,rabbit@REDACTED,rabbit@REDACTED] > > (myapp@REDACTED)2> run_on_all_nodes(). > [<27616.12336.0>,<27612.15341.39>,<27615.22399.50>] > > If I manually run the spawn command in the attached shell, it all works > fine: > > (myapp@REDACTED)3> [spawn(Node, fun() -> io:format("This runs on ~p~n", > [node()]) end) || Node <- nodes()]. > This runs on myapp@REDACTED > This runs on rabbit@REDACTED > This runs on rabbit@REDACTED > [<27616.12339.0>,<27612.15569.39>,<27615.22640.50>] > > > Any idea? > > > Thanks > Khitai > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From silver.surfertab@REDACTED Tue Dec 6 10:57:25 2016 From: silver.surfertab@REDACTED (Silver Surfer) Date: Tue, 6 Dec 2016 15:27:25 +0530 Subject: [erlang-questions] Erlang C-nodes crashed randomly with double free, memory corruption malloc messege In-Reply-To: <4082a7b9-c0ec-535c-4f17-860b46f48c88@ericsson.com> References: <4082a7b9-c0ec-535c-4f17-860b46f48c88@ericsson.com> Message-ID: Yes, I have tried valgrind; with valgrind the problem does not come. On Wed, Nov 16, 2016 at 6:49 PM, Sverker Eriksson < sverker.eriksson@REDACTED> wrote: > Have you tried to run your c-node under valgrind. > > http://valgrind.org/. > > > /Sverker > > > > On 11/15/2016 08:32 PM, Silver Surfer wrote: > >> Hi, >> I have an application running on otp-19.0, where there is a single erlang >> node and two c-nodes; several processes from erlang node continuously send >> messages to the c-node servers. The c-node accepts a single connection and >> process all the received messages (ERL_MSG-> ERL_REG_SEND, ERL_SEND) in >> separate threads except ERL_TICK, ERL_ERROR etc. >> >> So in the c-node main() a connection is accepted and all the messages are >> received (erl_receive_msg), if the message is ERL_TICK, ERL_ERROR etc. >> they >> are dealt appropriately by freeing the memory, else if the message is >> ERL_MSG then a thread is created and the message is passed to the thread >> where it is processed and a reply is send using erl_send; this approach of >> handling message through thread is taken as some of the operation >> performed >> by the c-node takes considerable amount of time which is more than the >> tick >> time (is there a better way to do this?). >> >> Now out of the two c-nodes one is crashing randomly (10 times in 24Hrs, >> more or less); both the c-nodes follows same architecture, only the >> operations they perform are different. In most of the times the c-node >> just >> goes down without giving any error reason and in 2 or 3 cases it crashes >> because of double free or memory corruption printer by malloc, the trace >> back points to erl_receive_msg. >> >> Another point observed is that, in the thread after erl_free_compound, >> when >> we look at the allocated blocks using erl_eterm_statistics(allocated, >> freed), it is 0 most of the times but sometimes it is non zero value, i.e. >> 9, 18, etc. >> >> Any help is appreciated. >> >> Greg >> >> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From khitai.pang@REDACTED Tue Dec 6 11:35:23 2016 From: khitai.pang@REDACTED (Khitai Pang) Date: Tue, 6 Dec 2016 10:35:23 +0000 Subject: [erlang-questions] processes spawned on remote nodes don't run In-Reply-To: References: Message-ID: Hi Adrian, Thank you for the information but unfortunately it's not the cause of my problem. If I add 'user' io:format(): (myapp@REDACTED)6> run_on_all_nodes(). [<27616.12396.0>,<27612.19603.39>,<27615.48.51>] (myapp@REDACTED)7> [spawn(Node, fun() -> io:format(user, "This runs on ~p~n", [node()]) end) || Node <- nodes()]. [<27616.12397.0>,<27612.19600.39>,<27615.77.51>] (myapp@REDACTED)8> [spawn(Node, fun() -> io:format("This runs on ~p~n", [node()]) end) || Node <- nodes()]. This runs on myapp@REDACTED This runs on rabbit@REDACTED This runs on rabbit@REDACTED [<27616.12398.0>,<27612.19631.39>,<27615.89.51>] It doesn't seem to be I/O channeling problem, it seems that the spawned processes just don't run... Thanks khitai On 2016/12/6 17:52, Adrian Roe wrote: This confused me for a while as well? To quote http://erlang.org/doc/man/erlang.html group_leader() -> pid() Returns the process identifier of the group leader for the process evaluating the function. Every process is a member of some process group and all groups have a group leader. All I/O from the group is channeled to the group leader. When a new process is spawned, it gets the same group leader as the spawning process. Initially, at system startup, init is both its own group leader and the group leader of all processes. There?s a helpful post here http://erlang.org/pipermail/erlang-questions/2008-April/034573.html that talks about exactly what you are seeing. If you change the io:format("This runs on ~p~n", [node()]) to io:format(user, "This runs on ~p~n", [node()]) I strongly suspect you will see the output you were expecting? Adrian On 6 Dec 2016, at 09:34, Khitai Pang <khitai.pang@REDACTED> wrote: Hi, I have the following function in user_default.erl: run_on_all_nodes() -> [spawn(Node, fun() -> io:format("This runs on ~p~n", [node()]) end) || Node <- nodes()]. When I run this function, processes can be spawned on remote nodes but it seems that they don't run: (myapp@REDACTED)1> nodes(). [myapp@REDACTED,rabbit@REDACTED,rabbit@REDACTED] (myapp@REDACTED)2> run_on_all_nodes(). [<27616.12336.0>,<27612.15341.39>,<27615.22399.50>] If I manually run the spawn command in the attached shell, it all works fine: (myapp@REDACTED)3> [spawn(Node, fun() -> io:format("This runs on ~p~n", [node()]) end) || Node <- nodes()]. This runs on myapp@REDACTED This runs on rabbit@REDACTED This runs on rabbit@REDACTED [<27616.12339.0>,<27612.15569.39>,<27615.22640.50>] Any idea? Thanks Khitai _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From walter.weinmann@REDACTED Tue Dec 6 11:37:35 2016 From: walter.weinmann@REDACTED (Walter Weinmann) Date: Tue, 6 Dec 2016 11:37:35 +0100 Subject: [erlang-questions] B-trees Message-ID: I would like to propose an EEP to implement a b-trees module similar to gb-trees but with pluggable persistence and pluggable sort functionality. A first implementation is available under https://github.com/walter-weinmann /b_trees. Any interest? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dszoboszlay@REDACTED Tue Dec 6 12:37:22 2016 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Tue, 06 Dec 2016 11:37:22 +0000 Subject: [erlang-questions] processes spawned on remote nodes don't run In-Reply-To: References: Message-ID: Do you have the same version of user_default on all the nodes? You try to execute a local fun remotely, it will only work if you have exactly the same version of the beams on all the nodes. Cheers, Daniel On Tue, 6 Dec 2016 at 11:35 Khitai Pang wrote: > Hi Adrian, > > Thank you for the information but unfortunately it's not the cause of my > problem. If I add 'user' io:format(): > > (myapp@REDACTED)6> run_on_all_nodes(). > [<27616.12396.0>,<27612.19603.39>,<27615.48.51>] > > (myapp@REDACTED)7> [spawn(Node, fun() -> io:format(user, "This runs on > ~p~n", [node()]) end) || Node <- nodes()]. > [<27616.12397.0>,<27612.19600.39>,<27615.77.51>] > > (myapp@REDACTED)8> [spawn(Node, fun() -> io:format("This runs on ~p~n", > [node()]) end) || Node <- nodes()]. > > This runs on myapp@REDACTED > This runs on rabbit@REDACTED > This runs on rabbit@REDACTED > [<27616.12398.0>,<27612.19631.39>,<27615.89.51>] > > It doesn't seem to be I/O channeling problem, it seems that the spawned > processes just don't run... > > > Thanks > > khitai > > > On 2016/12/6 17:52, Adrian Roe wrote: > > This confused me for a while as well? > > To quote http://erlang.org/doc/man/erlang.html > > group_leader() -> pid() > > Returns the process identifier of the group leader for the process > evaluating the function. > > Every process is a member of some process group and all groups have a *group > leader*. All I/O from the group is channeled to the group leader. When a > new process is spawned, it gets the same group leader as the spawning > process. Initially, at system startup, init is both its own group leader > and the group leader of all processes. > There?s a helpful post here > > http://erlang.org/pipermail/erlang-questions/2008-April/034573.html that > talks about exactly what you are seeing. > > If you change the > io:format("This runs on ~p~n", [node()]) > to > io:format(user, "This runs on ~p~n", [node()]) > > I strongly suspect you will see the output you were expecting? > > Adrian > > On 6 Dec 2016, at 09:34, Khitai Pang < > khitai.pang@REDACTED> wrote: > > Hi, > > I have the following function in user_default.erl: > > run_on_all_nodes() -> > [spawn(Node, fun() -> > io:format("This runs on ~p~n", [node()]) > end) || Node <- nodes()]. > > When I run this function, processes can be spawned on remote nodes but > it seems that they don't run: > > (myapp@REDACTED)1> nodes(). > [myapp@REDACTED,rabbit@REDACTED,rabbit@REDACTED] > > (myapp@REDACTED)2> run_on_all_nodes(). > [<27616.12336.0>,<27612.15341.39>,<27615.22399.50>] > > If I manually run the spawn command in the attached shell, it all works > fine: > > (myapp@REDACTED)3> [spawn(Node, fun() -> io:format("This runs on ~p~n", > [node()]) end) || Node <- nodes()]. > This runs on myapp@REDACTED > This runs on rabbit@REDACTED > This runs on rabbit@REDACTED > [<27616.12339.0>,<27612.15569.39>,<27615.22640.50>] > > > Any idea? > > > Thanks > Khitai > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sperber@REDACTED Tue Dec 6 13:02:07 2016 From: sperber@REDACTED (Michael Sperber) Date: Tue, 06 Dec 2016 13:02:07 +0100 Subject: [erlang-questions] Call for Participation: BOB 2017 (February 24, Berlin) Message-ID: As always, Erlang content at BOB! ================================================================ BOB 2017 Conference "What happens if we simply use what's best?" February 24, 2017 Berlin http://bobkonf.de/2017/ Program: http://bobkonf.de/2017/program.html Registration: http://bobkonf.de/2017/registration.html ================================================================ BOB is the conference for developers, architects and decision-makers to explore technologies beyond the mainstream in software development, and to find the best tools available to software developers today. Our goal is for all participants of BOB to return home with new insights that enable them to improve their own software development experiences. The program features 14 talks and 8 tutorials on current topics: http://bobkonf.de/2017/program.html The subject range of talks includes functional programming, advanced front-end development, and sophisticated uses of types. The tutorials feature introductions to Haskell, Swift, PureScript, React, QuickCheck, Agda, CRDTs and Servant. John Hughes will give the keynote talk. Registration is open online: http://bobkonf.de/2017/registration.html NOTE: The early-bird rates expire on January 23, 2017! BOB cooperates with the :clojured conference on the following day. There is a registration discount available for participants of both events. http://www.clojured.de/ From liuzhongzheng2012@REDACTED Tue Dec 6 14:16:10 2016 From: liuzhongzheng2012@REDACTED (Zhongzheng Liu) Date: Tue, 6 Dec 2016 21:16:10 +0800 Subject: [erlang-questions] How to profiling a nif library? Message-ID: Hi mail-list: I tried to compile a nif library with gcc -pg option, then run it in erlang. No profiling result outputed. I also tried sprof , No profiling result outputed. How to profiling a nif library of erlang? Thanks Zhongzheng Liu From jesper.louis.andersen@REDACTED Tue Dec 6 14:42:45 2016 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 06 Dec 2016 13:42:45 +0000 Subject: [erlang-questions] Idiomatically handling multiple validation checks In-Reply-To: References: Message-ID: On Tue, Dec 6, 2016 at 5:52 AM qp wrote: > Hi, I am new to Erlang and wrote this code for validating that the Name, > Action & Target atoms passed in to validRequest are all valid. > > Another solution is this: Let us start by assuming the presence of functions of this form: -spec valid_name(Target) -> {name, Name} | {error, Reason}. -spec valid_action(Target) -> {action, Name} | {error, Reason}. -spec valid_request(Target) -> {request, Name} | {error, Reason}. We would like to set up a verifier which either returns a context, or fails with the errors present in that context: -type context() :: #{ name := any(), action := any(), request := any(), _ => _ }. -spec verify(Target) -> {ok, context()} | {error, [Reason]}. The context contains input-coerced, verified data and is represented as a map in order to make it easier to users to process that context later on. verify(Target) -> ErrorF = fun ({error, _}) -> true; (_) -> false end, case lists:partition(ErrorF, [valid_name(Target), valid_action(Target), valid_request(Target)]) of {OK, []} -> {ok, maps:from_list(OK)}; {_, Errs} -> {error, Errs} end. The function uses 'lists:partition/2' to split the errors away from the valid outputs. Then we can simply match on the errors: if there are none, we proceed with the context. If there are errors, return them all. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Tue Dec 6 14:46:03 2016 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 6 Dec 2016 08:46:03 -0500 Subject: [erlang-questions] Idiomatically handling multiple validation checks In-Reply-To: <211577939.sQYGRkTBiL@changa> References: <211577939.sQYGRkTBiL@changa> Message-ID: <20161206134558.GA91959@fhebert-ltm2.internal.salesforce.com> On 12/06, zxq9 wrote: >That's one way to do it... but most code I've seen that makes heavy use >of `throw` has insane structural issues (either actual bugs, or mental >grenades with C++ style confusing execution). > > [...] > >Why not return `{error, Reason}` back up the chain of calls? Then the >top-level caller can choose whether to crash on an `{ok, Value}` or >`{error, Reason}` assertion match, or engage in their own insanity with >`throw` .. `catch` and other nonsense. > I find this depends a whole lot on how much validation there is to do. If there's 3-5 rules, then nested cases are easy to handle, to write, and to read. When you've got 15-30 rules if not more, then they become cumbersome extremely rapidly. For larger cases, there's a few discriminating questions I like to ask: - Do you have to interoperate with other pieces of code you haven't written or is this a thing you fully control? - Do you need to tell the user what failed or you're just pruning bad data? - Do you need to find all the errors or the first one you find will do? The first question can pretty much decide the solution for you, but adapters can always be built. The second question tells me how much information you have to be able to provide back. If you're just pruning and/or returning data, a quick and clean-ish pattern is really just to use throw and exceptions, but I like to have them named and in order: -spec validate(map()) -> {ok, map()} | {error, term()}. validate(Data) -> try min_age(18, Data), user_type(admin, Data), verified_credit_card(Data), ... user_status(regular, Data), % not suspended or validating {ok, Data} catch Reason -> {error, Reason} end. min_age(Req, #{age := Given}) when -> Req >= Given -> ok; min_age(_, #{age := Given}) -> throw({error, {too_young, Given}}); min_age(_, _) -> throw({error, age_missing}). ... This preserves a seemingly functional interface, allows to easily rearrange the validation order (and put rapid or frequent discriminators early). Now the downside with this is that if your user is an actual person and they have to massage their data, you're gonna give them the real shitty experience of "submitting the form 30 times to see what gives". Instead we can use validation functions like: -spec validate(map()) -> {ok, map()} | {error, [term()}]}. validate(Data) -> Funs = [fun(Data) -> min_age(18, Data) end ,fun(Data) -> user_type(admin, Data) end ,fun verified_credit_card/1 ... ,user_status(regular, Data) % not suspended or validating ], case validate(Data, Funs) of [] -> {ok, Data}; List -> {error, List} end. -spec validate(map(), [fun()]) -> [term()]. validate(Data, Funs) -> lists:filtermap( fun(F) -> case F(Data) of {error, Term} -> {true, Term}; _ -> false end end, Funs ). min_age(Req, #{age := Given}) when -> Req >= Given -> ok; min_age(_, #{age := Given}) -> {error, {too_young, Given}}; min_age(_, _) -> {error, age_missing}. ... What this does here is change filters like min_age/2 to be functional (no exception throwing!) and uses the lists:filtermap function to return only the invalid results in a list. This list is then packaged as an error and lets the caller know everything that went wrong. The interesting effect of doing this is especially in getting all of the small validation routines into a clear functional pattern. It would, on one hand, be simpler to test and reason about, and could be used with a try ... catch as earlier by just wrapping the result in a function like: throw_error({error, R}) -> throw(R); throw_error(X) -> X. This looks good, but the downside is that this functional approach is that each of the validation function may repeat previously done steps. For example, a lot of validation functions may only make sense to enable once basic prerequired validation has been done (i.e. the user is registered fully may be a requirement to having a lot of fields to validate in further points). the exception based approach makes this very simple by just aborting ASAP (depending on how you wrote it). It also means it may be a bit harder to follow through. What you may end up with with complex rules is possibly a bunch of stepwise validation of more complex sort or getting a hybrid format: validate(Data) -> try throw_errs(validate(Data, format_rules())), throw_errs(validate(Data, user_info_rules())), ... {ok, Data} catch Err -> {error, Err} end. throw_errs([]) -> ok; throw_errs(Other) -> throw(Other). user_info_rules() -> [fun verified_credit_card/1, ...]. ... The same code can be organized in nested case/of situations by doing an explicit check at every level, but I'll leave this one for you to imagine being written. Regards, Fred. From jvdvleuten@REDACTED Tue Dec 6 14:52:59 2016 From: jvdvleuten@REDACTED (Johan van der Vleuten) Date: Tue, 6 Dec 2016 14:52:59 +0100 Subject: [erlang-questions] Keeping application configuration parameters after installing new release Message-ID: Hi all, I am fairly new to Erlang, so reading a lot of docs and trying out lots of new things. The problem I am now facing and cannot get an elegant solution for is the following: When I upgrade a release, containing an application which has set application parameters using 'application:set_env/3', they are disgarded after having called 'release_handler:install_release/1'. This is described in the manual here: http://erlang.org/doc/design_principles/release_handling.html#id84983 quoting: This means that parameter values set in the other system configuration > files and values set using application:set_env/3 are disregarded. > When an installed release is made permanent, the system process init is > set to point out the new sys.config. The application using application:set_env/3 in this case is riak_core, which I have as a dependency. I circumnavigate this by including the old sys.config in the release (which has been generated by cuttlefish before) and then calling a function inside riak_core that re-appends the remaining application parameters they made earlier with 'application:set_env/3' calls. I include this function call in my relup file. However, this does not seem like an elegant solution to my problem. Is there another way to *not* discard the application parameters at all during an upgrade of a new release? How to use the same parameters currently in the running release? If I don't include a sys.config in the release folder ($ROOT/Releases/VSN), all system config and application parameters are cleared. I have checked this myself, unless I am doing something very wrong. I could also extract all current application's parameters using application:get_all_env/1 for every app running, before releasing, and put those in the new sys.config file. Could not find this problem or a solution in the mailing list before or on any other site. Maybe I am doing something wrong with my releases and upgrading them in general... Any other ideas or people facing this problem? Thanks, Johan -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Tue Dec 6 19:59:25 2016 From: lloyd@REDACTED (lloyd@REDACTED) Date: Tue, 6 Dec 2016 13:59:25 -0500 (EST) Subject: [erlang-questions] =?utf-8?q?file=3Alist=5Fdir/1_mystery?= Message-ID: <1481050765.371925439@apps.rackspace.com> Hello, Working with a fork of erlguten, I created an image directory under priv today but file:list_dir/1 can't find it. Bash sees it: Test 1: lloyd@REDACTED:~/EG/erlguten/priv$ ls -l total 20 -rw-rw-r-- 1 lloyd lloyd 179 Nov 1 16:45 font_locations drwxrwxr-x 5 lloyd lloyd 4096 Nov 1 16:45 fonts drwxrwxr-x 3 lloyd lloyd 4096 Nov 1 16:45 hyphenation drwxrwxr-x 2 lloyd lloyd 4096 Dec 3 14:12 images drwxrwxr-x 2 lloyd lloyd 4096 Nov 1 16:45 src Test 2: lloyd@REDACTED:~/EG/erlguten/priv$ cd /home/lloyd/EG/erlguten/priv/images lloyd@REDACTED:~/EG/erlguten/priv/images$ ls -l total 52 -rw-rw-r-- 1 lloyd lloyd 52820 Dec 3 14:12 book_cover.jpg But but file:list_dir/1 doesn't: 1> Priv = code:priv_dir(erlguten). "/home/lloyd/Erlang/erlguten/priv" 2> 2> Images = Priv ++ "/images". "/home/lloyd/Erlang/erlguten/priv/images" 3> 3> file:list_dir(Priv). {ok,["hyphenation","font_locations","fonts","src"]} 4> 4> file:list_dir(Images). {error,enoent} Note: erl started with lloyd@REDACTED:~/EG/erlguten$ -pa ebin -pa lib/*/ebin Here's the bash listing for ~/EG/erlguten: lloyd@REDACTED:~/EG/erlguten$ ls -l total 260 -rw-rw-r-- 1 lloyd lloyd 17376 Nov 1 18:48 comcast_bill.pdf drwxrwxr-x 2 lloyd lloyd 4096 Nov 2 19:10 deps drwxrwxr-x 2 lloyd lloyd 4096 Nov 1 16:45 doc drwxrwxr-x 2 lloyd lloyd 4096 Nov 29 17:12 ebin drwxrwxr-x 2 lloyd lloyd 4096 Nov 30 18:31 include -rw-rw-r-- 1 lloyd lloyd 32 Nov 1 16:45 install.mk drwxrwxr-x 3 lloyd lloyd 4096 Nov 1 17:42 lib -rw-rw-r-- 1 lloyd lloyd 1157 Nov 1 16:45 LICENSE -rw-rw-r-- 1 lloyd lloyd 206 Nov 1 16:45 Makefile drwxrwxr-x 6 lloyd lloyd 4096 Dec 6 12:47 priv -rw-rw-r-- 1 lloyd lloyd 7208 Nov 1 16:45 readme.md -rwxrwxr-x 1 lloyd lloyd 175879 Nov 2 16:56 rebar -rw-rw-r-- 1 lloyd lloyd 201 Nov 1 17:42 rebar.config drwxrwxr-x 11 lloyd lloyd 4096 Dec 2 16:19 src drwxrwxr-x 4 lloyd lloyd 4096 Nov 30 18:28 test I've tried rebooting, but that doesn't solve the problem. Are my aging eyes missing something? Or my aging brain overlooking something obvious? In short, how can I return images to my application? Many thanks, LRP From lloyd@REDACTED Tue Dec 6 20:03:59 2016 From: lloyd@REDACTED (lloyd@REDACTED) Date: Tue, 6 Dec 2016 14:03:59 -0500 (EST) Subject: [erlang-questions] =?utf-8?q?file=3Alist=5Fdir/1_mystery_--_a_clu?= =?utf-8?q?e?= Message-ID: <1481051039.999514314@apps.rackspace.com> Ah, my eyes did miss something: code:priv_dir returns the Erlang directory. 1> Priv = code:priv_dir(erlguten). "/home/lloyd/Erlang/erlguten/priv" But that still puzzles me. Where should I put my images? Thanks again, LRP Hello, Working with a fork of erlguten, I created an image directory under priv today but file:list_dir/1 can't find it. Bash sees it: Test 1: lloyd@REDACTED:~/EG/erlguten/priv$ ls -l total 20 -rw-rw-r-- 1 lloyd lloyd 179 Nov 1 16:45 font_locations drwxrwxr-x 5 lloyd lloyd 4096 Nov 1 16:45 fonts drwxrwxr-x 3 lloyd lloyd 4096 Nov 1 16:45 hyphenation drwxrwxr-x 2 lloyd lloyd 4096 Dec 3 14:12 images drwxrwxr-x 2 lloyd lloyd 4096 Nov 1 16:45 src Test 2: lloyd@REDACTED:~/EG/erlguten/priv$ cd /home/lloyd/EG/erlguten/priv/images lloyd@REDACTED:~/EG/erlguten/priv/images$ ls -l total 52 -rw-rw-r-- 1 lloyd lloyd 52820 Dec 3 14:12 book_cover.jpg But but file:list_dir/1 doesn't: 1> Priv = code:priv_dir(erlguten). "/home/lloyd/Erlang/erlguten/priv" 2> 2> Images = Priv ++ "/images". "/home/lloyd/Erlang/erlguten/priv/images" 3> 3> file:list_dir(Priv). {ok,["hyphenation","font_locations","fonts","src"]} 4> 4> file:list_dir(Images). {error,enoent} Note: erl started with lloyd@REDACTED:~/EG/erlguten$ -pa ebin -pa lib/*/ebin Here's the bash listing for ~/EG/erlguten: lloyd@REDACTED:~/EG/erlguten$ ls -l total 260 -rw-rw-r-- 1 lloyd lloyd 17376 Nov 1 18:48 comcast_bill.pdf drwxrwxr-x 2 lloyd lloyd 4096 Nov 2 19:10 deps drwxrwxr-x 2 lloyd lloyd 4096 Nov 1 16:45 doc drwxrwxr-x 2 lloyd lloyd 4096 Nov 29 17:12 ebin drwxrwxr-x 2 lloyd lloyd 4096 Nov 30 18:31 include -rw-rw-r-- 1 lloyd lloyd 32 Nov 1 16:45 install.mk drwxrwxr-x 3 lloyd lloyd 4096 Nov 1 17:42 lib -rw-rw-r-- 1 lloyd lloyd 1157 Nov 1 16:45 LICENSE -rw-rw-r-- 1 lloyd lloyd 206 Nov 1 16:45 Makefile drwxrwxr-x 6 lloyd lloyd 4096 Dec 6 12:47 priv -rw-rw-r-- 1 lloyd lloyd 7208 Nov 1 16:45 readme.md -rwxrwxr-x 1 lloyd lloyd 175879 Nov 2 16:56 rebar -rw-rw-r-- 1 lloyd lloyd 201 Nov 1 17:42 rebar.config drwxrwxr-x 11 lloyd lloyd 4096 Dec 2 16:19 src drwxrwxr-x 4 lloyd lloyd 4096 Nov 30 18:28 test I've tried rebooting, but that doesn't solve the problem. Are my aging eyes missing something? Or my aging brain overlooking something obvious? In short, how can I return images to my application? Many thanks, LRP From lloyd@REDACTED Tue Dec 6 20:19:11 2016 From: lloyd@REDACTED (lloyd@REDACTED) Date: Tue, 6 Dec 2016 14:19:11 -0500 (EST) Subject: [erlang-questions] =?utf-8?q?file=3Alist=5Fdir/1_mystery_--_a_clu?= =?utf-8?q?e_+_solved_it?= Message-ID: <1481051951.580125827@apps.rackspace.com> So sorry for interrupting your day. I just discovered that I had another erlguten directory in the Erlang directory. When I renamed it, my system found the correct erlguten directory. It's still no clear to me why this should happen. But, apologies. LRP Ah, my eyes did miss something: code:priv_dir returns the Erlang directory. 1> Priv = code:priv_dir(erlguten). "/home/lloyd/Erlang/erlguten/priv" But that still puzzles me. Where should I put my images? Thanks again, LRP Hello, Working with a fork of erlguten, I created an image directory under priv today but file:list_dir/1 can't find it. Bash sees it: Test 1: lloyd@REDACTED:~/EG/erlguten/priv$ ls -l total 20 -rw-rw-r-- 1 lloyd lloyd 179 Nov 1 16:45 font_locations drwxrwxr-x 5 lloyd lloyd 4096 Nov 1 16:45 fonts drwxrwxr-x 3 lloyd lloyd 4096 Nov 1 16:45 hyphenation drwxrwxr-x 2 lloyd lloyd 4096 Dec 3 14:12 images drwxrwxr-x 2 lloyd lloyd 4096 Nov 1 16:45 src Test 2: lloyd@REDACTED:~/EG/erlguten/priv$ cd /home/lloyd/EG/erlguten/priv/images lloyd@REDACTED:~/EG/erlguten/priv/images$ ls -l total 52 -rw-rw-r-- 1 lloyd lloyd 52820 Dec 3 14:12 book_cover.jpg But but file:list_dir/1 doesn't: 1> Priv = code:priv_dir(erlguten). "/home/lloyd/Erlang/erlguten/priv" 2> 2> Images = Priv ++ "/images". "/home/lloyd/Erlang/erlguten/priv/images" 3> 3> file:list_dir(Priv). {ok,["hyphenation","font_locations","fonts","src"]} 4> 4> file:list_dir(Images). {error,enoent} Note: erl started with lloyd@REDACTED:~/EG/erlguten$ -pa ebin -pa lib/*/ebin Here's the bash listing for ~/EG/erlguten: lloyd@REDACTED:~/EG/erlguten$ ls -l total 260 -rw-rw-r-- 1 lloyd lloyd 17376 Nov 1 18:48 comcast_bill.pdf drwxrwxr-x 2 lloyd lloyd 4096 Nov 2 19:10 deps drwxrwxr-x 2 lloyd lloyd 4096 Nov 1 16:45 doc drwxrwxr-x 2 lloyd lloyd 4096 Nov 29 17:12 ebin drwxrwxr-x 2 lloyd lloyd 4096 Nov 30 18:31 include -rw-rw-r-- 1 lloyd lloyd 32 Nov 1 16:45 install.mk drwxrwxr-x 3 lloyd lloyd 4096 Nov 1 17:42 lib -rw-rw-r-- 1 lloyd lloyd 1157 Nov 1 16:45 LICENSE -rw-rw-r-- 1 lloyd lloyd 206 Nov 1 16:45 Makefile drwxrwxr-x 6 lloyd lloyd 4096 Dec 6 12:47 priv -rw-rw-r-- 1 lloyd lloyd 7208 Nov 1 16:45 readme.md -rwxrwxr-x 1 lloyd lloyd 175879 Nov 2 16:56 rebar -rw-rw-r-- 1 lloyd lloyd 201 Nov 1 17:42 rebar.config drwxrwxr-x 11 lloyd lloyd 4096 Dec 2 16:19 src drwxrwxr-x 4 lloyd lloyd 4096 Nov 30 18:28 test I've tried rebooting, but that doesn't solve the problem. Are my aging eyes missing something? Or my aging brain overlooking something obvious? In short, how can I return images to my application? Many thanks, LRP From silviu.cpp@REDACTED Tue Dec 6 21:56:10 2016 From: silviu.cpp@REDACTED (Caragea Silviu) Date: Tue, 6 Dec 2016 22:56:10 +0200 Subject: [erlang-questions] erl_driver vs NIF Message-ID: Hello, Analyzing the project fast_tls by processone ( https://github.com/processone/fast_tls/blob/master/c_src/fast_tls_drv.c) it raised me a question: >From what I know the NIF is the fastest way to call c/c++ code from erlang. The disadvantage is that you need to return in under 1 ms in order to make sure the schedulers are not affected. using erl_driver and port_control is not the same ? because based on the documentation this is a sync call that blocks scheduler until the native method returns. I'm wrong or not ? Silviu -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtrlists@REDACTED Tue Dec 6 23:53:03 2016 From: rtrlists@REDACTED (Robert Raschke) Date: Tue, 6 Dec 2016 23:53:03 +0100 Subject: [erlang-questions] file:list_dir/1 mystery In-Reply-To: <1481050765.371925439@apps.rackspace.com> References: <1481050765.371925439@apps.rackspace.com> Message-ID: Hi Lloyd, In the bash examples your paths start with /home/lloyd/EG , whereas your erl examples start with /home/lloyd/Erlang Could that be it? Hth, Robby On 6 Dec 2016 19:59, wrote: > Hello, > > Working with a fork of erlguten, I created an image directory under priv > today but file:list_dir/1 can't find it. > > Bash sees it: > > Test 1: > > lloyd@REDACTED:~/EG/erlguten/priv$ ls -l > total 20 > -rw-rw-r-- 1 lloyd lloyd 179 Nov 1 16:45 font_locations > drwxrwxr-x 5 lloyd lloyd 4096 Nov 1 16:45 fonts > drwxrwxr-x 3 lloyd lloyd 4096 Nov 1 16:45 hyphenation > drwxrwxr-x 2 lloyd lloyd 4096 Dec 3 14:12 images > drwxrwxr-x 2 lloyd lloyd 4096 Nov 1 16:45 src > > Test 2: > > lloyd@REDACTED:~/EG/erlguten/priv$ cd /home/lloyd/EG/erlguten/priv/images > lloyd@REDACTED:~/EG/erlguten/priv/images$ ls -l > total 52 > -rw-rw-r-- 1 lloyd lloyd 52820 Dec 3 14:12 book_cover.jpg > > But but file:list_dir/1 doesn't: > > 1> Priv = code:priv_dir(erlguten). > "/home/lloyd/Erlang/erlguten/priv" > 2> > 2> Images = Priv ++ "/images". > "/home/lloyd/Erlang/erlguten/priv/images" > 3> > 3> file:list_dir(Priv). > {ok,["hyphenation","font_locations","fonts","src"]} > 4> > 4> file:list_dir(Images). > {error,enoent} > > Note: erl started with > > lloyd@REDACTED:~/EG/erlguten$ -pa ebin -pa lib/*/ebin > > Here's the bash listing for ~/EG/erlguten: > > lloyd@REDACTED:~/EG/erlguten$ ls -l > total 260 > -rw-rw-r-- 1 lloyd lloyd 17376 Nov 1 18:48 comcast_bill.pdf > drwxrwxr-x 2 lloyd lloyd 4096 Nov 2 19:10 deps > drwxrwxr-x 2 lloyd lloyd 4096 Nov 1 16:45 doc > drwxrwxr-x 2 lloyd lloyd 4096 Nov 29 17:12 ebin > drwxrwxr-x 2 lloyd lloyd 4096 Nov 30 18:31 include > -rw-rw-r-- 1 lloyd lloyd 32 Nov 1 16:45 install.mk > drwxrwxr-x 3 lloyd lloyd 4096 Nov 1 17:42 lib > -rw-rw-r-- 1 lloyd lloyd 1157 Nov 1 16:45 LICENSE > -rw-rw-r-- 1 lloyd lloyd 206 Nov 1 16:45 Makefile > drwxrwxr-x 6 lloyd lloyd 4096 Dec 6 12:47 priv > -rw-rw-r-- 1 lloyd lloyd 7208 Nov 1 16:45 readme.md > -rwxrwxr-x 1 lloyd lloyd 175879 Nov 2 16:56 rebar > -rw-rw-r-- 1 lloyd lloyd 201 Nov 1 17:42 rebar.config > drwxrwxr-x 11 lloyd lloyd 4096 Dec 2 16:19 src > drwxrwxr-x 4 lloyd lloyd 4096 Nov 30 18:28 test > > I've tried rebooting, but that doesn't solve the problem. > > Are my aging eyes missing something? Or my aging brain overlooking > something obvious? > > In short, how can I return images to my application? > > Many thanks, > > LRP > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtrlists@REDACTED Tue Dec 6 23:54:22 2016 From: rtrlists@REDACTED (Robert Raschke) Date: Tue, 6 Dec 2016 23:54:22 +0100 Subject: [erlang-questions] file:list_dir/1 mystery In-Reply-To: References: <1481050765.371925439@apps.rackspace.com> Message-ID: And I should learn to scan ahead in my email ? On 6 Dec 2016 23:53, "Robert Raschke" wrote: > Hi Lloyd, > > In the bash examples your paths start with /home/lloyd/EG , whereas your > erl examples start with /home/lloyd/Erlang > > Could that be it? > > Hth, > Robby > > On 6 Dec 2016 19:59, wrote: > >> Hello, >> >> Working with a fork of erlguten, I created an image directory under priv >> today but file:list_dir/1 can't find it. >> >> Bash sees it: >> >> Test 1: >> >> lloyd@REDACTED:~/EG/erlguten/priv$ ls -l >> total 20 >> -rw-rw-r-- 1 lloyd lloyd 179 Nov 1 16:45 font_locations >> drwxrwxr-x 5 lloyd lloyd 4096 Nov 1 16:45 fonts >> drwxrwxr-x 3 lloyd lloyd 4096 Nov 1 16:45 hyphenation >> drwxrwxr-x 2 lloyd lloyd 4096 Dec 3 14:12 images >> drwxrwxr-x 2 lloyd lloyd 4096 Nov 1 16:45 src >> >> Test 2: >> >> lloyd@REDACTED:~/EG/erlguten/priv$ cd /home/lloyd/EG/erlguten/priv/images >> lloyd@REDACTED:~/EG/erlguten/priv/images$ ls -l >> total 52 >> -rw-rw-r-- 1 lloyd lloyd 52820 Dec 3 14:12 book_cover.jpg >> >> But but file:list_dir/1 doesn't: >> >> 1> Priv = code:priv_dir(erlguten). >> "/home/lloyd/Erlang/erlguten/priv" >> 2> >> 2> Images = Priv ++ "/images". >> "/home/lloyd/Erlang/erlguten/priv/images" >> 3> >> 3> file:list_dir(Priv). >> {ok,["hyphenation","font_locations","fonts","src"]} >> 4> >> 4> file:list_dir(Images). >> {error,enoent} >> >> Note: erl started with >> >> lloyd@REDACTED:~/EG/erlguten$ -pa ebin -pa lib/*/ebin >> >> Here's the bash listing for ~/EG/erlguten: >> >> lloyd@REDACTED:~/EG/erlguten$ ls -l >> total 260 >> -rw-rw-r-- 1 lloyd lloyd 17376 Nov 1 18:48 comcast_bill.pdf >> drwxrwxr-x 2 lloyd lloyd 4096 Nov 2 19:10 deps >> drwxrwxr-x 2 lloyd lloyd 4096 Nov 1 16:45 doc >> drwxrwxr-x 2 lloyd lloyd 4096 Nov 29 17:12 ebin >> drwxrwxr-x 2 lloyd lloyd 4096 Nov 30 18:31 include >> -rw-rw-r-- 1 lloyd lloyd 32 Nov 1 16:45 install.mk >> drwxrwxr-x 3 lloyd lloyd 4096 Nov 1 17:42 lib >> -rw-rw-r-- 1 lloyd lloyd 1157 Nov 1 16:45 LICENSE >> -rw-rw-r-- 1 lloyd lloyd 206 Nov 1 16:45 Makefile >> drwxrwxr-x 6 lloyd lloyd 4096 Dec 6 12:47 priv >> -rw-rw-r-- 1 lloyd lloyd 7208 Nov 1 16:45 readme.md >> -rwxrwxr-x 1 lloyd lloyd 175879 Nov 2 16:56 rebar >> -rw-rw-r-- 1 lloyd lloyd 201 Nov 1 17:42 rebar.config >> drwxrwxr-x 11 lloyd lloyd 4096 Dec 2 16:19 src >> drwxrwxr-x 4 lloyd lloyd 4096 Nov 30 18:28 test >> >> I've tried rebooting, but that doesn't solve the problem. >> >> Are my aging eyes missing something? Or my aging brain overlooking >> something obvious? >> >> In short, how can I return images to my application? >> >> Many thanks, >> >> LRP >> >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Dec 6 23:59:27 2016 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 7 Dec 2016 11:59:27 +1300 Subject: [erlang-questions] Idiomatically handling multiple validation checks In-Reply-To: <2480782.vzNZox2jzC@changa> References: <2480782.vzNZox2jzC@changa> Message-ID: <14f7f606-271f-f4f0-7e4d-1e5c6398e9ac@cs.otago.ac.nz> Joe Duffy of Microsoft gave a talk about systems programming in C#. (He worked on a project that built an operating system + utilities in C#.) The talk is at https://www.infoq.com/presentations/csharp-systems-programming?utm_campaign=rightbar_v2&utm_source=infoq&utm_medium=presentations_link&utm_content=link_text He had some interesting things to say about exceptions and errors. * Exceptions are meant for recoverable errors; but many errors are not recoverable! * A bug is an error the programmer didn't expect; a recoverable error is an expected condition, resulting from programmatic data validation. * Treating bugs and recoverable errors homogeneously creates reliability problems. * If you?re going to fail, do it fast. * Exceptions [can] delay the inevitable and invite[] abuse. * Fail-fast ensures bugs are caught promptly before they can do more damage. * In [their] experience, 1:10 ratio of recoverable errors (exceptions) to bugs (fail-fast). Apart from that, I really liked zxq9's response. From Oliver.Korpilla@REDACTED Wed Dec 7 00:55:26 2016 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Wed, 7 Dec 2016 00:55:26 +0100 Subject: [erlang-questions] Idiomatically handling multiple validation checks In-Reply-To: <14f7f606-271f-f4f0-7e4d-1e5c6398e9ac@cs.otago.ac.nz> References: <2480782.vzNZox2jzC@changa>, <14f7f606-271f-f4f0-7e4d-1e5c6398e9ac@cs.otago.ac.nz> Message-ID: Hello. I really like this answer. Can't say much about Erlang on this one but in most languages exceptions unwind the stack and are really slow. While performance might not be so important in a true error state, the actual example given is nothing such. Validating data before using it is the bread and butter of any program that wants to be minimally robust. Invalid input is not an error, it's quite normal in many applications - be it entering into a web form or trying or reading a MP3 file with an encoding error. Applications should optimally deal with such scenarios efficiently. And exceptions are inefficient but can afford to be because of the assumption "it's all gone to crap anyway." If invalid input, however, is part of the normal operation of a system (like the Unix shell, also) then it might be wasteful depending on your overall performance constraints. (At least in Java or C++, again not knowledgeable on Erlang exceptions and implementation details.) >From an earlier post and addressed to that author: "Absolutely it can be done Haskell/ML way with values wrapped as {ok,_} or {error, _} and returned. This will make beautiful flow and will triple the chain runner function size, or you will have to add an {error, _} clause to each of your handlers. And exceptions still are going to happen, so those have to be caught and wrapped with {error, _} too." I think there is lack of clarity regarding idiomatic Erlang (as the thread title implies)... What is called the "Haskell/ML way" I see also as the Erlang way. Or the elixir way. Returning {error, ...} and handling that can make good readable code IMO. Exceptions are not meant to reduce code size nor remove case handling. Whenever you raise an exception you lose almost all context of what happened and give minimal information to somebody up the call stack. This "I don't care" style of error handling can be highly problematic and I've run into plenty of situations where the handler of the exception has no idea how to handle it or what went wrong. The example however, had one nice aspect to its flow. The nested functions for validating one element after another could be written as input |> check1 |> check2 |> check3 in elixir, and I do love that style (and find nested function calls on results of function calls very messy, syntax-wise). This whole chain can work nicely and is not bad style. The whole exception handling block around it, however, could be. This is like Java where if the thread dies your application dies with it so you build these hedge mazes of try/catch around everything and sully every function signature with exception declarations. If I run into an unrecoverable error in Erlang I have seen the light of dying and restarting. I don't pretend my dumb program anticipates all future scenarios and have fared really well with fast restart. I do however not use exceptions for recoverable scenarios because there are actually quite few. Since I work with implementing protocols most scenarios end in sending back a message anyway that terminates a scenario. case statements or a separation function head for unhandled scenarios are IMO more readable... It is really a shame Erlang does not seem to have the cond statement elixir has. It makes the code for validation clear as can be. You see, each cond statements is evaluated and we only care about the first one returning "true." So, you could write validators returning true or false and do this: cond do check1(some_input) -> {:error, ...} check2(other_input) -> {:error, ...} ... _default -> fun_for_valid_input(all_input) end You then have isolated all validation calls into one function in one place without any nesting or exception handling in a readable way. Each individual check can be written clearly as a separate function returning true on detecting a problem. Everything is very straightforward and should perform well. As an added bonus it has the short-cut semantics of the and statement but is way more flexible - the first true statement exits but with a meaningful value. And the individual condition checks can be _anything_. Hard to beat for this scenario IMO. It's not Erlang but at least Beam... Oliver Gesendet:?Dienstag, 06. Dezember 2016 um 23:59 Uhr Von:?"Richard A. O'Keefe" An:?erlang-questions@REDACTED Betreff:?Re: [erlang-questions] Idiomatically handling multiple validation checks Joe Duffy of Microsoft gave a talk about systems programming in C#. (He worked on a project that built an operating system + utilities in C#.) The talk is at https://www.infoq.com/presentations/csharp-systems-programming?utm_campaign=rightbar_v2&utm_source=infoq&utm_medium=presentations_link&utm_content=link_text He had some interesting things to say about exceptions and errors. * Exceptions are meant for recoverable errors; but many errors are not recoverable! * A bug is an error the programmer didn't expect; a recoverable error is an expected condition, resulting from programmatic data validation. * Treating bugs and recoverable errors homogeneously creates reliability problems. * If you?re going to fail, do it fast. * Exceptions [can] delay the inevitable and invite[] abuse. * Fail-fast ensures bugs are caught promptly before they can do more damage. * In [their] experience, 1:10 ratio of recoverable errors (exceptions) to bugs (fail-fast). Apart from that, I really liked zxq9's response. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions[http://erlang.org/mailman/listinfo/erlang-questions] From mononcqc@REDACTED Wed Dec 7 01:22:50 2016 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 6 Dec 2016 19:22:50 -0500 Subject: [erlang-questions] Idiomatically handling multiple validation checks In-Reply-To: <14f7f606-271f-f4f0-7e4d-1e5c6398e9ac@cs.otago.ac.nz> References: <2480782.vzNZox2jzC@changa> <14f7f606-271f-f4f0-7e4d-1e5c6398e9ac@cs.otago.ac.nz> Message-ID: <20161207002249.GJ684@ferdmbp.local> On 12/07, Richard A. O'Keefe wrote: >* Exceptions are meant for recoverable errors; > but many errors are not recoverable! >* A bug is an error the programmer didn't expect; > a recoverable error is an expected condition, > resulting from programmatic data validation. >* Treating bugs and recoverable errors homogeneously > creates reliability problems. That's one of the interesting things in Erlang; not all exceptions are equal: - throws are recoverable non-local returns - errors are usually bugs - exits signal that the current process cannot keep going - try ... catch does not mix them up unless especially asking for it (with a _:Whatever match) Their semantic distinction is something I find useful in every day systems in terms of how these things are handled for Erlang. raise an error for bugs, raise an exit for dead ends where the process does not make sense, raise an exception for cumbersome non-local returns (which then should get transformed to {error, Reason} past the functional interface). If you want to turn a recoverable error into a bug, you just match {ok, Res} = Exp, and if it's bad, you now get a badmatch. That kind of dynamic is not something I remember seeing in multiple languages before. From petergi@REDACTED Wed Dec 7 08:57:13 2016 From: petergi@REDACTED (Peter J Etheridge) Date: Wed, 07 Dec 2016 18:57:13 +1100 Subject: [erlang-questions] wxSizer Message-ID: Dear List, As mentioned in recent postings, learning wxErlang is challenging. Any improvement to the documentation will be gratefully received. In his wxErlang work-up Doug Edmunds states, 'the order of Sizers determines appearance'. I have set '9' for W & H, and '5' for spacers as these are not yet working. I will appreciate any suggestion for Spacer ordering and any links to other tutorials or examples. Thank you in advance. -module(text_in_box). -include_lib("wx/include/wx.hrl"). -compile([export_all]). -export([make_window/0]). make_window() -> ? S = wx:new(), ? F = wxFrame:new(S, 1, "text_in_box", [{size, {300, 300}}]), ? P = wxPanel:new(F), ? B = wxStaticBox:new(P, 2, "&Static box"), ? wxPanel:setBackgroundColour(P, ?wxLIGHT_GREY), ? wxStaticBox:setBackgroundColour(B, ?wxBLUE), ? wxStaticBox:setLabel(B, "Static box"), ? T = wxStaticText:new(P, 3, "Static text", []), ? wxStaticText:setLabel(T, "Vertical static text"), ? Bs = wxStaticBoxSizer:new(?wxHORIZONTAL, P, [{label, "Horizontal static box sizer"}]), ? Ts = wxBoxSizer:new(?wxVERTICAL), ? wxSizer:add(Bs, B, []), ? wxSizer:add(Bs, 9, 9, []), ? wxSizer:add(Bs, 9, 9, []), ? wxSizer:add(Ts, T, []), ? wxSizer:add(Ts, 9, 9, []), ? wxSizer:addSpacer(Ts, 5), ? wxSizer:addSpacer(Bs, 5), ? wxSizer:add(Bs, Ts, []), ? wxPanel:setSizer(P, Bs), ? wxFrame:show(F), ? % create two listeners ? wxFrame:connect(F, close_window), ? wxPanel:connect(P, command_button_clicked), ? loop(F), ? {F, Bs, Ts, self()}. loop(S) -> ? receive ??? #wx{event = #wxCommand{type = command_event}} -> ????? wxWindow:close(S, []), ????? loop(S); ??? #wx{event = #wxClose{type = close_window}} -> ????? ok ? end. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Wed Dec 7 09:57:48 2016 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 7 Dec 2016 09:57:48 +0100 Subject: [erlang-questions] erl_driver vs NIF In-Reply-To: References: Message-ID: Hello, On Tue, Dec 6, 2016 at 9:56 PM, Caragea Silviu wrote: > > From what I know the NIF is the fastest way to call c/c++ code from > erlang. The disadvantage is that you need to return in under 1 ms in order > to make sure the schedulers are not affected. > > using erl_driver and port_control is not the same ? because based on the > documentation this is a sync call that blocks scheduler until the native > method returns. > The same ~1 ms limit exists for port callbacks as nif function calls. For some reason there is a rumor circling around that you somehow circumvent the limit if you convert your nif code to a linked-in driver. This is not the case. The only thing you do by converting the code from a nif to linked-in driver is gain access to more tools to partitions the work (mostly driver_select[1], but also some other things like the async thread pool), and you possibly make some of the problems that long running native code can cause less likely. Lukas 1: Soon this limit will be removed by the inclusion of enif_select, https://github.com/erlang/otp/pull/1264 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Tobias.Schlager@REDACTED Wed Dec 7 09:58:35 2016 From: Tobias.Schlager@REDACTED (Tobias Schlager) Date: Wed, 7 Dec 2016 08:58:35 +0000 Subject: [erlang-questions] wxSizer In-Reply-To: References: Message-ID: <12F2115FD1CCEE4294943B2608A18FA301C2891EA3@MAIL01.win.lbaum.eu> Hi Peter, I don't know much about the Erlang port of wxWidgets. However, I have done some work with native wxWidgets recently. From what I can see, the API seems to be a straight mapping to the C++ code. This would lead to the conclusion that the same rules and documentation can be applied to the Erlang port. To understand layout and spacing I would suggest starting with the 'Sizers Overview' [1]. You could also have a look at the zetcode tutorials [2] which are pretty good and easy to understand. To address your specific question: Windows (every UI element is a window in wxWidgets) and sizers appear in the order they are added to their (parent) sizer (not their parent window). The C++ library also has support to insert windows into sizers at a specific index. Usually, you simply add items in the order you want them to appear in the layout. The normal workflow is to create the UI elements, then add them to a sizer in the order you want them. After this you can add this sizer to another sizer or set it as the main sizer of the frame, finally layout the window. Regards Tobias [1] http://docs.wxwidgets.org/stable/overview_sizer.html [2] http://zetcode.com/gui/wxwidgets/ From kennethlakin@REDACTED Wed Dec 7 10:13:57 2016 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Wed, 7 Dec 2016 01:13:57 -0800 Subject: [erlang-questions] SSL server session cache table issues: Message-ID: <92d86075-ec8e-33e7-f492-44dba4314193@gmail.com> I'm using OTP 19.1.3 on a 32-bit Linux system. From what I can tell, there's no documented way to control the number of entries in the server SSL session cache ETS table. (That is, the server_ssl_otp_session_cache table.) Passing the {reuse_sessions, false} SSL server option does not prevent new entries in the table. The obvious (but undocumented) ways to control the number of entries in the server SSL session cache appear to fail to work. Is there a way to control the size of the cache, without overriding the session cache callback with the (undocumented) session_cb ssl application option? Is my only hope to use the session_cb application option to switch to a noop session cache manager? Details: I've software that acts as a TLS server but has no use for the SSL server session cache. I've a load test that causes one new TLS connection to be created, used, and (AFAICT, cleanly) destroyed roughly every half-second. This seems to create one entry in server_ssl_otp_session_cache roughly every half-second. This table appears to grow without bound. Conversation from the OTP 18.3-ish days seems to claim that this table has a limit of 1000 entries. My testing indicates that this is untrue; that the table has no size limit. Additional testing reveals that neither the session_lifetime nor the session_cache_server_max ssl application options appear to have any effect. A 20 minute run results in a ~2000 item table, even with both of these options set to 1. I'm making use of the (documented!) cb_info ssl server option to provide an alternate, reliable transport for the TLS data. Might this be interacting poorly with the session cache cleaning code? Thanks! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From silviu.cpp@REDACTED Wed Dec 7 10:38:47 2016 From: silviu.cpp@REDACTED (Caragea Silviu) Date: Wed, 7 Dec 2016 11:38:47 +0200 Subject: [erlang-questions] erl_driver vs NIF In-Reply-To: References: Message-ID: Hello, Thanks a lot for your response. Silviu On Wed, Dec 7, 2016 at 10:57 AM, Lukas Larsson wrote: > Hello, > > On Tue, Dec 6, 2016 at 9:56 PM, Caragea Silviu > wrote: > >> >> From what I know the NIF is the fastest way to call c/c++ code from >> erlang. The disadvantage is that you need to return in under 1 ms in order >> to make sure the schedulers are not affected. >> >> using erl_driver and port_control is not the same ? because based on the >> documentation this is a sync call that blocks scheduler until the native >> method returns. >> > > The same ~1 ms limit exists for port callbacks as nif function calls. > > For some reason there is a rumor circling around that you somehow > circumvent the limit if you convert your nif code to a linked-in driver. > This is not the case. The only thing you do by converting the code from a > nif to linked-in driver is gain access to more tools to partitions the work > (mostly driver_select[1], but also some other things like the async thread > pool), and you possibly make some of the problems that long running native > code can cause less likely. > > Lukas > > 1: Soon this limit will be removed by the inclusion of enif_select, > https://github.com/erlang/otp/pull/1264 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rj@REDACTED Wed Dec 7 12:21:22 2016 From: rj@REDACTED (Richard Jones) Date: Wed, 7 Dec 2016 11:21:22 +0000 Subject: [erlang-questions] Unimplemented {buffer, integer()} option in ssl module - bug? In-Reply-To: References: Message-ID: Ok. Imo the docs aren't really clear on this. In lieu of any patch, it would be nice to see more explicit docs listing which gen_tcp options are emulated by SSL and which aren't. for now I am emulating the option in my socket wrapper module :) thanks RJ On 2 December 2016 at 13:57, Ingela Andin wrote: > Hi! > > 2016-12-02 11:43 GMT+01:00 Alex S. : >> >> >> 2 ???. 2016 ?., ? 13:28, Ingela Andin ???????(?): >> >> >> >> 2016-12-01 12:37 GMT+01:00 Richard Jones : >>> >>> The {buffer, integer()} option from gen_tcp is useful for breaking up >>> incoming data into manageable sizes, especially when in {packet, line} >>> mode. >>> >>> The docs suggest (kinda) that the gen_tcp options should also work >>> when connecting via the ssl module, but this isn' >>> >>> always the case - >>> the buffer option is silently ignored by ssl:connect >> >> >> Actually it is not ignored it just is not has no noticeable affect to you. >> >>> Is this >>> >>> a bug? Tested on erl 17, 18, 19 >>> >> >> For this option to work as expected for you it needs to be emulated by the >> ssl process as are packet related options. So it is a missing feature. >> >> Regards Ingela Erlang/OTP- team >> >> I would argue that the combination of these two effects can also be >> described as ?buffer_size is pushed a level lower? (where it is obviously >> not useful to the user of ssl module), and I would additionally argue that >> erroneously so. The option needs to be either explicitly not supported (with >> a note in the documentation), or do the right thing, not do some invisible >> thing that has no positive effect. > > > > I am not disagreeing with you on that. I think we can consider it a bug > that this options is let through, and a missing feature that it is not > emulated. The reasoning is that ssl shall not do option checking on gen_tcp > options that may be used and does note affect ssl. But some options must be > emulated. This option was clearly missed. Anyone wanting this option is > welcome to make a PR. > > Regards Ingela Erlang/OTP- team > > From ulf@REDACTED Wed Dec 7 12:51:03 2016 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 7 Dec 2016 12:51:03 +0100 Subject: [erlang-questions] wildcarded resource counters in Gproc In-Reply-To: References: Message-ID: 2016-12-01 22:45 GMT+01:00 Garrett Smith : > I need to set aside some time - and compared to some of the recent > events in global politics, this is not a super big deal. You wound me. ;-) I am working on a more general solution. It's about as painful as I suspected, but you were of course right in voicing your objections. That was the reason why I posted the question, after all. I'll return shortly with a new PR. BR, Ulf -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Wed Dec 7 14:30:59 2016 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 7 Dec 2016 08:30:59 -0500 Subject: [erlang-questions] SSL server session cache table issues: In-Reply-To: <92d86075-ec8e-33e7-f492-44dba4314193@gmail.com> References: <92d86075-ec8e-33e7-f492-44dba4314193@gmail.com> Message-ID: <20161207133058.GB91959@fhebert-ltm2.internal.salesforce.com> On 12/07, Kenneth Lakin wrote: >I'm using OTP 19.1.3 on a 32-bit Linux system. > >From what I can tell, there's no documented way to control the number of >entries in the server SSL session cache ETS table. (That is, the >server_ssl_otp_session_cache table.) Passing the {reuse_sessions, false} >SSL server option does not prevent new entries in the table. The obvious >(but undocumented) ways to control the number of entries in the server >SSL session cache appear to fail to work. > The option is cleverly named; it mentions 'reuse_sessions' and from what I recall, that's what it does; it prevents reuse of sessions, but not their creation. >Is there a way to control the size of the cache, without overriding the >session cache callback with the (undocumented) session_cb ssl >application option? Is my only hope to use the session_cb application >option to switch to a noop session cache manager? > Not that I know of. It's been easier in our set up to just override the behaviour to give a null cache than anything else. Otherwise, just the sequential scans of the cache table to clean up the sessions ended up costing us a lot in terms of latency, kind of forcing a stop-the-world GC onto the SSL connections since if I recall, the sequential scans blocked the table, which in turn block the session manager process. We further detected more scans happening due to a PEM cache, which we patched options to bypass it in https://github.com/erlang/otp/pull/1143 -- the OTP team merged it in for OTP 19.1 From steven.charles.davis@REDACTED Wed Dec 7 15:29:52 2016 From: steven.charles.davis@REDACTED (Steve Davis) Date: Wed, 7 Dec 2016 08:29:52 -0600 Subject: [erlang-questions] wxSizer Message-ID: <156E320B-41CD-40BE-A2AF-B243806A1B54@gmail.com> Hi Peter, I remember struggling with sizers when trying to code up a gs-style interface to wx. This module may contain a few pointers that would be helpful: https://github.com/komone/gx/blob/master/src/ui/gx_ui_sizer.erl /s -------------- next part -------------- An HTML attachment was scrubbed... URL: From dangud@REDACTED Wed Dec 7 15:40:17 2016 From: dangud@REDACTED (Dan Gudmundsson) Date: Wed, 07 Dec 2016 14:40:17 +0000 Subject: [erlang-questions] wxSizer In-Reply-To: <156E320B-41CD-40BE-A2AF-B243806A1B54@gmail.com> References: <156E320B-41CD-40BE-A2AF-B243806A1B54@gmail.com> Message-ID: And maybe run wx:demo() and look at the sizer examples or the code for wx:demo(). Examples exist but for documentation you will have to look at wxWidgets which docs are not that great either. On Wed, Dec 7, 2016 at 3:30 PM Steve Davis wrote: > Hi Peter, > > I remember struggling with sizers when trying to code up a gs-style > interface to wx. This module may contain a few pointers that would be > helpful: > > https://github.com/komone/gx/blob/master/src/ui/gx_ui_sizer.erl > > /s > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steven.charles.davis@REDACTED Wed Dec 7 15:42:44 2016 From: steven.charles.davis@REDACTED (Steve Davis) Date: Wed, 7 Dec 2016 08:42:44 -0600 Subject: [erlang-questions] wxSizer In-Reply-To: References: <156E320B-41CD-40BE-A2AF-B243806A1B54@gmail.com> Message-ID: <2528A21A-2CB2-4BAD-8C80-3B11796BE72F@gmail.com> Also, I remember the wxPy tutorials helping me a lot. https://wiki.wxpython.org/SizerTutorials All examples are in python but easy to read and the concepts are the same. > On Dec 7, 2016, at 8:40 AM, Dan Gudmundsson wrote: > > And maybe run wx:demo() and look at the sizer examples or the code for wx:demo(). > Examples exist but for documentation you will have to look at wxWidgets which docs are not that great either. > > On Wed, Dec 7, 2016 at 3:30 PM Steve Davis > wrote: > Hi Peter, > > I remember struggling with sizers when trying to code up a gs-style interface to wx. This module may contain a few pointers that would be helpful: > > https://github.com/komone/gx/blob/master/src/ui/gx_ui_sizer.erl > > /s > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.goertzen@REDACTED Wed Dec 7 18:43:37 2016 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Wed, 07 Dec 2016 17:43:37 +0000 Subject: [erlang-questions] Do you wanna play with enif_select? In-Reply-To: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> References: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> Message-ID: I'm excited to see progress in the area, but I am struggling to understand a few things in the enif_select API: 1. Why are the resource extensions required? Couldn't a 'stop' message be sent to the calling process which could then call a plain old NIF to shut things down? 2. Does the ref parameter have to be a ref? An arbitrary term would be useful. Thanks, Dan. On Fri, Dec 2, 2016 at 9:22 AM Sverker Eriksson < sverker.eriksson@REDACTED> wrote: > https://github.com/erlang/otp/pull/1264 > > > This PR is introducing a new feature enif_select() as a way for > NIFs to ask for asynchronous notification when a file descriptor > (a socket for example) is ready for (more) reading or writing. > > The API is not carved in stone yet and we would like some feedback. > So please, take enif_select for a spin and share your thoughts. > > /Sverker, Erlang/OTP @ Ericsson > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikpelinux@REDACTED Wed Dec 7 18:56:43 2016 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Wed, 7 Dec 2016 18:56:43 +0100 Subject: [erlang-questions] B-trees In-Reply-To: References: Message-ID: <22600.19803.930291.556507@gargle.gargle.HOWL> Walter Weinmann writes: > I would like to propose an EEP to implement a b-trees module similar > to gb-trees > but with pluggable persistence and pluggable sort functionality. > > A first implementation is available under https://github.com/walter-weinmann > /b_trees. > > Any interest? Well, I'm working on something similar: B-trees with I/O handled by user-provided callbacks; see . There are some TODOs related to tuning storage overheads and performance, but apart from that it's complete for our use case. As for having one in OTP: that would be nice, especially if OTP adds some BIFs to speed up the page search and manipulation routines. From sverker.eriksson@REDACTED Wed Dec 7 19:37:49 2016 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Wed, 7 Dec 2016 19:37:49 +0100 Subject: [erlang-questions] Do you wanna play with enif_select? In-Reply-To: References: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> Message-ID: <376022d6-36c0-2373-dfa1-b5e08cc06424@ericsson.com> On 12/07/2016 06:43 PM, Daniel Goertzen wrote: > I'm excited to see progress in the area, but I am struggling to understand > a few things in the enif_select API: > > 1. Why are the resource extensions required? Couldn't a 'stop' message be > sent to the calling process which could then call a plain old NIF to shut > things down? The idea was to also be able to handle cases when the process is no longer alive. For example, if it crashes due to some error. The stop function can then be called by the VM to do the cleanup and avoid leaking file descriptors. > 2. Does the ref parameter have to be a ref? An arbitrary term would be > useful. The current implementation only works for local refs and immediate terms (atoms, small integers, local pids and ports). I thought I should keep it simple to begin with, but it would be no problem to accept other terms. Just some more code to handle the dynamic allocation as the terms need to be copied from and to the process heap. /Sverker From wde@REDACTED Wed Dec 7 20:43:26 2016 From: wde@REDACTED (wde@REDACTED) Date: Wed, 7 Dec 2016 20:43:26 +0100 Subject: [erlang-questions] Erlang driver for OBD2 ELM327 Message-ID: <9442b52e-41b1-1dae-8f81-f91208b89745@free.fr> Dear All, Does someone know if an Erlang driver exists to connect OBD2 ELM327 devices ? Regards, wde From daniel.goertzen@REDACTED Wed Dec 7 21:41:40 2016 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Wed, 07 Dec 2016 20:41:40 +0000 Subject: [erlang-questions] Do you wanna play with enif_select? In-Reply-To: <376022d6-36c0-2373-dfa1-b5e08cc06424@ericsson.com> References: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> <376022d6-36c0-2373-dfa1-b5e08cc06424@ericsson.com> Message-ID: The presence of the resource makes sense. But why the special close/stop methods? Is the existing destructor insufficient in some way? On Wed, Dec 7, 2016 at 12:38 PM Sverker Eriksson < sverker.eriksson@REDACTED> wrote: > > > On 12/07/2016 06:43 PM, Daniel Goertzen wrote: > > I'm excited to see progress in the area, but I am struggling to > understand > > a few things in the enif_select API: > > > > 1. Why are the resource extensions required? Couldn't a 'stop' message be > > sent to the calling process which could then call a plain old NIF to shut > > things down? > > The idea was to also be able to handle cases when the process > is no longer alive. For example, if it crashes due to some error. > The stop function can then be called by the VM to do the cleanup > and avoid leaking file descriptors. > > > > 2. Does the ref parameter have to be a ref? An arbitrary term would be > > useful. > The current implementation only works for local refs and > immediate terms (atoms, small integers, local pids and ports). > I thought I should keep it simple to begin with, but it would be > no problem to accept other terms. Just some more code to > handle the dynamic allocation as the terms need to be copied > from and to the process heap. > > > /Sverker > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennethlakin@REDACTED Wed Dec 7 23:41:23 2016 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Wed, 7 Dec 2016 14:41:23 -0800 Subject: [erlang-questions] SSL server session cache table issues: In-Reply-To: <20161207133058.GB91959@fhebert-ltm2.internal.salesforce.com> References: <92d86075-ec8e-33e7-f492-44dba4314193@gmail.com> <20161207133058.GB91959@fhebert-ltm2.internal.salesforce.com> Message-ID: <9f33c080-e0e4-f17b-a018-edc8d025abe1@gmail.com> On 12/07/2016 05:30 AM, Fred Hebert wrote: > The option is cleverly named ... it prevents reuse of sessions, but not > their creation. Is it true that there's no way to change connection-specific SSL server or client options without tearing down the connection and recreating it? If it is, then shouldn't {reuse_sessions, false} prevent session caching for that connection? Or might the assumption be that a session created and cached with a {r_s, false} connection could be later reused with a {r_s, true} connection? > We further detected more scans happening due to a PEM cache, which we > patched options to bypass it in https://github.com/erlang/otp/pull/1143 > -- the OTP team merged it in for OTP 19.1 I was going to complain that bypass_pem_cache option was undocumented, but I just found the ssl_app page that documents that option, as well as all of the SSL app options that I _thought_ were undocumented. I now feel quite a bit sheepish. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ok@REDACTED Thu Dec 8 03:20:48 2016 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 8 Dec 2016 15:20:48 +1300 Subject: [erlang-questions] B-trees In-Reply-To: <22600.19803.930291.556507@gargle.gargle.HOWL> References: <22600.19803.930291.556507@gargle.gargle.HOWL> Message-ID: <1e34a921-ef76-49e7-334f-1c1458e11c82@cs.otago.ac.nz> What are the advantages of B-trees as an internal data structure? Would it be possible to layer something like this on top of lmdb? From walter.weinmann@REDACTED Thu Dec 8 07:36:16 2016 From: walter.weinmann@REDACTED (Walter Weinmann) Date: Thu, 8 Dec 2016 07:36:16 +0100 Subject: [erlang-questions] B-trees In-Reply-To: <1e34a921-ef76-49e7-334f-1c1458e11c82@cs.otago.ac.nz> References: <22600.19803.930291.556507@gargle.gargle.HOWL> <1e34a921-ef76-49e7-334f-1c1458e11c82@cs.otago.ac.nz> Message-ID: As a pure internal data structure you would prefer binary trees - I did include similar tests with the gb-trees module in my performance analysis. gb_trees is much faster. The advantage of the b_trees module comes in connection with the storage of large data blocks on external memory. I had a quick look on emdb (https://github.com/alepharchives/emdb) which is the Erlang wrapper of LMDB. I couldn't see a traversal operation. So it seems to be difficult to do range scans. The main advantages of the b_trees module (https://github.com/walter- weinmann/b_trees) are: - pure Erlang implementation, - very similar API to the gb_trees module, - implementation of standard algorithms (see Cormen, Introduction to Algorithms, Chapter 18 B-Trees) - key sort is pluggable - data storage is pluggable So I suppose the b_trees module would only be applied in connection with external memory. On 8 December 2016 at 03:20, Richard A. O'Keefe wrote: > What are the advantages of B-trees as an internal data structure? > Would it be possible to layer something like this on top of lmdb? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- ___________________________________________________________________________ *Walter Weinmann* ___________________________________________________________________________ Obertorplatz 4 CH-4130 Rheinfelden Tel +41 (0)61 841 06 10 ___________________________________________________________________________ Schulstrasse 1 CH-6037 Root Tel +41 (0)41 530 39 70 ___________________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Thu Dec 8 08:04:14 2016 From: sergej.jurecko@REDACTED (=?utf-8?Q?Sergej_Jure=C4=8Dko?=) Date: Thu, 8 Dec 2016 08:04:14 +0100 Subject: [erlang-questions] B-trees In-Reply-To: References: <22600.19803.930291.556507@gargle.gargle.HOWL> <1e34a921-ef76-49e7-334f-1c1458e11c82@cs.otago.ac.nz> Message-ID: > On 8 Dec 2016, at 07:36, Walter Weinmann wrote: > > I had a quick look on emdb (https :// github .com/ alepharchives / emdb ) which is the Erlang wrapper of LMDB. I couldn't see a traversal operation. So it seems to be difficult to do range scans. The C lmdb api has everything one might need. This erlang library however should never be used by anyone. It does disk operations directly on scheduler threads. regards, Sergej -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Thu Dec 8 08:47:29 2016 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 8 Dec 2016 20:47:29 +1300 Subject: [erlang-questions] B-trees In-Reply-To: References: <22600.19803.930291.556507@gargle.gargle.HOWL> <1e34a921-ef76-49e7-334f-1c1458e11c82@cs.otago.ac.nz> Message-ID: <5aeea330-2b11-de04-1d0c-e9e39069c066@cs.otago.ac.nz> On 8/12/16 7:36 PM, Walter Weinmann wrote: Every time I've tried to use B-trees, I've run SPLAT! into a wall: - the B-tree data structure wants to read and write single pages - and wants to store a fairly large minimum number of keys on a page - disc pages aren't *that* big - my keys would usually be small but *might* be large (well, kilobytes rather than megabytes, but that's enough to mess up B-trees) - dealing with this meant storing hash(Key) -> {Key,Value} instead, which meant * having to handle duplicates that aren't real duplicates (again rare, but since it *could* happen...) * losing the ordering properties of the B-tree. How do you handle the keys-might-be-large problem? For what it's worth, LMDB does traversal; if emdb doesn't, so much the worse for emdb. LMDB has some very nice reliability properties. From ingela.andin@REDACTED Thu Dec 8 11:09:39 2016 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 8 Dec 2016 11:09:39 +0100 Subject: [erlang-questions] SSL server session cache table issues: In-Reply-To: <9f33c080-e0e4-f17b-a018-edc8d025abe1@gmail.com> References: <92d86075-ec8e-33e7-f492-44dba4314193@gmail.com> <20161207133058.GB91959@fhebert-ltm2.internal.salesforce.com> <9f33c080-e0e4-f17b-a018-edc8d025abe1@gmail.com> Message-ID: Hi! To start I will address questions earlier in the thread.The max session is not a hard limit, when max is reached a process of removing all previous sessions is started. It might be a problem if you have a constant high load. Maybe we should rethink!? The thing is the manager process must manage the session ids (used for resumption) and to not make this bottleneck to big we have to keep session data around for a little while as to let already spawned connection processes read it from the public ets table. 2016-12-07 23:41 GMT+01:00 Kenneth Lakin : > On 12/07/2016 05:30 AM, Fred Hebert wrote: > > The option is cleverly named ... it prevents reuse of sessions, but not > > their creation. > > Is it true that there's no way to change connection-specific SSL server > or client options without tearing down the connection and recreating it? > If it is, then shouldn't {reuse_sessions, false} prevent session caching > for that connection? Or might the assumption be that a session created > and cached with a {r_s, false} connection could be later reused with a > {r_s, true} connection? > > TLS sessions are not connection specific, so the assumption is that a later connection may want to reuse the session, and that by default you want to reuse a sessions if possible. I also think there is reason why RFC 5077 - "Transport Layer Security (TLS) Session Resumption without Server-Side State" was written. > > We further detected more scans happening due to a PEM cache, which we > > patched options to bypass it in https://github.com/erlang/otp/pull/1143 > > -- the OTP team merged it in for OTP 19.1 > > I am working at the moment to move the PEM cache to its own process, so that its work should not interrupt the manager process. We will keep the bypass option but it should no longer be needed to avoid cache overhead for certificates provided in DER format. > I was going to complain that bypass_pem_cache option was undocumented, > but I just found the ssl_app page that documents that option, as well as > all of the SSL app options that I _thought_ were undocumented. I now > feel quite a bit sheepish. > I can take the opportunity to say that our docs has to kind of man pages, man(6) the applications man page so to speak, that documents for instance application variables. An man(3) pages that document the API of the application. The main API of the application is implemented in a module with the same name as the application, but there can be more API modules. I would guess that the HTML version of the docs are the most used version. I am just saying there is a reason for the structure and knowing about it makes it easier to find things. Regards Ingela Erlang/OTP team - Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger@REDACTED Thu Dec 8 12:09:49 2016 From: roger@REDACTED (Roger Lipscombe) Date: Thu, 8 Dec 2016 11:09:49 +0000 Subject: [erlang-questions] Adjusting the error level for SSL alerts? Message-ID: When using ssl, it's possible to set the 'log_alert' value to true or false. If true, SSL/TLS alerts are logged using error_logger:format. If false, they're simply dropped. See https://github.com/erlang/otp/blob/OTP-19.1.6/lib/ssl/src/ssl_connection.erl#L2352 error_logger:format outputs the message as an 'error'. We use lager with a custom backend that turns error reports into emails. This means that we have to use {log_alert, false}, lest we get inundated with email reports of SSL alerts due to random port scans, etc. I was wondering whether it would be possible to control the error level that ssl uses. That is: instead of {log_alert, true} and {log_alert, false}, we could, instead, use that to define the log level, such as {log_alert, false | info | warning | error}, and have that translate to: no-op, error_logger:info_msg, error_logger:warning_msg, etc. Then the TLS alerts would naturally arrive as lager messages at 'info', 'warning' or 'error' level, as appropriate. This would, in turn, mean that they get written to our log file, but don't result in noisy error emails, unless we want them to. Obviously, 'true' would be treated as a synonym for 'error', for backwards-compatibility. Alternatively, an extra option, 'log_alert_level', defaulting to 'error', might make more sense. More simply, but less flexible, and having global effect, changing error_logger:format to error_logger:warning_msg would allow coarse-grained control of the resulting logging level by use of the +W emulator flag. (Aside: the +W default value appears to have changed at some point between 17.5 and 19.1...). Does any of this sound like a sensible proposal? If so, I'll try to find time to put together a pull request. -- Cheers, Roger. From bengt.kleberg@REDACTED Thu Dec 8 13:51:15 2016 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Thu, 8 Dec 2016 13:51:15 +0100 Subject: [erlang-questions] sftp closing on big file Message-ID: <896b6559-3941-f438-2a51-4a1f74134e1f@ericsson.com> Greetings, Using erlang/otp_19.1 I attempt to write a 1171 MB file using ssh_sftp:write_file/4, but it fails with {error,closed} around (different amount at each attempt) 1000 MB. Manually doing "scp" works. Many other, smaller, files also work with write_file/4. I could look at this using Wireshark, but perhaps there is something obvious that somebody could suggest/knows? bengt From erlang@REDACTED Thu Dec 8 13:52:17 2016 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 8 Dec 2016 13:52:17 +0100 Subject: [erlang-questions] Erlang for an audio streaming server project? In-Reply-To: References: Message-ID: On Tue, Dec 6, 2016 at 1:42 AM, Edgar H wrote: > Thanks a lot for the replies guys, really appreciate it! > > From what you've told me so far I think have the following options... > > 1) Stream into Icecast directly from Java and forget about Erlang (I > wouldn't want to take this one - It isn't challenging and won't learn > anything new, but would get the job done). Shameless plug: https://pragprog.com/book/jaerlang2/programming-erlang (section 17.6) has all the code for a SHOUTcast server (which is pretty similar to icecast) - there's also so code to read ID3 headers etc This is not production quality - but illustrates the principles involved and works fine for a home server I'm not sure why you need codecs - all you need to do is splice up the data (say MP3) and stick in a few headers (or is this what a codec does?) /Joe > 2) Send raw data from Java to the mini-audio-streaming-server coded by me > (or an existing HTTP server like cowboy) which implements an encoder > developed by me or can I blindly send the data pre-encoded from Java using > ffmpeg without checking which format is it? (If I know for sure the kind of > content I'm streaming, should I just bother about the MIME-types?). > 3) (If I understood Max's reply correctly) Develop the codecs I need in > Erlang and then work with them to encode the raw data and stream it. At this > point, with my current knowledge in codecs (null) and time for the project, > it seems like an impossible goal to reach for me... > > Eric, the codecs I need are mainly mp3 and vorbis. > > 2016-12-05 16:14 GMT+01:00 Eric des Courtis : >> >> I agree with Max ditch the Java it will just slow you down. What codecs do >> you need anyhow? >> >> On Sun, Dec 4, 2016 at 2:20 PM, Max Lapshin wrote: >>> >>> You will have very interesting deployment pain if you take java with you. >>> >>> It is not clear what for to waste time on it? >>> >>> There are 2-3 codecs and 3-4 transport payloads for audio. Just write all >>> this in erlang, it is not hard. >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From kennethlakin@REDACTED Thu Dec 8 14:39:10 2016 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Thu, 8 Dec 2016 05:39:10 -0800 Subject: [erlang-questions] SSL server session cache table issues: In-Reply-To: References: <92d86075-ec8e-33e7-f492-44dba4314193@gmail.com> <20161207133058.GB91959@fhebert-ltm2.internal.salesforce.com> <9f33c080-e0e4-f17b-a018-edc8d025abe1@gmail.com> Message-ID: <8bfb3051-9a4f-83e3-31f9-30b0dfbf19d7@gmail.com> On 12/08/2016 02:09 AM, Ingela Andin wrote: > I am just saying there is a reason for the [documentation] structure > and knowing about it makes it easier to find things. No doubt! In a follow-up email I mentioned that I'd found the ssl_app documentation, which contained the documentation that I thought didn't exist. I remain sheepish. > TLS sessions are not connection specific, so the assumption is that a later > connection may want to reuse the session, and that by default you want to > reuse a sessions if possible. That makes sense. Thanks for the explanation! > The max session is not a hard limit, when max is reached > a process of removing all previous sessions is started. > It might be a problem if you have a constant high load. Actually, the ever-growing session cache table problem seems to be caused by starting my application with the rebar3 shell. When I invoke erl directly, the session cache table is regularly purged of old data. I'll do some more digging and then maybe report a bug with the rebar3 people. Sorry for the noise! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ingela.andin@REDACTED Thu Dec 8 15:31:40 2016 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 8 Dec 2016 15:31:40 +0100 Subject: [erlang-questions] SSL server session cache table issues: In-Reply-To: <8bfb3051-9a4f-83e3-31f9-30b0dfbf19d7@gmail.com> References: <92d86075-ec8e-33e7-f492-44dba4314193@gmail.com> <20161207133058.GB91959@fhebert-ltm2.internal.salesforce.com> <9f33c080-e0e4-f17b-a018-edc8d025abe1@gmail.com> <8bfb3051-9a4f-83e3-31f9-30b0dfbf19d7@gmail.com> Message-ID: Hi! It might be if they have a session callback handling module and do not implement the size function. Regards Ingela Erlang/OTP team - Ericsson AB 2016-12-08 14:39 GMT+01:00 Kenneth Lakin : > On 12/08/2016 02:09 AM, Ingela Andin wrote: > > I am just saying there is a reason for the [documentation] structure > > and knowing about it makes it easier to find things. > > No doubt! In a follow-up email I mentioned that I'd found the ssl_app > documentation, which contained the documentation that I thought didn't > exist. I remain sheepish. > > > TLS sessions are not connection specific, so the assumption is that a > later > > connection may want to reuse the session, and that by default you want to > > reuse a sessions if possible. > > That makes sense. Thanks for the explanation! > > > The max session is not a hard limit, when max is reached > > a process of removing all previous sessions is started. > > It might be a problem if you have a constant high load. > > Actually, the ever-growing session cache table problem seems to be > caused by starting my application with the rebar3 shell. When I invoke > erl directly, the session cache table is regularly purged of old data. > > I'll do some more digging and then maybe report a bug with the rebar3 > people. Sorry for the noise! > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lars@REDACTED Thu Dec 8 15:53:53 2016 From: lars@REDACTED (Lars Hesel Christensen) Date: Thu, 8 Dec 2016 15:53:53 +0100 Subject: [erlang-questions] B-trees In-Reply-To: References: <22600.19803.930291.556507@gargle.gargle.HOWL> <1e34a921-ef76-49e7-334f-1c1458e11c82@cs.otago.ac.nz> Message-ID: On 12/08/2016 07:36 AM, Walter Weinmann wrote: > I had a quick look on emdb (https://github.com/alepharchives/emdb) which > is the Erlang wrapper of LMDB. I couldn't see a traversal operation. So > it seems to be difficult to do range scans. I just stumbled across this LMDB driver which looks like it has at least partial cursor support: https://github.com/zambal/elmdb Lars -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From ingela.andin@REDACTED Thu Dec 8 16:06:22 2016 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 8 Dec 2016 16:06:22 +0100 Subject: [erlang-questions] Adjusting the error level for SSL alerts? In-Reply-To: References: Message-ID: Hi! 2016-12-08 12:09 GMT+01:00 Roger Lipscombe : > When using ssl, it's possible to set the 'log_alert' value to true or > false. If true, SSL/TLS alerts are logged using error_logger:format. > If false, they're simply dropped. > > See https://github.com/erlang/otp/blob/OTP-19.1.6/lib/ssl/src/ > ssl_connection.erl#L2352 > > error_logger:format outputs the message as an 'error'. > > We use lager with a custom backend that turns error reports into > emails. This means that we have to use {log_alert, false}, lest we get > inundated with email reports of SSL alerts due to random port scans, > etc. > > I was wondering whether it would be possible to control the error > level that ssl uses. That is: instead of {log_alert, true} and > {log_alert, false}, we could, instead, use that to define the log > level, such as {log_alert, false | info | warning | error}, and have > that translate to: no-op, error_logger:info_msg, > error_logger:warning_msg, etc. > > Then the TLS alerts would naturally arrive as lager messages at > 'info', 'warning' or 'error' level, as appropriate. > > This would, in turn, mean that they get written to our log file, but > don't result in noisy error emails, unless we want them to. > > Obviously, 'true' would be treated as a synonym for 'error', for > backwards-compatibility. This sounds ok to me. Alternatively, an extra option, > 'log_alert_level', defaulting to 'error', might make more sense. > > I have mixed feelings about this. There are already so many options that keeping the number of options down when possible seem appealing. Also I am not a big fan of having different options that that have an impact on each other (other than overriding), granted such options already exist (atleast on inet level). > More simply, but less flexible, and having global effect, changing > error_logger:format to error_logger:warning_msg would allow > coarse-grained control of the resulting logging level by use of the +W > emulator flag. > > Seem like a less general solution that could easily be broken by someone not knowing the rational behind it. Regards Ingela Erlang/OTP team - Ericsson AB (Aside: the +W default value appears to have changed at some point > between 17.5 and 19.1...). > > Does any of this sound like a sensible proposal? If so, I'll try to > find time to put together a pull request. > > -- > Cheers, > Roger. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vances@REDACTED Thu Dec 8 16:12:52 2016 From: vances@REDACTED (Vance Shipley) Date: Thu, 8 Dec 2016 09:12:52 -0600 Subject: [erlang-questions] 18th Anniversary Message-ID: On this day, the 18th anniversary of the open source release of Erlang/OTP, let's all give thanks to the OTP team and Ericsson for their massive contributions. I for one am deeply grateful. http://web.archive.org/web/19991009002753/www.erlang.se/onlinenews/ErlangOTPos.shtml -- -Vance From ingela.andin@REDACTED Thu Dec 8 16:14:27 2016 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 8 Dec 2016 16:14:27 +0100 Subject: [erlang-questions] sftp closing on big file In-Reply-To: <896b6559-3941-f438-2a51-4a1f74134e1f@ericsson.com> References: <896b6559-3941-f438-2a51-4a1f74134e1f@ericsson.com> Message-ID: Hi! Have you tried the latest maint branch? I know that atleast related problems has been fixed. Regards Ingela Erlang/OTP Team - Ericsson AB 2016-12-08 13:51 GMT+01:00 Bengt Kleberg : > Greetings, > > Using erlang/otp_19.1 I attempt to write a 1171 MB file using > ssh_sftp:write_file/4, but it fails with {error,closed} around (different > amount at each attempt) 1000 MB. > > Manually doing "scp" works. Many other, smaller, files also work with > write_file/4. > > I could look at this using Wireshark, but perhaps there is something > obvious that somebody could suggest/knows? > > > bengt > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Thu Dec 8 16:17:10 2016 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 8 Dec 2016 16:17:10 +0100 Subject: [erlang-questions] Do you wanna play with enif_select? In-Reply-To: References: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> <376022d6-36c0-2373-dfa1-b5e08cc06424@ericsson.com> Message-ID: <00841860-04a4-8e41-ad6c-999608460988@ericsson.com> On 12/07/2016 09:41 PM, Daniel Goertzen wrote: > The presence of the resource makes sense. But why the special close/stop > methods? Is the existing destructor insufficient in some way? > The resource destructor will not be called until all references to the resource is gone (by term GC and/or enif_release_resource calls from NIF code). It must be possible to close a file descriptor (think TCP connection) associated with a resource without having to wait for the GC. If the resource have been shared among a number of processes it might take a while until all references are purged on all heaps and the destructor is finally called. Also, it is undefined behaviour (according to POSIX) to close a file descriptor that another thread is doing select/poll on. So, to safely call close on a file descriptor we first may have to wait for a polling thread to wake up. We don't want blocking waits between scheduler threads, so we need a scheduled callback when the fd is safe to close. Summary: We don't want to wait for the GC but we may need to wait for polling thread, hence the stop/close callback (or whatever it should be called). /Sverker From bchesneau@REDACTED Thu Dec 8 16:19:43 2016 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 08 Dec 2016 15:19:43 +0000 Subject: [erlang-questions] B-trees In-Reply-To: References: <22600.19803.930291.556507@gargle.gargle.HOWL> <1e34a921-ef76-49e7-334f-1c1458e11c82@cs.otago.ac.nz> Message-ID: well having support of lmdb is quite orthogonal to have a btree in stdlib. At first lmdb is not really a btree anyway and also comes with its own problems when it's about storing the data on disk. I would love personally to have a memory B+Tree in erlang where you can set a custom function to compare keys. B+Tree have some uses cases that gb_btree can't replace imo. - benoit On Thu, Dec 8, 2016 at 3:54 PM Lars Hesel Christensen wrote: > > > On 12/08/2016 07:36 AM, Walter Weinmann wrote: > > I had a quick look on emdb (https://github.com/alepharchives/emdb) which > > is the Erlang wrapper of LMDB. I couldn't see a traversal operation. So > > it seems to be difficult to do range scans. > > I just stumbled across this LMDB driver which looks like it has at least > partial cursor support: https://github.com/zambal/elmdb > > Lars > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.kleberg@REDACTED Thu Dec 8 16:27:07 2016 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Thu, 8 Dec 2016 16:27:07 +0100 Subject: [erlang-questions] sftp closing on big file In-Reply-To: References: <896b6559-3941-f438-2a51-4a1f74134e1f@ericsson.com> Message-ID: Sorry, but I am actually supposed to use R16B03 (which I have not tried). I test with v19, since I thought it would increase my chances of not getting "upgrade" as the answer. I apologise for this deception. bengt On 12/08/2016 04:14 PM, Ingela Andin wrote: > Hi! > > Have you tried the latest maint branch? I know that atleast related > problems has been fixed. > > Regards Ingela Erlang/OTP Team - Ericsson AB > > 2016-12-08 13:51 GMT+01:00 Bengt Kleberg >: > > Greetings, > > Using erlang/otp_19.1 I attempt to write a 1171 MB file using > ssh_sftp:write_file/4, but it fails with {error,closed} around > (different amount at each attempt) 1000 MB. > > Manually doing "scp" works. Many other, smaller, files also work > with write_file/4. > > I could look at this using Wireshark, but perhaps there is > something obvious that somebody could suggest/knows? > > > bengt > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chassoul@REDACTED Thu Dec 8 19:49:38 2016 From: chassoul@REDACTED (Jean Chassoul) Date: Thu, 8 Dec 2016 13:49:38 -0500 Subject: [erlang-questions] 18th Anniversary In-Reply-To: References: Message-ID: yay! congrats On Thu, Dec 8, 2016 at 10:12 AM, Vance Shipley wrote: > On this day, the 18th anniversary of the open source release of > Erlang/OTP, let's all give thanks to the OTP team and Ericsson for > their massive contributions. I for one am deeply grateful. > > http://web.archive.org/web/19991009002753/www.erlang.se/ > onlinenews/ErlangOTPos.shtml > > -- > -Vance > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Thu Dec 8 22:35:18 2016 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 08 Dec 2016 21:35:18 +0000 Subject: [erlang-questions] listen on multiple interfaces Message-ID: Is there a way to listen on multiple interface using gen_tcp. It seeps we can only choose one using ip, ifaddr. Any hint is welcome :) - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.goertzen@REDACTED Fri Dec 9 00:06:09 2016 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Thu, 08 Dec 2016 23:06:09 +0000 Subject: [erlang-questions] Do you wanna play with enif_select? In-Reply-To: <00841860-04a4-8e41-ad6c-999608460988@ericsson.com> References: <800149ff-cf9c-1b9a-b54f-5f44ad138544@ericsson.com> <376022d6-36c0-2373-dfa1-b5e08cc06424@ericsson.com> <00841860-04a4-8e41-ad6c-999608460988@ericsson.com> Message-ID: Thank you for answering my questions. I understand this API much better now and look forward to its release. On Thu, Dec 8, 2016 at 9:17 AM Sverker Eriksson < sverker.eriksson@REDACTED> wrote: > On 12/07/2016 09:41 PM, Daniel Goertzen wrote: > > The presence of the resource makes sense. But why the special close/stop > > methods? Is the existing destructor insufficient in some way? > > > > The resource destructor will not be called until all references > to the resource is gone (by term GC and/or enif_release_resource > calls from NIF code). > > It must be possible to close a file descriptor (think TCP connection) > associated with a resource without having to wait for the GC. If the > resource have been shared among a number of processes it might > take a while until all references are purged on all heaps and the > destructor is finally called. > > Also, it is undefined behaviour (according to POSIX) to close a file > descriptor > that another thread is doing select/poll on. So, to safely call close on > a file > descriptor we first may have to wait for a polling thread to wake up. We > don't want blocking waits between scheduler threads, so we need a > scheduled callback when the fd is safe to close. > > > Summary: > We don't want to wait for the GC but we may need to wait for polling > thread, hence the stop/close callback (or whatever it should be called). > > /Sverker > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Fri Dec 9 07:09:14 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 09 Dec 2016 06:09:14 +0000 Subject: [erlang-questions] listen on multiple interfaces In-Reply-To: References: Message-ID: {0,0,0,0} Le jeu. 8 d?c. 2016 ? 22:35, Benoit Chesneau a ?crit : > Is there a way to listen on multiple interface using gen_tcp. It seeps we > can only choose one using ip, ifaddr. Any hint is welcome :) > > - benoit > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Fri Dec 9 07:50:05 2016 From: bchesneau@REDACTED (Benoit Chesneau) Date: Fri, 09 Dec 2016 06:50:05 +0000 Subject: [erlang-questions] listen on multiple interfaces In-Reply-To: References: Message-ID: that will listen on all interfaces though. i'm looking for a way to listen on a set of interfaces in fact, sorry to not have been clear :) On Fri, 9 Dec 2016 at 07:09, Frank Muller wrote: > {0,0,0,0} > > Le jeu. 8 d?c. 2016 ? 22:35, Benoit Chesneau a > ?crit : > > Is there a way to listen on multiple interface using gen_tcp. It seeps we > can only choose one using ip, ifaddr. Any hint is welcome :) > > - benoit > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilya.khaprov@REDACTED Fri Dec 9 07:50:45 2016 From: ilya.khaprov@REDACTED (Ilya Khaprov) Date: Fri, 9 Dec 2016 06:50:45 +0000 Subject: [erlang-questions] listen on multiple interfaces In-Reply-To: References: , Message-ID: It means ALL IPv4 addresses? Multiple isn?t always all. AFAIK you can listen either on all or on 1. Ilya From: Frank Muller Sent: Friday, December 9, 2016 09:09 AM To: Benoit Chesneau; erlang-questions@REDACTED Subject: Re: [erlang-questions] listen on multiple interfaces {0,0,0,0} Le jeu. 8 d?c. 2016 ? 22:35, Benoit Chesneau > a ?crit : Is there a way to listen on multiple interface using gen_tcp. It seeps we can only choose one using ip, ifaddr. Any hint is welcome :) - benoit _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From per@REDACTED Fri Dec 9 08:23:27 2016 From: per@REDACTED (Per Hedeland) Date: Fri, 9 Dec 2016 08:23:27 +0100 Subject: [erlang-questions] listen on multiple interfaces In-Reply-To: References: Message-ID: On 2016-12-09 07:50, Benoit Chesneau wrote: > that will listen on all interfaces though. i'm looking for a way to listen > on a set of interfaces in fact, sorry to not have been clear :) Yes, that works fine, you just need to create a listen socket for each. That's how sockets work, nothing to do with gen_tcp or Erlang. And to be more precise, despite large amounts of text that would have you believe otherwise, you can't listen on interfaces, only on IP addresses (an interface may have many IP addresses configured, and except for the wildcard case, you need a listen socket for each if you want them all - and an interface may of course also have *no* IP addresses configured, in which case you don't have anything to listen on for that interface). --Per Hedeland From serge@REDACTED Fri Dec 9 14:15:39 2016 From: serge@REDACTED (Serge Aleynikov) Date: Fri, 9 Dec 2016 08:15:39 -0500 Subject: [erlang-questions] Google Drive API Message-ID: Hi, Is there any open source project that implements Google Drive API? I've seen this one, but it's not functional: https://github.com/mithereal/erlang-googledrive. Thank you. Serge -------------- next part -------------- An HTML attachment was scrubbed... URL: From Dinislam.Salikhov@REDACTED Fri Dec 9 14:19:05 2016 From: Dinislam.Salikhov@REDACTED (Salikhov Dinislam) Date: Fri, 9 Dec 2016 16:19:05 +0300 Subject: [erlang-questions] erts_port_control() and binp Message-ID: <06de95fa-dc34-a175-50fe-22d5f3c2d7a9@kaspersky.com> Hello! I'm looking through VM sources now. In erts/emulator/beam/erl_port.h there is a 'struct ErtsProc2PortSigData_' with a union as the last field. One of the values of this union is 'struct control'. I'm interesting in the field binp of the struct control. I grepped through the sources and found that the struct is used only in a few places in erts/emulator/beam/io.c Except the places where the field is set, it's used only as an argument of cleanup_scheduled_control() in the same file. So what's the meaning of the field binp? What is it needed for? Salikhov Dinislam From mattevans123@REDACTED Fri Dec 9 15:56:25 2016 From: mattevans123@REDACTED (Matthew Evans) Date: Fri, 9 Dec 2016 14:56:25 +0000 Subject: [erlang-questions] R17 - Possible wedged scheduler Message-ID: Hi, We just hit a situation where it appeared that 1 scheduler was wedged. Some parts of our application were working, but others appeared to be stuck. I could connect via a cnode application and an escript, but I couldn't connect via the Erlang shell. We have an escript that does rpc calls, some worked, others (e.g. anything to the code server or tracing failed) failed. CPU load was minimal at the time, and heart didn't complain. We only have a single NIF, but this is not called on this hardware variant. We do use CNODE to talk to C applications. We are running R17, Intel quad core CPU on Debian. This is the first time this has been seen, so the questions are: 1. Has anyone seen this before? 2. What can we do if we hit this condition in the future to debug? 3. Since heart doesn't detect this can anyone think of any alternative mechanisms? Thanks Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From Dinislam.Salikhov@REDACTED Fri Dec 9 16:00:15 2016 From: Dinislam.Salikhov@REDACTED (Salikhov Dinislam) Date: Fri, 9 Dec 2016 18:00:15 +0300 Subject: [erlang-questions] erts_port_control() and binp In-Reply-To: <06de95fa-dc34-a175-50fe-22d5f3c2d7a9@kaspersky.com> References: <06de95fa-dc34-a175-50fe-22d5f3c2d7a9@kaspersky.com> Message-ID: <34124044-cab6-3b10-932e-b95f002cd2fc@kaspersky.com> Finally, I've figured it out: bufp either contains a copy of data passed to port as binary or points to the data inside the passed binary. In the latter case refcount of the binary is incremented and the pointer to binary is kept in binp field, so the buffer in use wouldn't be garbage collected. Salikhov Dinislam On 12/09/2016 04:19 PM, Salikhov Dinislam wrote: > Hello! > > I'm looking through VM sources now. > In erts/emulator/beam/erl_port.h there is a 'struct > ErtsProc2PortSigData_' with a union as the last field. > One of the values of this union is 'struct control'. > I'm interesting in the field binp of the struct control. > > I grepped through the sources and found that the struct is used only > in a few places in erts/emulator/beam/io.c > Except the places where the field is set, it's used only as an > argument of cleanup_scheduled_control() in the same file. > So what's the meaning of the field binp? What is it needed for? > > Salikhov Dinislam From Dinislam.Salikhov@REDACTED Fri Dec 9 16:14:40 2016 From: Dinislam.Salikhov@REDACTED (Salikhov Dinislam) Date: Fri, 9 Dec 2016 18:14:40 +0300 Subject: [erlang-questions] erts_port_control() and binp In-Reply-To: References: <06de95fa-dc34-a175-50fe-22d5f3c2d7a9@kaspersky.com> <34124044-cab6-3b10-932e-b95f002cd2fc@kaspersky.com> Message-ID: <22c3d55d-eb95-679a-9512-35a057f26823@kaspersky.com> See ./erts/emulator/beam/io.c erts_port_control() calls erts_refc_inc() if original binary should be used. Then the pointer to binary is kept in sigdp->u.control.binp. When the data is not needed any more, cleanup_scheduled_control() is called. It calls erts_refc_dectest() if original binary is used. On 12/09/2016 06:02 PM, Frank Muller wrote: > Which code is responsible of incr/decr binary refcount? > > > Le ven. 9 d?c. 2016 ? 16:01, Salikhov Dinislam > > a ?crit : > > Finally, I've figured it out: > > bufp either contains a copy of data passed to port as binary or points > > to the data inside the passed binary. > > In the latter case refcount of the binary is incremented and the > pointer > > to binary is kept in binp field, so the buffer in use wouldn't be > > garbage collected. > > > > Salikhov Dinislam > > > > On 12/09/2016 04:19 PM, Salikhov Dinislam wrote: > > > Hello! > > > > > > I'm looking through VM sources now. > > > In erts/emulator/beam/erl_port.h there is a 'struct > > > ErtsProc2PortSigData_' with a union as the last field. > > > One of the values of this union is 'struct control'. > > > I'm interesting in the field binp of the struct control. > > > > > > I grepped through the sources and found that the struct is used only > > > in a few places in erts/emulator/beam/io.c > > > Except the places where the field is set, it's used only as an > > > argument of cleanup_scheduled_control() in the same file. > > > So what's the meaning of the field binp? What is it needed for? > > > > > > Salikhov Dinislam > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattevans123@REDACTED Fri Dec 9 17:35:21 2016 From: mattevans123@REDACTED (Matthew Evans) Date: Fri, 9 Dec 2016 16:35:21 +0000 Subject: [erlang-questions] R17 - Possible wedged scheduler In-Reply-To: References: Message-ID: Happened again, it appears that code_server is wedged: admin@REDACTED:~$ doErlangFun "erlang:process_info(whereis(code_server))." [{registered_name,code_server}, {current_function,{code_server,cpc_recv,4}}, {initial_call,{erlang,apply,2}}, {status,waiting}, {message_queue_len,23}, {messages,[{code_call,<6805.4097.0>,{ensure_loaded,switch_type_module}}, {code_call,<6805.4146.0>,{ensure_loaded,switch_type_module}}, {code_call,<6805.941.0>,{ensure_loaded,pc_port_autoneg}}, {code_call,<6805.541.0>,{ensure_loaded,plexxiStatistics_types}}, {code_call,<6805.520.0>,{ensure_loaded,switch_type_module}}, {code_call,<6805.5123.0>,{ensure_loaded,secondary_erlang_node}}, {code_call,<6805.5122.0>,{ensure_loaded,secondary_erlang_node}}, {code_call,<6805.5162.0>,{ensure_loaded,icmp}}, {code_call,<6805.5321.0>, {ensure_loaded,mac_entries_record_handler}}, {code_call,<6805.5483.0>,{ensure_loaded,icmp}}, {code_call,<6805.6647.0>,{ensure_loaded,icmp}}, {code_call,<6805.7232.0>,{ensure_loaded,icmp}}, {code_call,<6805.7274.0>,{ensure_loaded,icmp}}, {code_call,<6805.7304.0>,{ensure_loaded,icmp}}, {code_call,<6805.8889.0>, {ensure_loaded,mac_entries_record_handler}}, {code_call,<6805.8951.0>, {ensure_loaded,mac_entries_record_handler}}, {code_call,<6805.576.0>, {ensure_loaded,cross_connect_unicast_utils}}, {code_call,<6805.19300.12>,{ensure_loaded,shell}}, {code_call,<6805.20313.12>,{ensure_loaded,shell}}, {code_call,<6805.21339.12>,{ensure_loaded,dbg}}, {code_call,<6805.31109.13>,get_mode}, {code_call,<6805.1255.14>,get_mode}, {system,{<6805.2521.14>,#Ref<6805.0.23.35356>},get_status}]}, {links,[<6805.11.0>]}, {dictionary,[{any_native_code_loaded,false}]}, {trap_exit,true}, {error_handler,error_handler}, {priority,normal}, {group_leader,<6805.9.0>}, {total_heap_size,86071}, {heap_size,10958}, {stack_size,25}, {reductions,13172282}, {garbage_collection,[{min_bin_vheap_size,46422}, {min_heap_size,233}, {fullsweep_after,65535}, {minor_gcs,71}]}, {suspending,[]}] admin@REDACTED:~$ ________________________________ From: erlang-questions-bounces@REDACTED on behalf of Matthew Evans Sent: Friday, December 9, 2016 9:56 AM To: Erlang/OTP discussions Subject: [erlang-questions] R17 - Possible wedged scheduler Hi, We just hit a situation where it appeared that 1 scheduler was wedged. Some parts of our application were working, but others appeared to be stuck. I could connect via a cnode application and an escript, but I couldn't connect via the Erlang shell. We have an escript that does rpc calls, some worked, others (e.g. anything to the code server or tracing failed) failed. CPU load was minimal at the time, and heart didn't complain. We only have a single NIF, but this is not called on this hardware variant. We do use CNODE to talk to C applications. We are running R17, Intel quad core CPU on Debian. This is the first time this has been seen, so the questions are: 1. Has anyone seen this before? 2. What can we do if we hit this condition in the future to debug? 3. Since heart doesn't detect this can anyone think of any alternative mechanisms? Thanks Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Fri Dec 9 18:09:14 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 9 Dec 2016 20:09:14 +0300 Subject: [erlang-questions] R17 - Possible wedged scheduler In-Reply-To: References: Message-ID: Is your disk ok? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Fri Dec 9 19:03:55 2016 From: mjtruog@REDACTED (Michael Truog) Date: Fri, 9 Dec 2016 10:03:55 -0800 Subject: [erlang-questions] R17 - Possible wedged scheduler In-Reply-To: References: Message-ID: <584AF20B.4050202@gmail.com> If this is scheduler collapse, it would mean you have a port driver or NIF that has internal latency greater than 1 millisecond. To handle scheduler collapse, you can use the erl command line option "-heart" combined with: heart:set_options([check_schedulers]). (see http://erlang.org/doc/man/heart.html#set_options-1 ) That will allow the system to restart when schedulers have collapsed. A test that is meant to cause scheduler collapse is at: https://github.com/basho/nifwait So you could use that to prove to yourself that the behavior is the same with the system you are having problems with. On 12/09/2016 08:35 AM, Matthew Evans wrote: > > Happened again, it appears that code_server is wedged: > > > admin@REDACTED:~$ doErlangFun "erlang:process_info(whereis(code_server))." > > [{registered_name,code_server}, > > {current_function,{code_server,cpc_recv,4}}, > > {initial_call,{erlang,apply,2}}, > > {status,waiting}, > > {message_queue_len,23}, > > {messages,[{code_call,<6805.4097.0>,{ensure_loaded,switch_type_module}}, > > {code_call,<6805.4146.0>,{ensure_loaded,switch_type_module}}, > > {code_call,<6805.941.0>,{ensure_loaded,pc_port_autoneg}}, > > {code_call,<6805.541.0>,{ensure_loaded,plexxiStatistics_types}}, > > {code_call,<6805.520.0>,{ensure_loaded,switch_type_module}}, > > {code_call,<6805.5123.0>,{ensure_loaded,secondary_erlang_node}}, > > {code_call,<6805.5122.0>,{ensure_loaded,secondary_erlang_node}}, > > {code_call,<6805.5162.0>,{ensure_loaded,icmp}}, > > {code_call,<6805.5321.0>, > > {ensure_loaded,mac_entries_record_handler}}, > > {code_call,<6805.5483.0>,{ensure_loaded,icmp}}, > > {code_call,<6805.6647.0>,{ensure_loaded,icmp}}, > > {code_call,<6805.7232.0>,{ensure_loaded,icmp}}, > > {code_call,<6805.7274.0>,{ensure_loaded,icmp}}, > > {code_call,<6805.7304.0>,{ensure_loaded,icmp}}, > > {code_call,<6805.8889.0>, > > {ensure_loaded,mac_entries_record_handler}}, > > {code_call,<6805.8951.0>, > > {ensure_loaded,mac_entries_record_handler}}, > > {code_call,<6805.576.0>, > > {ensure_loaded,cross_connect_unicast_utils}}, > > {code_call,<6805.19300.12>,{ensure_loaded,shell}}, > > {code_call,<6805.20313.12>,{ensure_loaded,shell}}, > > {code_call,<6805.21339.12>,{ensure_loaded,dbg}}, > > {code_call,<6805.31109.13>,get_mode}, > > {code_call,<6805.1255.14>,get_mode}, > > {system,{<6805.2521.14>,#Ref<6805.0.23.35356>},get_status}]}, > > {links,[<6805.11.0>]}, > > {dictionary,[{any_native_code_loaded,false}]}, > > {trap_exit,true}, > > {error_handler,error_handler}, > > {priority,normal}, > > {group_leader,<6805.9.0>}, > > {total_heap_size,86071}, > > {heap_size,10958}, > > {stack_size,25}, > > {reductions,13172282}, > > {garbage_collection,[{min_bin_vheap_size,46422}, > > {min_heap_size,233}, > > {fullsweep_after,65535}, > > {minor_gcs,71}]}, > > {suspending,[]}] > > admin@REDACTED:~$ > > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > *From:* erlang-questions-bounces@REDACTED on behalf of Matthew Evans > *Sent:* Friday, December 9, 2016 9:56 AM > *To:* Erlang/OTP discussions > *Subject:* [erlang-questions] R17 - Possible wedged scheduler > > Hi, > > > We just hit a situation where it appeared that 1 scheduler was wedged. Some parts of our application were working, but others appeared to be stuck. I could connect via a cnode application and an escript, but I couldn't connect via the Erlang shell. We have an escript that does rpc calls, some worked, others (e.g. anything to the code server or tracing failed) failed. > > > CPU load was minimal at the time, and heart didn't complain. We only have a single NIF, but this is not called on this hardware variant. We do use CNODE to talk to C applications. > > > We are running R17, Intel quad core CPU on Debian. > > > This is the first time this has been seen, so the questions are: > > > 1. Has anyone seen this before? > > 2. What can we do if we hit this condition in the future to debug? > > 3. Since heart doesn't detect this can anyone think of any alternative mechanisms? > > > Thanks > > > Matt > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattevans123@REDACTED Fri Dec 9 19:31:36 2016 From: mattevans123@REDACTED (Matthew Evans) Date: Fri, 9 Dec 2016 18:31:36 +0000 Subject: [erlang-questions] R17 - Possible wedged scheduler In-Reply-To: <584AF20B.4050202@gmail.com> References: , <584AF20B.4050202@gmail.com> Message-ID: Thanks, I'll give that a go, Looking at code_server.erl it does appear that there exists a possibility of a deadlock. In so much as function cpc_recv/4 does a receive with no timeout. Thanks again Matt ________________________________ From: Michael Truog Sent: Friday, December 9, 2016 1:03 PM To: Matthew Evans; Erlang/OTP discussions Subject: Re: [erlang-questions] R17 - Possible wedged scheduler If this is scheduler collapse, it would mean you have a port driver or NIF that has internal latency greater than 1 millisecond. To handle scheduler collapse, you can use the erl command line option "-heart" combined with: heart:set_options([check_schedulers]). (see http://erlang.org/doc/man/heart.html#set_options-1 ) That will allow the system to restart when schedulers have collapsed. A test that is meant to cause scheduler collapse is at: https://github.com/basho/nifwait [https://avatars0.githubusercontent.com/u/176293?v=3&s=400] GitHub - basho/nifwait: Utility to test effect of blocking ... github.com README.md Utility to test effect of blocking NIF on Erlang scheduler. Test spawns several processes, all which will start in the run queue of the current scheduler. So you could use that to prove to yourself that the behavior is the same with the system you are having problems with. On 12/09/2016 08:35 AM, Matthew Evans wrote: Happened again, it appears that code_server is wedged: admin@REDACTED:~$ doErlangFun "erlang:process_info(whereis(code_server))." [{registered_name,code_server}, {current_function,{code_server,cpc_recv,4}}, {initial_call,{erlang,apply,2}}, {status,waiting}, {message_queue_len,23}, {messages,[{code_call,<6805.4097.0>,{ensure_loaded,switch_type_module}}, {code_call,<6805.4146.0>,{ensure_loaded,switch_type_module}}, {code_call,<6805.941.0>,{ensure_loaded,pc_port_autoneg}}, {code_call,<6805.541.0>,{ensure_loaded,plexxiStatistics_types}}, {code_call,<6805.520.0>,{ensure_loaded,switch_type_module}}, {code_call,<6805.5123.0>,{ensure_loaded,secondary_erlang_node}}, {code_call,<6805.5122.0>,{ensure_loaded,secondary_erlang_node}}, {code_call,<6805.5162.0>,{ensure_loaded,icmp}}, {code_call,<6805.5321.0>, {ensure_loaded,mac_entries_record_handler}}, {code_call,<6805.5483.0>,{ensure_loaded,icmp}}, {code_call,<6805.6647.0>,{ensure_loaded,icmp}}, {code_call,<6805.7232.0>,{ensure_loaded,icmp}}, {code_call,<6805.7274.0>,{ensure_loaded,icmp}}, {code_call,<6805.7304.0>,{ensure_loaded,icmp}}, {code_call,<6805.8889.0>, {ensure_loaded,mac_entries_record_handler}}, {code_call,<6805.8951.0>, {ensure_loaded,mac_entries_record_handler}}, {code_call,<6805.576.0>, {ensure_loaded,cross_connect_unicast_utils}}, {code_call,<6805.19300.12>,{ensure_loaded,shell}}, {code_call,<6805.20313.12>,{ensure_loaded,shell}}, {code_call,<6805.21339.12>,{ensure_loaded,dbg}}, {code_call,<6805.31109.13>,get_mode}, {code_call,<6805.1255.14>,get_mode}, {system,{<6805.2521.14>,#Ref<6805.0.23.35356>},get_status}]}, {links,[<6805.11.0>]}, {dictionary,[{any_native_code_loaded,false}]}, {trap_exit,true}, {error_handler,error_handler}, {priority,normal}, {group_leader,<6805.9.0>}, {total_heap_size,86071}, {heap_size,10958}, {stack_size,25}, {reductions,13172282}, {garbage_collection,[{min_bin_vheap_size,46422}, {min_heap_size,233}, {fullsweep_after,65535}, {minor_gcs,71}]}, {suspending,[]}] admin@REDACTED:~$ ________________________________ From: erlang-questions-bounces@REDACTED on behalf of Matthew Evans Sent: Friday, December 9, 2016 9:56 AM To: Erlang/OTP discussions Subject: [erlang-questions] R17 - Possible wedged scheduler Hi, We just hit a situation where it appeared that 1 scheduler was wedged. Some parts of our application were working, but others appeared to be stuck. I could connect via a cnode application and an escript, but I couldn't connect via the Erlang shell. We have an escript that does rpc calls, some worked, others (e.g. anything to the code server or tracing failed) failed. CPU load was minimal at the time, and heart didn't complain. We only have a single NIF, but this is not called on this hardware variant. We do use CNODE to talk to C applications. We are running R17, Intel quad core CPU on Debian. This is the first time this has been seen, so the questions are: 1. Has anyone seen this before? 2. What can we do if we hit this condition in the future to debug? 3. Since heart doesn't detect this can anyone think of any alternative mechanisms? Thanks Matt _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.14159@REDACTED Fri Dec 9 19:35:59 2016 From: j.14159@REDACTED (Jeremy Pierre) Date: Fri, 09 Dec 2016 18:35:59 +0000 Subject: [erlang-questions] Alpaca (formerly MLFE) v0.2.5 released Message-ID: Hi all, Alpaca (formerly MLFE/ML-flavoured Erlang) v0.2.5 is now available. Alpaca is an ML-inspired functional programming language for the Erlang VM. Source here: https://github.com/alpaca-lang/alpaca v0.2.5 tag: https://github.com/alpaca-lang/alpaca/tree/v0.2.5 Big changes in this release: - project rename - early infix function support (operators, e.g. `|>`) - patterns in function arguments (e.g. `opt_map f Some x = Some (f x)`) Please see https://github.com/alpaca-lang/alpaca/blob/master/ChangeLog.org for the full list of changes including contributors. Please join us in the #alpaca-lang IRC channel on freenode for help and/or discussion of the language and its direction. Special thanks to everyone who participated in the renaming discussion and in particular Tilman Holschuh for suggesting the name "Alpaca". Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From Oliver.Korpilla@REDACTED Fri Dec 9 19:52:07 2016 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Fri, 9 Dec 2016 19:52:07 +0100 Subject: [erlang-questions] Alpaca (formerly MLFE) v0.2.5 released In-Reply-To: References: Message-ID: Hello, Jeremy. Love the name, curious about the concept, will keep a tab on this. :) All the best, Oliver? Gesendet:?Freitag, 09. Dezember 2016 um 19:35 Uhr Von:?"Jeremy Pierre" An:?"erlang-questions@REDACTED" Betreff:?[erlang-questions] Alpaca (formerly MLFE) v0.2.5 released Hi all, ? Alpaca (formerly MLFE/ML-flavoured Erlang) v0.2.5 is now available.? Alpaca is an ML-inspired functional programming language for the Erlang VM. ? Source here: ?https://github.com/alpaca-lang/alpaca v0.2.5 tag: ?https://github.com/alpaca-lang/alpaca/tree/v0.2.5[https://github.com/alpaca-lang/alpaca/tree/v0.2.5] ? Big changes in this release: - project rename - early infix function support (operators, e.g. `|>`) - patterns in function arguments (e.g. `opt_map f Some x = Some (f x)`) ? Please see https://github.com/alpaca-lang/alpaca/blob/master/ChangeLog.org[https://github.com/alpaca-lang/alpaca/blob/master/ChangeLog.org] for the full list of changes including contributors. ? Please join us in the #alpaca-lang IRC channel on freenode for help and/or discussion of the language and its direction. ? Special thanks to everyone who participated in the renaming discussion and in particular Tilman Holschuh for suggesting the name "Alpaca". ? Jeremy_______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions[http://erlang.org/mailman/listinfo/erlang-questions] From frank.muller.erl@REDACTED Sat Dec 10 00:15:58 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 09 Dec 2016 23:15:58 +0000 Subject: [erlang-questions] Fast directory walker Message-ID: Hi I would like to improve the speed of my directory walker. walk(Dir) -> {ok, Files} = prim_file:list_dir(Dir), walk(Dir, Files). walk(Dir, [ Basename | Rest ]) -> Path = filename:join([ Dir, Basename ]), case filelib:is_dir(Path) of true -> walk(Path); false -> io:format("~s~n", [Path]), filelib:file_size(Path) end, walk(Dir, Rest); walk(_, []) -> ok. Compared to almost anything i found on the web, it?s still very slow: > timer:tc(fun() -> dir:walk("/usr/share") end). {4662361,ok} The idea behind it is to build something similar to The Platinum Searcher (in Go, extremely fast): https://github.com/monochromegane/the_platinum_searcher Advices very appreciated on how to improve its speed. /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang.org@REDACTED Sat Dec 10 00:50:27 2016 From: erlang.org@REDACTED (Stanislaw Klekot) Date: Sat, 10 Dec 2016 00:50:27 +0100 Subject: [erlang-questions] Fast directory walker In-Reply-To: References: Message-ID: <20161209235027.GA2814@jarowit.net> On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller wrote: > I would like to improve the speed of my directory walker. > > walk(Dir) -> > {ok, Files} = prim_file:list_dir(Dir), > walk(Dir, Files). Why prim_file:list_dir() instead of file:list_dir()? The former is undocumented internal function. [...] > Compared to almost anything i found on the web, it?s still very slow: > > timer:tc(fun() -> dir:walk("/usr/share") end). > {4662361,ok} What is it this "anything you found on the web"? And how did you run your comparisons? There's a large difference between first and second consequent run caused by OS' directory cache, and there's large difference between simply walking through the directory and walking with printing something to the screen for every file. Then there's also your using filelib:is_dir() and then filelib:file_size(), which means two stat(2) calls, while you only need to do it once per file (file:read_file_info()). -- Stanislaw Klekot From mjtruog@REDACTED Sat Dec 10 03:12:19 2016 From: mjtruog@REDACTED (Michael Truog) Date: Fri, 9 Dec 2016 18:12:19 -0800 Subject: [erlang-questions] Fast directory walker In-Reply-To: References: Message-ID: <584B6483.70206@gmail.com> On 12/09/2016 03:15 PM, Frank Muller wrote: > Hi > > I would like to improve the speed of my directory walker. > > walk(Dir) -> > {ok, Files} = prim_file:list_dir(Dir), > walk(Dir, Files). > > walk(Dir, [ Basename | Rest ]) -> > Path = filename:join([ Dir, Basename ]), > case filelib:is_dir(Path) of > true -> > walk(Path); > false -> > io:format("~s~n", [Path]), > filelib:file_size(Path) > end, > walk(Dir, Rest); > walk(_, []) -> > ok. > > > Compared to almost anything i found on the web, it?s still very slow: > > timer:tc(fun() -> dir:walk("/usr/share") end). > {4662361 ,ok} Have you tried filelib:fold_files/5 (http://erlang.org/doc/man/filelib.html#fold_files-5) ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Sat Dec 10 09:28:44 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sat, 10 Dec 2016 08:28:44 +0000 Subject: [erlang-questions] Fast directory walker In-Reply-To: <20161209235027.GA2814@jarowit.net> References: <20161209235027.GA2814@jarowit.net> Message-ID: Hi Stanislaw First, I don't care if I've to use documented/undocumented calls as long as I can achieve my goal: faster dir walking. And you're right, here is a detailed comparison with other scripting languages: In my /usr/share, there?s: 2580 directories 28953 files 1. Erlang (no io:format/1, just recurse): walk(Dir) -> {ok, Files} = file:list_dir(Dir), walk(Dir, Files). walk(Dir, [ Basename | Rest ]) -> Path = filename:join([ Dir, Basename ]), case filelib:is_dir(Path) of true -> walk(Path); false -> %% io:format("~s~n", [Path]), filelib:file_size(Path) end, walk(Dir, Rest); walk(_, []) -> ok. timer:tc(fun() -> directoy:walker("/usr/share") end). {4662361,ok} 2. Python (this code even count the size of dir): From: http://stackoverflow.com/questions/1392413/calculating-a-directory-size-using-python import os def get_size(start_path = '.'): total_size = 0 for dirpath, dirnames, filenames in os.walk(start_path): for f in filenames: fp = os.path.join(dirpath, f) total_size += os.path.getsize(fp) return total_size print get_size() $ cd /usr/share $ time dir_walker.py 432034130 0.25 real 0.13 user 0.10 sys 2. Perl (same, count dir size) http://www.perlmonks.org/?node_id=168974 use File::Find; my $size = 0; find(sub { $size += -s if -f $_ }, "/usr/share"); $ time perl dir_walker.pl 432034130 0.13 real 0.05 user 0.08 sys 3. Ruby (same, count dir size): def directory_size(path) path << '/' unless path.end_with?('/') raise RuntimeError, "#{path} is not a directory" unless File.directory?(path) total_size = 0 Dir["#{path}**/*"].each do |f| total_size += File.size(f) if File.file?(f) && File.size?(f) end total_size end puts directory_size '/usr/share? $ time walker.rb 432028422 0.21 real 0.09 user 0.11 sys 4. Lua: From: http://lua-users.org/wiki/DirTreeIterator require "lfs" function dirtree(dir) assert(dir and dir ~= "", "directory parameter is missing or empty") if string.sub(dir, -1) == "/" then dir=string.sub(dir, 1, -2) end local function yieldtree(dir) for entry in lfs.dir(dir) do if entry ~= "." and entry ~= ".." then entry=dir.."/"..entry local attr=lfs.attributes(entry) coroutine.yield(entry,attr) if attr.mode == "directory" then yieldtree(entry) end end end end return coroutine.wrap(function() yieldtree(dir) end) end for filename, attr in dirtree("/usr/share") do print(attr.mode, filename) end $ luarocks install luafilesystem $ time lua walker.lua > /dev/null 0.30 real 0.16 user 0.14 sys Do you need more? Thanks for you help. /Frank Le sam. 10 d?c. 2016 ? 00:51, Stanislaw Klekot a ?crit : > On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller wrote: > > > I would like to improve the speed of my directory walker. > > > > > > walk(Dir) -> > > > {ok, Files} = prim_file:list_dir(Dir), > > > walk(Dir, Files). > > > > Why prim_file:list_dir() instead of file:list_dir()? The former is > > undocumented internal function. > > > > [...] > > > Compared to almost anything i found on the web, it?s still very slow: > > > > timer:tc(fun() -> dir:walk("/usr/share") end). > > > {4662361,ok} > > > > What is it this "anything you found on the web"? And how did you run > > your comparisons? There's a large difference between first and second > > consequent run caused by OS' directory cache, and there's large > > difference between simply walking through the directory and walking with > > printing something to the screen for every file. > > > > Then there's also your using filelib:is_dir() and then > > filelib:file_size(), which means two stat(2) calls, while you only need > > to do it once per file (file:read_file_info()). > > > > -- > > Stanislaw Klekot > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sat Dec 10 09:34:24 2016 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 10 Dec 2016 08:34:24 +0000 Subject: [erlang-questions] Fast directory walker In-Reply-To: References: <20161209235027.GA2814@jarowit.net> Message-ID: did you try with the wildcard function? something like: https://github.com/benoitc/esync/blob/master/src/esync.erl#L151 On Sat, 10 Dec 2016 at 09:29, Frank Muller wrote: > Hi Stanislaw > > First, I don't care if I've to use documented/undocumented calls as long > as I can achieve my goal: faster dir walking. > > And you're right, here is a detailed comparison with other scripting > languages: > > In my /usr/share, there?s: > 2580 directories > 28953 files > > 1. Erlang (no io:format/1, just recurse): > > walk(Dir) -> > {ok, Files} = file:list_dir(Dir), > walk(Dir, Files). > > walk(Dir, [ Basename | Rest ]) -> > Path = filename:join([ Dir, Basename ]), > case filelib:is_dir(Path) of > true -> > walk(Path); > false -> > %% io:format("~s~n", [Path]), > filelib:file_size(Path) > end, > walk(Dir, Rest); > walk(_, []) -> > ok. > > timer:tc(fun() -> directoy:walker("/usr/share") end). > {4662361,ok} > > 2. Python (this code even count the size of dir): > From: > http://stackoverflow.com/questions/1392413/calculating-a-directory-size-using-python > > import os > def get_size(start_path = '.'): > total_size = 0 > for dirpath, dirnames, filenames in os.walk(start_path): > for f in filenames: > fp = os.path.join(dirpath, f) > total_size += os.path.getsize(fp) > return total_size > > print get_size() > > $ cd /usr/share > $ time dir_walker.py > 432034130 > 0.25 real 0.13 user 0.10 sys > > 2. Perl (same, count dir size) > http://www.perlmonks.org/?node_id=168974 > > use File::Find; > my $size = 0; > find(sub { $size += -s if -f $_ }, "/usr/share"); > > $ time perl dir_walker.pl > 432034130 > 0.13 real 0.05 user 0.08 sys > > 3. Ruby (same, count dir size): > > def directory_size(path) > path << '/' unless path.end_with?('/') > raise RuntimeError, "#{path} is not a directory" unless > File.directory?(path) > total_size = 0 > Dir["#{path}**/*"].each do |f| > total_size += File.size(f) if File.file?(f) && File.size?(f) > end > total_size > end > puts directory_size '/usr/share? > > $ time walker.rb > 432028422 > 0.21 real 0.09 user 0.11 sys > > 4. Lua: > From: http://lua-users.org/wiki/DirTreeIterator > > require "lfs" > > function dirtree(dir) > assert(dir and dir ~= "", "directory parameter is missing or empty") > if string.sub(dir, -1) == "/" then > dir=string.sub(dir, 1, -2) > end > > local function yieldtree(dir) > for entry in lfs.dir(dir) do > if entry ~= "." and entry ~= ".." then > entry=dir.."/"..entry > local attr=lfs.attributes(entry) > coroutine.yield(entry,attr) > if attr.mode == "directory" then > yieldtree(entry) > end > end > end > end > > return coroutine.wrap(function() yieldtree(dir) end) > end > > for filename, attr in dirtree("/usr/share") do > print(attr.mode, filename) > end > > $ luarocks install luafilesystem > $ time lua walker.lua > /dev/null > 0.30 real 0.16 user 0.14 sys > > Do you need more? > > Thanks for you help. > /Frank > > Le sam. 10 d?c. 2016 ? 00:51, Stanislaw Klekot a > ?crit : > > On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller wrote: > > > I would like to improve the speed of my directory walker. > > > > > > walk(Dir) -> > > > {ok, Files} = prim_file:list_dir(Dir), > > > walk(Dir, Files). > > > > Why prim_file:list_dir() instead of file:list_dir()? The former is > > undocumented internal function. > > > > [...] > > > Compared to almost anything i found on the web, it?s still very slow: > > > > timer:tc(fun() -> dir:walk("/usr/share") end). > > > {4662361,ok} > > > > What is it this "anything you found on the web"? And how did you run > > your comparisons? There's a large difference between first and second > > consequent run caused by OS' directory cache, and there's large > > difference between simply walking through the directory and walking with > > printing something to the screen for every file. > > > > Then there's also your using filelib:is_dir() and then > > filelib:file_size(), which means two stat(2) calls, while you only need > > to do it once per file (file:read_file_info()). > > > > -- > > Stanislaw Klekot > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Sat Dec 10 09:36:54 2016 From: sergej.jurecko@REDACTED (=?UTF-8?Q?Sergej_Jure=C4=8Dko?=) Date: Sat, 10 Dec 2016 09:36:54 +0100 Subject: [erlang-questions] Fast directory walker In-Reply-To: References: <20161209235027.GA2814@jarowit.net> Message-ID: Stop using filelib functions. Use file:read_file_info and file:list_dir. Sergej On Dec 10, 2016 9:29 AM, "Frank Muller" wrote: > Hi Stanislaw > > First, I don't care if I've to use documented/undocumented calls as long > as I can achieve my goal: faster dir walking. > > And you're right, here is a detailed comparison with other scripting > languages: > > In my /usr/share, there?s: > 2580 directories > 28953 files > > 1. Erlang (no io:format/1, just recurse): > > walk(Dir) -> > {ok, Files} = file:list_dir(Dir), > walk(Dir, Files). > > walk(Dir, [ Basename | Rest ]) -> > Path = filename:join([ Dir, Basename ]), > case filelib:is_dir(Path) of > true -> > walk(Path); > false -> > %% io:format("~s~n", [Path]), > filelib:file_size(Path) > end, > walk(Dir, Rest); > walk(_, []) -> > ok. > > timer:tc(fun() -> directoy:walker("/usr/share") end). > {4662361,ok} > > 2. Python (this code even count the size of dir): > From: http://stackoverflow.com/questions/1392413/ > calculating-a-directory-size-using-python > > import os > def get_size(start_path = '.'): > total_size = 0 > for dirpath, dirnames, filenames in os.walk(start_path): > for f in filenames: > fp = os.path.join(dirpath, f) > total_size += os.path.getsize(fp) > return total_size > > print get_size() > > $ cd /usr/share > $ time dir_walker.py > 432034130 > 0.25 real 0.13 user 0.10 sys > > 2. Perl (same, count dir size) > http://www.perlmonks.org/?node_id=168974 > > use File::Find; > my $size = 0; > find(sub { $size += -s if -f $_ }, "/usr/share"); > > $ time perl dir_walker.pl > 432034130 > 0.13 real 0.05 user 0.08 sys > > 3. Ruby (same, count dir size): > > def directory_size(path) > path << '/' unless path.end_with?('/') > raise RuntimeError, "#{path} is not a directory" unless > File.directory?(path) > total_size = 0 > Dir["#{path}**/*"].each do |f| > total_size += File.size(f) if File.file?(f) && File.size?(f) > end > total_size > end > puts directory_size '/usr/share? > > $ time walker.rb > 432028422 > 0.21 real 0.09 user 0.11 sys > > 4. Lua: > From: http://lua-users.org/wiki/DirTreeIterator > > require "lfs" > > function dirtree(dir) > assert(dir and dir ~= "", "directory parameter is missing or empty") > if string.sub(dir, -1) == "/" then > dir=string.sub(dir, 1, -2) > end > > local function yieldtree(dir) > for entry in lfs.dir(dir) do > if entry ~= "." and entry ~= ".." then > entry=dir.."/"..entry > local attr=lfs.attributes(entry) > coroutine.yield(entry,attr) > if attr.mode == "directory" then > yieldtree(entry) > end > end > end > end > > return coroutine.wrap(function() yieldtree(dir) end) > end > > for filename, attr in dirtree("/usr/share") do > print(attr.mode, filename) > end > > $ luarocks install luafilesystem > $ time lua walker.lua > /dev/null > 0.30 real 0.16 user 0.14 sys > > Do you need more? > > Thanks for you help. > /Frank > > Le sam. 10 d?c. 2016 ? 00:51, Stanislaw Klekot a > ?crit : > >> On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller wrote: >> >> > I would like to improve the speed of my directory walker. >> >> > >> >> > walk(Dir) -> >> >> > {ok, Files} = prim_file:list_dir(Dir), >> >> > walk(Dir, Files). >> >> >> >> Why prim_file:list_dir() instead of file:list_dir()? The former is >> >> undocumented internal function. >> >> >> >> [...] >> >> > Compared to almost anything i found on the web, it?s still very slow: >> >> > > timer:tc(fun() -> dir:walk("/usr/share") end). >> >> > {4662361,ok} >> >> >> >> What is it this "anything you found on the web"? And how did you run >> >> your comparisons? There's a large difference between first and second >> >> consequent run caused by OS' directory cache, and there's large >> >> difference between simply walking through the directory and walking with >> >> printing something to the screen for every file. >> >> >> >> Then there's also your using filelib:is_dir() and then >> >> filelib:file_size(), which means two stat(2) calls, while you only need >> >> to do it once per file (file:read_file_info()). >> >> >> >> -- >> >> Stanislaw Klekot >> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sat Dec 10 09:42:20 2016 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 10 Dec 2016 08:42:20 +0000 Subject: [erlang-questions] Fast directory walker In-Reply-To: References: <20161209235027.GA2814@jarowit.net> Message-ID: this is kind of bullshit (sorry ;).... at the end this is what does the helpers in filelib: https://github.com/erlang/otp/blob/maint/lib/stdlib/src/filelib.erl#L257 except if you have a better algorithm in mind i don't se the point of rewriting something that is aleaready existing ... On Sat, 10 Dec 2016 at 09:36, Sergej Jure?ko wrote: > Stop using filelib functions. Use file:read_file_info and file:list_dir. > > Sergej > > On Dec 10, 2016 9:29 AM, "Frank Muller" > wrote: > > Hi Stanislaw > > First, I don't care if I've to use documented/undocumented calls as long > as I can achieve my goal: faster dir walking. > > And you're right, here is a detailed comparison with other scripting > languages: > > In my /usr/share, there?s: > 2580 directories > 28953 files > > 1. Erlang (no io:format/1, just recurse): > > walk(Dir) -> > {ok, Files} = file:list_dir(Dir), > walk(Dir, Files). > > walk(Dir, [ Basename | Rest ]) -> > Path = filename:join([ Dir, Basename ]), > case filelib:is_dir(Path) of > true -> > walk(Path); > false -> > %% io:format("~s~n", [Path]), > filelib:file_size(Path) > end, > walk(Dir, Rest); > walk(_, []) -> > ok. > > timer:tc(fun() -> directoy:walker("/usr/share") end). > {4662361,ok} > > 2. Python (this code even count the size of dir): > From: > http://stackoverflow.com/questions/1392413/calculating-a-directory-size-using-python > > import os > def get_size(start_path = '.'): > total_size = 0 > for dirpath, dirnames, filenames in os.walk(start_path): > for f in filenames: > fp = os.path.join(dirpath, f) > total_size += os.path.getsize(fp) > return total_size > > print get_size() > > $ cd /usr/share > $ time dir_walker.py > 432034130 > 0.25 real 0.13 user 0.10 sys > > 2. Perl (same, count dir size) > http://www.perlmonks.org/?node_id=168974 > > use File::Find; > my $size = 0; > find(sub { $size += -s if -f $_ }, "/usr/share"); > > $ time perl dir_walker.pl > 432034130 > 0.13 real 0.05 user 0.08 sys > > 3. Ruby (same, count dir size): > > def directory_size(path) > path << '/' unless path.end_with?('/') > raise RuntimeError, "#{path} is not a directory" unless > File.directory?(path) > total_size = 0 > Dir["#{path}**/*"].each do |f| > total_size += File.size(f) if File.file?(f) && File.size?(f) > end > total_size > end > puts directory_size '/usr/share? > > $ time walker.rb > 432028422 > 0.21 real 0.09 user 0.11 sys > > 4. Lua: > From: http://lua-users.org/wiki/DirTreeIterator > > require "lfs" > > function dirtree(dir) > assert(dir and dir ~= "", "directory parameter is missing or empty") > if string.sub(dir, -1) == "/" then > dir=string.sub(dir, 1, -2) > end > > local function yieldtree(dir) > for entry in lfs.dir(dir) do > if entry ~= "." and entry ~= ".." then > entry=dir.."/"..entry > local attr=lfs.attributes(entry) > coroutine.yield(entry,attr) > if attr.mode == "directory" then > yieldtree(entry) > end > end > end > end > > return coroutine.wrap(function() yieldtree(dir) end) > end > > for filename, attr in dirtree("/usr/share") do > print(attr.mode, filename) > end > > $ luarocks install luafilesystem > $ time lua walker.lua > /dev/null > 0.30 real 0.16 user 0.14 sys > > Do you need more? > > Thanks for you help. > /Frank > > Le sam. 10 d?c. 2016 ? 00:51, Stanislaw Klekot a > ?crit : > > On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller wrote: > > > I would like to improve the speed of my directory walker. > > > > > > walk(Dir) -> > > > {ok, Files} = prim_file:list_dir(Dir), > > > walk(Dir, Files). > > > > Why prim_file:list_dir() instead of file:list_dir()? The former is > > undocumented internal function. > > > > [...] > > > Compared to almost anything i found on the web, it?s still very slow: > > > > timer:tc(fun() -> dir:walk("/usr/share") end). > > > {4662361,ok} > > > > What is it this "anything you found on the web"? And how did you run > > your comparisons? There's a large difference between first and second > > consequent run caused by OS' directory cache, and there's large > > difference between simply walking through the directory and walking with > > printing something to the screen for every file. > > > > Then there's also your using filelib:is_dir() and then > > filelib:file_size(), which means two stat(2) calls, while you only need > > to do it once per file (file:read_file_info()). > > > > -- > > Stanislaw Klekot > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Sat Dec 10 09:43:14 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sat, 10 Dec 2016 08:43:14 +0000 Subject: [erlang-questions] Fast directory walker In-Reply-To: <584B6483.70206@gmail.com> References: <584B6483.70206@gmail.com> Message-ID: Hi Michael, Better, but still ~2 seconds: > timer:tc(fun() -> filelib:fold_files("/usr/share", ".*", true, fun(F, N) -> N + 1 end, 0) end). {1993074,28953} If I get it correctly the call matches only on files, not on dirs. /Frank Le sam. 10 d?c. 2016 ? 03:12, Michael Truog a ?crit : > > > On 12/09/2016 03:15 PM, Frank Muller > > wrote: > > > > > > > > > Hi > > > > > > > > I would like to improve the speed of my directory > > walker. > > > > > > > > walk(Dir) -> > > > {ok, Files} = prim_file:list_dir(Dir), > > > walk(Dir, Files). > > > > > > > > walk(Dir, [ Basename | Rest ]) -> > > > Path = filename:join([ Dir, Basename ]), > > > case filelib:is_dir(Path) of > > > true -> > > > walk(Path); > > > false -> > > > io:format("~s~n", [Path]), > > > filelib:file_size(Path) > > > end, > > > walk(Dir, Rest); > > > walk(_, []) -> > > > ok. > > > > > > > > > > > > > > > Compared > > to almost anything i found on the web, it?s still very slow: > > > > > > timer:tc(fun() -> dir:walk("/usr/share") > > end). > > > > > {4662361,ok} > > > > > > > > Have you tried filelib:fold_files/5 > > (http://erlang.org/doc/man/filelib.html#fold_files-5) ? > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Sat Dec 10 10:05:51 2016 From: sergej.jurecko@REDACTED (=?utf-8?Q?Sergej_Jure=C4=8Dko?=) Date: Sat, 10 Dec 2016 10:05:51 +0100 Subject: [erlang-questions] Fast directory walker In-Reply-To: References: <20161209235027.GA2814@jarowit.net> Message-ID: <97AE1B90-8D72-4C39-82F8-33FB127322C8@gmail.com> read_file_info does the job of is_dir and file_size in a single call. That was the intention. Also use file:read_file_info(name,[raw]) Sergej > On 10 Dec 2016, at 09:42, Benoit Chesneau wrote: > > this is kind of bullshit (sorry ;).... at the end this is what does the helpers in filelib: > https://github.com/erlang/otp/blob/maint/lib/stdlib/src/filelib.erl#L257 > > except if you have a better algorithm in mind i don't se the point of rewriting something that is aleaready existing ... > > On Sat, 10 Dec 2016 at 09:36, Sergej Jure?ko > wrote: > Stop using filelib functions. Use file:read_file_info and file:list_dir. > > Sergej > > On Dec 10, 2016 9:29 AM, "Frank Muller" > wrote: > Hi Stanislaw > > First, I don't care if I've to use documented/undocumented calls as long as I can achieve my goal: faster dir walking. > > And you're right, here is a detailed comparison with other scripting languages: > > In my /usr/share, there?s: > 2580 directories > 28953 files > > 1. Erlang (no io:format/1, just recurse): > > walk(Dir) -> > {ok, Files} = file:list_dir(Dir), > walk(Dir, Files). > > walk(Dir, [ Basename | Rest ]) -> > Path = filename:join([ Dir, Basename ]), > case filelib:is_dir(Path) of > true -> > walk(Path); > false -> > %% io:format("~s~n", [Path]), > filelib:file_size(Path) > end, > walk(Dir, Rest); > walk(_, []) -> > ok. > > timer:tc(fun() -> directoy:walker("/usr/share") end). > {4662361 ,ok} > > 2. Python (this code even count the size of dir): > From: http://stackoverflow.com/questions/1392413/calculating-a-directory-size-using-python > > import os > def get_size(start_path = '.'): > total_size = 0 > for dirpath, dirnames, filenames in os.walk(start_path): > for f in filenames: > fp = os.path.join(dirpath, f) > total_size += os.path.getsize(fp) > return total_size > > print get_size() > > $ cd /usr/share > $ time dir_walker.py > 432034130 > 0.25 real 0.13 user 0.10 sys > > 2. Perl (same, count dir size) > http://www.perlmonks.org/?node_id=168974 > > use File::Find; > my $size = 0; > find(sub { $size += -s if -f $_ }, "/usr/share"); > > $ time perl dir_walker.pl > 432034130 > 0.13 real 0.05 user 0.08 sys > > 3. Ruby (same, count dir size): > > def directory_size(path) > path << '/' unless path.end_with?('/') > raise RuntimeError, "#{path} is not a directory" unless File.directory?(path) > total_size = 0 > Dir["#{path}**/*"].each do |f| > total_size += File.size(f) if File.file?(f) && File.size?(f) > end > total_size > end > puts directory_size '/usr/share? > > $ time walker.rb > 432028422 > 0.21 real 0.09 user 0.11 sys > > 4. Lua: > From: http://lua-users.org/wiki/DirTreeIterator > > require "lfs" > > function dirtree(dir) > assert(dir and dir ~= "", "directory parameter is missing or empty") > if string.sub(dir, -1) == "/" then > dir=string.sub(dir, 1, -2) > end > > local function yieldtree(dir) > for entry in lfs.dir(dir) do > if entry ~= "." and entry ~= ".." then > entry=dir.."/"..entry > local attr=lfs.attributes(entry) > coroutine.yield(entry,attr) > if attr.mode == "directory" then > yieldtree(entry) > end > end > end > end > > return coroutine.wrap(function() yieldtree(dir) end) > end > > for filename, attr in dirtree("/usr/share") do > print(attr.mode, filename) > end > > $ luarocks install luafilesystem > $ time lua walker.lua > /dev/null > 0.30 real 0.16 user 0.14 sys > > Do you need more? > > Thanks for you help. > /Frank > > Le sam. 10 d?c. 2016 ? 00:51, Stanislaw Klekot > a ?crit : > On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller wrote: > > > I would like to improve the speed of my directory walker. > > > > > > walk(Dir) -> > > > {ok, Files} = prim_file:list_dir(Dir), > > > walk(Dir, Files). > > > > Why prim_file:list_dir() instead of file:list_dir()? The former is > > undocumented internal function. > > > > [...] > > > Compared to almost anything i found on the web, it?s still very slow: > > > > timer:tc(fun() -> dir:walk("/usr/share") end). > > > {4662361,ok} > > > > What is it this "anything you found on the web"? And how did you run > > your comparisons? There's a large difference between first and second > > consequent run caused by OS' directory cache, and there's large > > difference between simply walking through the directory and walking with > > printing something to the screen for every file. > > > > Then there's also your using filelib:is_dir() and then > > filelib:file_size(), which means two stat(2) calls, while you only need > > to do it once per file (file:read_file_info()). > > > > -- > > Stanislaw Klekot > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Sat Dec 10 10:20:49 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sat, 10 Dec 2016 09:20:49 +0000 Subject: [erlang-questions] Fast directory walker In-Reply-To: <97AE1B90-8D72-4C39-82F8-33FB127322C8@gmail.com> References: <20161209235027.GA2814@jarowit.net> <97AE1B90-8D72-4C39-82F8-33FB127322C8@gmail.com> Message-ID: Combining previous hints (Benoit, Sergej): -module(directory). -include_lib("kernel/include/file.hrl"). -export([walker/1]). walker(Path) -> case file:read_file_info(Path, [raw]) of {ok, #file_info{type = regular}} -> 1; _ -> %% not care about symlink for now, assume a directory Children = filelib:wildcard(Path ++ "/*"), lists:foldl(fun(P, N) -> N + walker(P) end, 0, Children) end. > timer:tc(fun() -> directory:walker("/usr/share") end). {1611688, <1611688,28953>*28953 <1611688,28953>*} I'm only counting number of files in this case. /Frank Le sam. 10 d?c. 2016 ? 10:05, Sergej Jure?ko a ?crit : > read_file_info does the job of is_dir and file_size in a single call. That > was the intention. > > Also use file:read_file_info(name,[raw]) > > > Sergej > > On 10 Dec 2016, at 09:42, Benoit Chesneau wrote: > > this is kind of bullshit (sorry ;).... at the end this is what does the > helpers in filelib: > https://github.com/erlang/otp/blob/maint/lib/stdlib/src/filelib.erl#L257 > > except if you have a better algorithm in mind i don't se the point of > rewriting something that is aleaready existing ... > > On Sat, 10 Dec 2016 at 09:36, Sergej Jure?ko > wrote: > > Stop using filelib functions. Use file:read_file_info and file:list_dir. > > Sergej > > On Dec 10, 2016 9:29 AM, "Frank Muller" > wrote: > > Hi Stanislaw > > First, I don't care if I've to use documented/undocumented calls as long > as I can achieve my goal: faster dir walking. > > And you're right, here is a detailed comparison with other scripting > languages: > > In my /usr/share, there?s: > 2580 directories > 28953 files > > 1. Erlang (no io:format/1, just recurse): > > walk(Dir) -> > {ok, Files} = file:list_dir(Dir), > walk(Dir, Files). > > walk(Dir, [ Basename | Rest ]) -> > Path = filename:join([ Dir, Basename ]), > case filelib:is_dir(Path) of > true -> > walk(Path); > false -> > %% io:format("~s~n", [Path]), > filelib:file_size(Path) > end, > walk(Dir, Rest); > walk(_, []) -> > ok. > > timer:tc(fun() -> directoy:walker("/usr/share") end). > {4662361,ok} > > 2. Python (this code even count the size of dir): > From: > http://stackoverflow.com/questions/1392413/calculating-a-directory-size-using-python > > import os > def get_size(start_path = '.'): > total_size = 0 > for dirpath, dirnames, filenames in os.walk(start_path): > for f in filenames: > fp = os.path.join(dirpath, f) > total_size += os.path.getsize(fp) > return total_size > > print get_size() > > $ cd /usr/share > $ time dir_walker.py > 432034130 > 0.25 real 0.13 user 0.10 sys > > 2. Perl (same, count dir size) > http://www.perlmonks.org/?node_id=168974 > > use File::Find; > my $size = 0; > find(sub { $size += -s if -f $_ }, "/usr/share"); > > $ time perl dir_walker.pl > 432034130 > 0.13 real 0.05 user 0.08 sys > > 3. Ruby (same, count dir size): > > def directory_size(path) > path << '/' unless path.end_with?('/') > raise RuntimeError, "#{path} is not a directory" unless > File.directory?(path) > total_size = 0 > Dir["#{path}**/*"].each do |f| > total_size += File.size(f) if File.file?(f) && File.size?(f) > end > total_size > end > puts directory_size '/usr/share? > > $ time walker.rb > 432028422 > 0.21 real 0.09 user 0.11 sys > > 4. Lua: > From: http://lua-users.org/wiki/DirTreeIterator > > require "lfs" > > function dirtree(dir) > assert(dir and dir ~= "", "directory parameter is missing or empty") > if string.sub(dir, -1) == "/" then > dir=string.sub(dir, 1, -2) > end > > local function yieldtree(dir) > for entry in lfs.dir(dir) do > if entry ~= "." and entry ~= ".." then > entry=dir.."/"..entry > local attr=lfs.attributes(entry) > coroutine.yield(entry,attr) > if attr.mode == "directory" then > yieldtree(entry) > end > end > end > end > > return coroutine.wrap(function() yieldtree(dir) end) > end > > for filename, attr in dirtree("/usr/share") do > print(attr.mode, filename) > end > > $ luarocks install luafilesystem > $ time lua walker.lua > /dev/null > 0.30 real 0.16 user 0.14 sys > > Do you need more? > > Thanks for you help. > /Frank > > Le sam. 10 d?c. 2016 ? 00:51, Stanislaw Klekot a > ?crit : > > On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller wrote: > > > I would like to improve the speed of my directory walker. > > > > > > walk(Dir) -> > > > {ok, Files} = prim_file:list_dir(Dir), > > > walk(Dir, Files). > > > > Why prim_file:list_dir() instead of file:list_dir()? The former is > > undocumented internal function. > > > > [...] > > > Compared to almost anything i found on the web, it?s still very slow: > > > > timer:tc(fun() -> dir:walk("/usr/share") end). > > > {4662361,ok} > > > > What is it this "anything you found on the web"? And how did you run > > your comparisons? There's a large difference between first and second > > consequent run caused by OS' directory cache, and there's large > > difference between simply walking through the directory and walking with > > printing something to the screen for every file. > > > > Then there's also your using filelib:is_dir() and then > > filelib:file_size(), which means two stat(2) calls, while you only need > > to do it once per file (file:read_file_info()). > > > > -- > > Stanislaw Klekot > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Sat Dec 10 10:21:28 2016 From: mjtruog@REDACTED (Michael Truog) Date: Sat, 10 Dec 2016 01:21:28 -0800 Subject: [erlang-questions] Fast directory walker In-Reply-To: References: <584B6483.70206@gmail.com> Message-ID: <584BC918.7070500@gmail.com> On 12/10/2016 12:43 AM, Frank Muller wrote: > Hi Michael, > > Better, but still ~2 seconds: > > > timer:tc(fun() -> filelib:fold_files("/usr/share", ".*", true, fun(F, N) -> N + 1 end, 0) end). > {1993074,28953 } > > If I get it correctly the call matches only on files, not on dirs. Yes, the regex only matches on filenames, not directory names. Currently, the filelib:fold_files/5 function requires a regex so you are likely paying a penalty for it matching on a wildcard regex on each filename. So, it would probably be nice to have a filelib:fold_files/4 that doesn't require a regex. Best Regards, Michael > > /Frank > > Le sam. 10 d?c. 2016 ? 03:12, Michael Truog > a ?crit : > > > > On 12/09/2016 03:15 PM, Frank Muller > > wrote: > > > > >> >> >> >> >> Hi >> >> >> >> >> >> >> >> I would like to improve the speed of my directory >> >> walker. >> >> >> >> >> >> >> >> walk(Dir) -> >> >> >> {ok, Files} = prim_file:list_dir(Dir), >> >> >> walk(Dir, Files). >> >> >> >> >> >> >> >> walk(Dir, [ Basename | Rest ]) -> >> >> >> Path = filename:join([ Dir, Basename ]), >> >> >> case filelib:is_dir(Path) of >> >> >> true -> >> >> >> walk(Path); >> >> >> false -> >> >> >> io:format("~s~n", [Path]), >> >> >> filelib:file_size(Path) >> >> >> end, >> >> >> walk(Dir, Rest); >> >> >> walk(_, []) -> >> >> >> ok. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Compared >> >> to almost anything i found on the web, it?s still very slow: >> >> >> >> >> > timer:tc(fun() -> dir:walk("/usr/share") >> >> end). >> >> >> >> >> {4662361 ,ok} >> >> > > > > > > Have you tried filelib:fold_files/5 > > (http://erlang.org/doc/man/filelib.html#fold_files-5) ? > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fly@REDACTED Sat Dec 10 11:33:33 2016 From: fly@REDACTED (Fred Youhanaie) Date: Sat, 10 Dec 2016 10:33:33 +0000 Subject: [erlang-questions] Fast directory walker In-Reply-To: References: <20161209235027.GA2814@jarowit.net> <97AE1B90-8D72-4C39-82F8-33FB127322C8@gmail.com> Message-ID: <3f1637b8-8f53-e2b7-dd0b-7255896cd32c@anydata.co.uk> Out of interest, what do you get when you run the same benchmark twice in succession? The buffer cache would play a role on the second and subsequent attempts. For your code below, running on my oldish laptop, I get {35667060,158949} and {8606920,158949}. You can use "blockdev --flushbufs " to ensure the buffers are clean before running the benchmarks, especially when comparing different versions and languages. Having said that, the erlang version does look slow, especially when compared with the shell equivalent "time find /usr/share -print | wc -l" $ sudo blockdev --flushbufs /dev/... $ time find /usr/share -print | wc -l 186911 real 0m32.446s user 0m0.796s sys 0m1.808s $ time find /usr/share -print | wc -l 186911 real 0m0.336s user 0m0.152s sys 0m0.200s Perhaps there is room for improvement within the library itself! Cheers, f. On 10/12/16 09:20, Frank Muller wrote: > Combining previous hints (Benoit, Sergej): > > -module(directory). > -include_lib("kernel/include/file.hrl"). > -export([walker/1]). > > walker(Path) -> > case file:read_file_info(Path, [raw]) of > {ok, #file_info{type = regular}} -> > 1; > _ -> %% not care about symlink for now, assume a directory > Children = filelib:wildcard(Path ++ "/*"), > lists:foldl(fun(P, N) -> N + walker(P) end, 0, Children) > end. > >> timer:tc(fun() -> directory:walker("/usr/share") end). > {1611688, /28953 /} > > I'm only counting number of files in this case. > > /Frank > > Le sam. 10 d?c. 2016 ? 10:05, Sergej Jure?ko > a ?crit : > > read_file_info does the job of is_dir and file_size in a single call. That was the intention. > > Also use file:read_file_info(name,[raw]) > > > Sergej > >> On 10 Dec 2016, at 09:42, Benoit Chesneau > wrote: >> >> this is kind of bullshit (sorry ;).... at the end this is what does the helpers in filelib: >> https://github.com/erlang/otp/blob/maint/lib/stdlib/src/filelib.erl#L257 >> >> except if you have a better algorithm in mind i don't se the point of rewriting something that is aleaready existing ... >> >> On Sat, 10 Dec 2016 at 09:36, Sergej Jure?ko > wrote: >> >> Stop using filelib functions. Use file:read_file_info and file:list_dir. >> >> Sergej >> >> On Dec 10, 2016 9:29 AM, "Frank Muller" > wrote: >> >> Hi Stanislaw >> >> First, I don't care if I've to use documented/undocumented calls as long as I can achieve my goal: faster dir walking. >> >> And you're right, here is a detailed comparison with other scripting languages: >> >> In my /usr/share, there?s: >> 2580 directories >> 28953 files >> >> 1. Erlang (no io:format/1, just recurse): >> >> walk(Dir) -> >> {ok, Files} = file:list_dir(Dir), >> walk(Dir, Files). >> >> walk(Dir, [ Basename | Rest ]) -> >> Path = filename:join([ Dir, Basename ]), >> case filelib:is_dir(Path) of >> true -> >> walk(Path); >> false -> >> %% io:format("~s~n", [Path]), >> filelib:file_size(Path) >> end, >> walk(Dir, Rest); >> walk(_, []) -> >> ok. >> >> timer:tc(fun() -> directoy:walker("/usr/share") end). >> {4662361 ,ok} >> >> 2. Python (this code even count the size of dir): >> From: http://stackoverflow.com/questions/1392413/calculating-a-directory-size-using-python >> >> import os >> def get_size(start_path = '.'): >> total_size = 0 >> for dirpath, dirnames, filenames in os.walk(start_path): >> for f in filenames: >> fp = os.path.join(dirpath, f) >> total_size += os.path.getsize(fp) >> return total_size >> >> print get_size() >> >> $ cd /usr/share >> $ time dir_walker.py >> 432034130 >> 0.25 real 0.13 user 0.10 sys >> >> 2. Perl (same, count dir size) >> http://www.perlmonks.org/?node_id=168974 >> >> use File::Find; >> my $size = 0; >> find(sub { $size += -s if -f $_ }, "/usr/share"); >> >> $ time perl dir_walker.pl >> 432034130 >> 0.13 real 0.05 user 0.08 sys >> >> 3. Ruby (same, count dir size): >> >> def directory_size(path) >> path << '/' unless path.end_with?('/') >> raise RuntimeError, "#{path} is not a directory" unless File.directory?(path) >> total_size = 0 >> Dir["#{path}**/*"].each do |f| >> total_size += File.size(f) if File.file?(f) && File.size?(f) >> end >> total_size >> end >> puts directory_size '/usr/share? >> >> $ time walker.rb >> 432028422 >> 0.21 real 0.09 user 0.11 sys >> >> 4. Lua: >> From: http://lua-users.org/wiki/DirTreeIterator >> >> require "lfs" >> >> function dirtree(dir) >> assert(dir and dir ~= "", "directory parameter is missing or empty") >> if string.sub(dir, -1) == "/" then >> dir=string.sub(dir, 1, -2) >> end >> >> local function yieldtree(dir) >> for entry in lfs.dir(dir) do >> if entry ~= "." and entry ~= ".." then >> entry=dir.."/"..entry >> local attr=lfs.attributes(entry) >> coroutine.yield(entry,attr) >> if attr.mode == "directory" then >> yieldtree(entry) >> end >> end >> end >> end >> >> return coroutine.wrap(function() yieldtree(dir) end) >> end >> >> for filename, attr in dirtree("/usr/share") do >> print(attr.mode, filename) >> end >> >> $ luarocks install luafilesystem >> $ time lua walker.lua > /dev/null >> 0.30 real 0.16 user 0.14 sys >> >> Do you need more? >> >> Thanks for you help. >> /Frank >> >> Le sam. 10 d?c. 2016 ? 00:51, Stanislaw Klekot > a ?crit : >> >> On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller wrote: >> >> > I would like to improve the speed of my directory walker. >> >> > >> >> > walk(Dir) -> >> >> > {ok, Files} = prim_file:list_dir(Dir), >> >> > walk(Dir, Files). >> >> >> >> Why prim_file:list_dir() instead of file:list_dir()? The former is >> >> undocumented internal function. >> >> >> >> [...] >> >> > Compared to almost anything i found on the web, it?s still very slow: >> >> > > timer:tc(fun() -> dir:walk("/usr/share") end). >> >> > {4662361,ok} >> >> >> >> What is it this "anything you found on the web"? And how did you run >> >> your comparisons? There's a large difference between first and second >> >> consequent run caused by OS' directory cache, and there's large >> >> difference between simply walking through the directory and walking with >> >> printing something to the screen for every file. >> >> >> >> Then there's also your using filelib:is_dir() and then >> >> filelib:file_size(), which means two stat(2) calls, while you only need >> >> to do it once per file (file:read_file_info()). >> >> >> >> -- >> >> Stanislaw Klekot >> >> From sergej.jurecko@REDACTED Sat Dec 10 11:57:28 2016 From: sergej.jurecko@REDACTED (=?utf-8?Q?Sergej_Jure=C4=8Dko?=) Date: Sat, 10 Dec 2016 11:57:28 +0100 Subject: [erlang-questions] Fast directory walker In-Reply-To: References: <20161209235027.GA2814@jarowit.net> <97AE1B90-8D72-4C39-82F8-33FB127322C8@gmail.com> Message-ID: <209180C1-5897-4DF8-B759-215FD25C7128@gmail.com> A faster version that returns size of folder. Pth++"/"++H is faster than filename:join, and directly calling prim_file is also faster. walker1(Path) -> case prim_file:list_dir(Path) of {ok,L} -> walker1(Path,L,0); _ -> 0 end. walker1(Pth,["."|T],Sz) -> walker1(Pth,T,Sz); walker1(Pth,[".."|T],Sz) -> walker1(Pth,T,Sz); walker1(Pth,[H|T],Sz) -> Nm = Pth++"/"++H, case prim_file:read_file_info(Nm) of {ok,#file_info{type = regular, size = FS}} -> walker1(Pth,T,Sz+FS); {ok,#file_info{type = directory}} -> case prim_file:list_dir(Nm) of {ok,L} -> walker1(Pth, T, walker1(Nm,L,Sz)); _ -> walker1(Pth, T, Sz) end; _ -> walker1(Pth,T,Sz) end; walker1(_,[],Sz) -> Sz. > On 10 Dec 2016, at 10:20, Frank Muller wrote: > > Combining previous hints (Benoit, Sergej): > > -module(directory). > -include_lib("kernel/include/file.hrl"). > -export([walker/1]). > > walker(Path) -> > case file:read_file_info(Path, [raw]) of > {ok, #file_info{type = regular}} -> > 1; > _ -> %% not care about symlink for now, assume a directory > Children = filelib:wildcard(Path ++ "/*"), > lists:foldl(fun(P, N) -> N + walker(P) end, 0, Children) > end. > > > timer:tc(fun() -> directory:walker("/usr/share") end). > {1611688, 28953 } > > I'm only counting number of files in this case. > > /Frank > > Le sam. 10 d?c. 2016 ? 10:05, Sergej Jure?ko > a ?crit : > read_file_info does the job of is_dir and file_size in a single call. That was the intention. > > Also use file:read_file_info(name,[raw]) > > > Sergej > >> On 10 Dec 2016, at 09:42, Benoit Chesneau > wrote: >> >> this is kind of bullshit (sorry ;).... at the end this is what does the helpers in filelib: >> https://github.com/erlang/otp/blob/maint/lib/stdlib/src/filelib.erl#L257 >> >> except if you have a better algorithm in mind i don't se the point of rewriting something that is aleaready existing ... >> >> On Sat, 10 Dec 2016 at 09:36, Sergej Jure?ko > wrote: >> Stop using filelib functions. Use file:read_file_info and file:list_dir. >> >> Sergej >> >> On Dec 10, 2016 9:29 AM, "Frank Muller" > wrote: >> Hi Stanislaw >> >> First, I don't care if I've to use documented/undocumented calls as long as I can achieve my goal: faster dir walking. >> >> And you're right, here is a detailed comparison with other scripting languages: >> >> In my /usr/share, there?s: >> 2580 directories >> 28953 files >> >> 1. Erlang (no io:format/1, just recurse): >> >> walk(Dir) -> >> {ok, Files} = file:list_dir(Dir), >> walk(Dir, Files). >> >> walk(Dir, [ Basename | Rest ]) -> >> Path = filename:join([ Dir, Basename ]), >> case filelib:is_dir(Path) of >> true -> >> walk(Path); >> false -> >> %% io:format("~s~n", [Path]), >> filelib:file_size(Path) >> end, >> walk(Dir, Rest); >> walk(_, []) -> >> ok. >> >> timer:tc(fun() -> directoy:walker("/usr/share") end). >> {4662361 ,ok} >> >> 2. Python (this code even count the size of dir): >> From: http://stackoverflow.com/questions/1392413/calculating-a-directory-size-using-python >> >> import os >> def get_size(start_path = '.'): >> total_size = 0 >> for dirpath, dirnames, filenames in os.walk(start_path): >> for f in filenames: >> fp = os.path.join(dirpath, f) >> total_size += os.path.getsize(fp) >> return total_size >> >> print get_size() >> >> $ cd /usr/share >> $ time dir_walker.py >> 432034130 >> 0.25 real 0.13 user 0.10 sys >> >> 2. Perl (same, count dir size) >> http://www.perlmonks.org/?node_id=168974 >> >> use File::Find; >> my $size = 0; >> find(sub { $size += -s if -f $_ }, "/usr/share"); >> >> $ time perl dir_walker.pl >> 432034130 >> 0.13 real 0.05 user 0.08 sys >> >> 3. Ruby (same, count dir size): >> >> def directory_size(path) >> path << '/' unless path.end_with?('/') >> raise RuntimeError, "#{path} is not a directory" unless File.directory?(path) >> total_size = 0 >> Dir["#{path}**/*"].each do |f| >> total_size += File.size(f) if File.file?(f) && File.size?(f) >> end >> total_size >> end >> puts directory_size '/usr/share? >> >> $ time walker.rb >> 432028422 >> 0.21 real 0.09 user 0.11 sys >> >> 4. Lua: >> From: http://lua-users.org/wiki/DirTreeIterator >> >> require "lfs" >> >> function dirtree(dir) >> assert(dir and dir ~= "", "directory parameter is missing or empty") >> if string.sub(dir, -1) == "/" then >> dir=string.sub(dir, 1, -2) >> end >> >> local function yieldtree(dir) >> for entry in lfs.dir(dir) do >> if entry ~= "." and entry ~= ".." then >> entry=dir.."/"..entry >> local attr=lfs.attributes(entry) >> coroutine.yield(entry,attr) >> if attr.mode == "directory" then >> yieldtree(entry) >> end >> end >> end >> end >> >> return coroutine.wrap(function() yieldtree(dir) end) >> end >> >> for filename, attr in dirtree("/usr/share") do >> print(attr.mode, filename) >> end >> >> $ luarocks install luafilesystem >> $ time lua walker.lua > /dev/null >> 0.30 real 0.16 user 0.14 sys >> >> Do you need more? >> >> Thanks for you help. >> /Frank >> >> Le sam. 10 d?c. 2016 ? 00:51, Stanislaw Klekot > a ?crit : >> On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller wrote: >> >> > I would like to improve the speed of my directory walker. >> >> > >> >> > walk(Dir) -> >> >> > {ok, Files} = prim_file:list_dir(Dir), >> >> > walk(Dir, Files). >> >> >> >> Why prim_file:list_dir() instead of file:list_dir()? The former is >> >> undocumented internal function. >> >> >> >> [...] >> >> > Compared to almost anything i found on the web, it?s still very slow: >> >> > > timer:tc(fun() -> dir:walk("/usr/share") end). >> >> > {4662361,ok} >> >> >> >> What is it this "anything you found on the web"? And how did you run >> >> your comparisons? There's a large difference between first and second >> >> consequent run caused by OS' directory cache, and there's large >> >> difference between simply walking through the directory and walking with >> >> printing something to the screen for every file. >> >> >> >> Then there's also your using filelib:is_dir() and then >> >> filelib:file_size(), which means two stat(2) calls, while you only need >> >> to do it once per file (file:read_file_info()). >> >> >> >> -- >> >> Stanislaw Klekot >> >> >> >> >> _______________________________________________ >> >> >> erlang-questions mailing list >> >> >> erlang-questions@REDACTED >> >> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> >> >> _______________________________________________ >> >> >> erlang-questions mailing list >> >> >> erlang-questions@REDACTED >> >> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Sat Dec 10 12:02:26 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 10 Dec 2016 14:02:26 +0300 Subject: [erlang-questions] Fast directory walker In-Reply-To: <209180C1-5897-4DF8-B759-215FD25C7128@gmail.com> References: <20161209235027.GA2814@jarowit.net> <97AE1B90-8D72-4C39-82F8-33FB127322C8@gmail.com> <209180C1-5897-4DF8-B759-215FD25C7128@gmail.com> Message-ID: All these discussions about faster or slower are not considering very important thing: all these operations are going through singleton process file_server. All this speed will become nothing on single core. This hackish way: {ok, H} = prim_file:start(), {ok, Entries} = prim_file:list_dir(H, Path) works on multicore -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Sat Dec 10 12:25:45 2016 From: erlang@REDACTED (Joe Armstrong) Date: Sat, 10 Dec 2016 12:25:45 +0100 Subject: [erlang-questions] Fast directory walker In-Reply-To: <3f1637b8-8f53-e2b7-dd0b-7255896cd32c@anydata.co.uk> References: <20161209235027.GA2814@jarowit.net> <97AE1B90-8D72-4C39-82F8-33FB127322C8@gmail.com> <3f1637b8-8f53-e2b7-dd0b-7255896cd32c@anydata.co.uk> Message-ID: This is very interesting - I've often wondered why directory traversal is faster in C than Erlang since all Erlang is doing is calling C primitives - I think measuring the times for this is difficult - if for example you run a python test immediately followed by an erlang test, I'd expect you get a different result than if you run the erlang test with a "cold" cache - running the python program will have the side effect of loading various memory buffers - I'd also expect different results on different OS and different results depending upon your physical memory sizes and the size of the trees you are traversing. I've often wondered if fast C programs achieve their speed by directly poking around in the underlying inode structures, but this would involve detailed knowledge of the underlying file system (ie is an ext3, HFS+, FAT32 etc.) Programs like rsync and the Dropbox sync algorithm seem unreasonably fast to me - I wonder if they poke around in the underlying OS representation of the file system and not use a more portable and easier to use interface. /Joe On Sat, Dec 10, 2016 at 11:33 AM, Fred Youhanaie wrote: > > Out of interest, what do you get when you run the same benchmark twice in > succession? > > The buffer cache would play a role on the second and subsequent attempts. > For your code below, running on my oldish laptop, I get {35667060,158949} > and {8606920,158949}. > > You can use "blockdev --flushbufs " to ensure the buffers are clean > before running the benchmarks, especially when comparing different versions > and languages. > > Having said that, the erlang version does look slow, especially when > compared with the shell equivalent "time find /usr/share -print | wc -l" > > $ sudo blockdev --flushbufs /dev/... > > $ time find /usr/share -print | wc -l > 186911 > > real 0m32.446s > user 0m0.796s > sys 0m1.808s > > $ time find /usr/share -print | wc -l > 186911 > > real 0m0.336s > user 0m0.152s > sys 0m0.200s > > > Perhaps there is room for improvement within the library itself! > > Cheers, > f. > > > On 10/12/16 09:20, Frank Muller wrote: >> >> Combining previous hints (Benoit, Sergej): >> >> -module(directory). >> -include_lib("kernel/include/file.hrl"). >> -export([walker/1]). >> >> walker(Path) -> >> case file:read_file_info(Path, [raw]) of >> {ok, #file_info{type = regular}} -> >> 1; >> _ -> %% not care about symlink for now, assume a directory >> Children = filelib:wildcard(Path ++ "/*"), >> lists:foldl(fun(P, N) -> N + walker(P) end, 0, Children) >> end. >> >>> timer:tc(fun() -> directory:walker("/usr/share") end). >> >> {1611688, /28953 /} >> >> I'm only counting number of files in this case. >> >> /Frank >> >> Le sam. 10 d?c. 2016 ? 10:05, Sergej Jure?ko > > a ?crit : >> >> read_file_info does the job of is_dir and file_size in a single call. >> That was the intention. >> >> Also use file:read_file_info(name,[raw]) >> >> >> Sergej >> >>> On 10 Dec 2016, at 09:42, Benoit Chesneau >> > wrote: >>> >>> this is kind of bullshit (sorry ;).... at the end this is what does >>> the helpers in filelib: >>> >>> https://github.com/erlang/otp/blob/maint/lib/stdlib/src/filelib.erl#L257 >>> >>> except if you have a better algorithm in mind i don't se the point of >>> rewriting something that is aleaready existing ... >>> >>> On Sat, 10 Dec 2016 at 09:36, Sergej Jure?ko >>> > wrote: >>> >>> Stop using filelib functions. Use file:read_file_info and >>> file:list_dir. >>> >>> Sergej >>> >>> On Dec 10, 2016 9:29 AM, "Frank Muller" >>> > wrote: >>> >>> Hi Stanislaw >>> >>> First, I don't care if I've to use documented/undocumented >>> calls as long as I can achieve my goal: faster dir walking. >>> >>> And you're right, here is a detailed comparison with other >>> scripting languages: >>> >>> In my /usr/share, there?s: >>> 2580 directories >>> 28953 files >>> >>> 1. Erlang (no io:format/1, just recurse): >>> >>> walk(Dir) -> >>> {ok, Files} = file:list_dir(Dir), >>> walk(Dir, Files). >>> >>> walk(Dir, [ Basename | Rest ]) -> >>> Path = filename:join([ Dir, Basename ]), >>> case filelib:is_dir(Path) of >>> true -> >>> walk(Path); >>> false -> >>> %% io:format("~s~n", [Path]), >>> filelib:file_size(Path) >>> end, >>> walk(Dir, Rest); >>> walk(_, []) -> >>> ok. >>> >>> timer:tc(fun() -> directoy:walker("/usr/share") end). >>> {4662361 ,ok} >>> >>> 2. Python (this code even count the size of dir): >>> From: >>> http://stackoverflow.com/questions/1392413/calculating-a-directory-size-using-python >>> >>> import os >>> def get_size(start_path = '.'): >>> total_size = 0 >>> for dirpath, dirnames, filenames in os.walk(start_path): >>> for f in filenames: >>> fp = os.path.join(dirpath, f) >>> total_size += os.path.getsize(fp) >>> return total_size >>> >>> print get_size() >>> >>> $ cd /usr/share >>> $ time dir_walker.py >>> 432034130 >>> 0.25 real 0.13 user 0.10 sys >>> >>> 2. Perl (same, count dir size) >>> http://www.perlmonks.org/?node_id=168974 >>> >>> use File::Find; >>> my $size = 0; >>> find(sub { $size += -s if -f $_ }, "/usr/share"); >>> >>> $ time perl dir_walker.pl >>> 432034130 >>> 0.13 real 0.05 user 0.08 sys >>> >>> 3. Ruby (same, count dir size): >>> >>> def directory_size(path) >>> path << '/' unless path.end_with?('/') >>> raise RuntimeError, "#{path} is not a directory" unless >>> File.directory?(path) >>> total_size = 0 >>> Dir["#{path}**/*"].each do |f| >>> total_size += File.size(f) if File.file?(f) && >>> File.size?(f) >>> end >>> total_size >>> end >>> puts directory_size '/usr/share? >>> >>> $ time walker.rb >>> 432028422 >>> >>> 0.21 real 0.09 user 0.11 sys >>> >>> 4. Lua: >>> From: http://lua-users.org/wiki/DirTreeIterator >>> >>> require "lfs" >>> >>> function dirtree(dir) >>> assert(dir and dir ~= "", "directory parameter is missing >>> or empty") >>> if string.sub(dir, -1) == "/" then >>> dir=string.sub(dir, 1, -2) >>> end >>> >>> local function yieldtree(dir) >>> for entry in lfs.dir(dir) do >>> if entry ~= "." and entry ~= ".." then >>> entry=dir.."/"..entry >>> local attr=lfs.attributes(entry) >>> coroutine.yield(entry,attr) >>> if attr.mode == "directory" then >>> yieldtree(entry) >>> end >>> end >>> end >>> end >>> >>> return coroutine.wrap(function() yieldtree(dir) end) >>> end >>> >>> for filename, attr in dirtree("/usr/share") do >>> print(attr.mode, filename) >>> end >>> >>> $ luarocks install luafilesystem >>> $ time lua walker.lua > /dev/null >>> 0.30 real 0.16 user 0.14 sys >>> >>> Do you need more? >>> >>> Thanks for you help. >>> /Frank >>> >>> Le sam. 10 d?c. 2016 ? 00:51, Stanislaw Klekot >>> > a ?crit : >>> >>> On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller >>> wrote: >>> >>> > I would like to improve the speed of my directory >>> walker. >>> >>> > >>> >>> > walk(Dir) -> >>> >>> > {ok, Files} = prim_file:list_dir(Dir), >>> >>> > walk(Dir, Files). >>> >>> >>> >>> Why prim_file:list_dir() instead of file:list_dir()? The >>> former is >>> >>> undocumented internal function. >>> >>> >>> >>> [...] >>> >>> > Compared to almost anything i found on the web, it?s >>> still very slow: >>> >>> > > timer:tc(fun() -> dir:walk("/usr/share") end). >>> >>> > {4662361,ok} >>> >>> >>> >>> What is it this "anything you found on the web"? And how >>> did you run >>> >>> your comparisons? There's a large difference between >>> first and second >>> >>> consequent run caused by OS' directory cache, and there's >>> large >>> >>> difference between simply walking through the directory >>> and walking with >>> >>> printing something to the screen for every file. >>> >>> >>> >>> Then there's also your using filelib:is_dir() and then >>> >>> filelib:file_size(), which means two stat(2) calls, while >>> you only need >>> >>> to do it once per file (file:read_file_info()). >>> >>> >>> >>> -- >>> >>> Stanislaw Klekot >>> >>> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From frank.muller.erl@REDACTED Sat Dec 10 12:32:51 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sat, 10 Dec 2016 11:32:51 +0000 Subject: [erlang-questions] Fast directory walker In-Reply-To: References: <20161209235027.GA2814@jarowit.net> <97AE1B90-8D72-4C39-82F8-33FB127322C8@gmail.com> <209180C1-5897-4DF8-B759-215FD25C7128@gmail.com> Message-ID: Max, Very interesting. What's the benefit of starting this prim_file server? Does it make avoid the singleton process "file_server"? I'll bench this hack with Sergeij's version and post the result here. /Frank Le sam. 10 d?c. 2016 ? 12:02, Max Lapshin a ?crit : > All these discussions about faster or slower are not considering very > important thing: all these operations are going through singleton process > file_server. All this speed will become nothing on single core. > > > This hackish way: > > {ok, H} = prim_file:start(), > {ok, Entries} = prim_file:list_dir(H, Path) > > > works on multicore > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fly@REDACTED Sat Dec 10 14:13:16 2016 From: fly@REDACTED (Fred Youhanaie) Date: Sat, 10 Dec 2016 13:13:16 +0000 Subject: [erlang-questions] Fast directory walker In-Reply-To: References: <20161209235027.GA2814@jarowit.net> <97AE1B90-8D72-4C39-82F8-33FB127322C8@gmail.com> <3f1637b8-8f53-e2b7-dd0b-7255896cd32c@anydata.co.uk> Message-ID: <01cbcb75-68dc-1d4d-eb69-49ae338e19f4@anydata.co.uk> I just had a look at the GNU find source, it uses fts, man fts(3). I didn't bother digging deeper into glibc to see how fts is handling the inodes. Frank's earlier non-Erlang examples seem to be using dedicated file hierarchy walker modules, perhaps those are using the fts, or similar, C functions. In which case they are being compared with the pure erlang implementation of the tree walker. Staying with pure erlang version, a concurrent/multi-process tree walker could be an interesting project in itself :) Cheers, f. On 10/12/16 11:25, Joe Armstrong wrote: > This is very interesting - I've often wondered why directory traversal > is faster in C than > Erlang since all Erlang is doing is calling C primitives - > > I think measuring the times for this is difficult - if for example you > run a python test immediately followed by an erlang test, I'd expect > you get a different result than if you run the erlang > test with a "cold" cache - running the python program will have the > side effect of loading > various memory buffers - I'd also expect different results on > different OS and different > results depending upon your physical memory sizes and the size of the > trees you are traversing. > > I've often wondered if fast C programs achieve their speed by directly > poking around in the > underlying inode structures, but this would involve detailed knowledge > of the underlying > file system (ie is an ext3, HFS+, FAT32 etc.) > > Programs like rsync and the Dropbox sync algorithm seem unreasonably > fast to me - > I wonder if they poke around in the underlying OS representation of > the file system and not > use a more portable and easier to use interface. > > /Joe > > > > > On Sat, Dec 10, 2016 at 11:33 AM, Fred Youhanaie wrote: >> >> Out of interest, what do you get when you run the same benchmark twice in >> succession? > > > >> >> The buffer cache would play a role on the second and subsequent attempts. >> For your code below, running on my oldish laptop, I get {35667060,158949} >> and {8606920,158949}. >> >> You can use "blockdev --flushbufs " to ensure the buffers are clean >> before running the benchmarks, especially when comparing different versions >> and languages. >> >> Having said that, the erlang version does look slow, especially when >> compared with the shell equivalent "time find /usr/share -print | wc -l" >> >> $ sudo blockdev --flushbufs /dev/... >> >> $ time find /usr/share -print | wc -l >> 186911 >> >> real 0m32.446s >> user 0m0.796s >> sys 0m1.808s >> >> $ time find /usr/share -print | wc -l >> 186911 >> >> real 0m0.336s >> user 0m0.152s >> sys 0m0.200s >> >> >> Perhaps there is room for improvement within the library itself! >> >> Cheers, >> f. >> >> >> On 10/12/16 09:20, Frank Muller wrote: >>> >>> Combining previous hints (Benoit, Sergej): >>> >>> -module(directory). >>> -include_lib("kernel/include/file.hrl"). >>> -export([walker/1]). >>> >>> walker(Path) -> >>> case file:read_file_info(Path, [raw]) of >>> {ok, #file_info{type = regular}} -> >>> 1; >>> _ -> %% not care about symlink for now, assume a directory >>> Children = filelib:wildcard(Path ++ "/*"), >>> lists:foldl(fun(P, N) -> N + walker(P) end, 0, Children) >>> end. >>> >>>> timer:tc(fun() -> directory:walker("/usr/share") end). >>> >>> {1611688, /28953 /} >>> >>> I'm only counting number of files in this case. >>> >>> /Frank >>> >>> Le sam. 10 d?c. 2016 ? 10:05, Sergej Jure?ko >> > a ?crit : >>> >>> read_file_info does the job of is_dir and file_size in a single call. >>> That was the intention. >>> >>> Also use file:read_file_info(name,[raw]) >>> >>> >>> Sergej >>> >>>> On 10 Dec 2016, at 09:42, Benoit Chesneau >>> > wrote: >>>> >>>> this is kind of bullshit (sorry ;).... at the end this is what does >>>> the helpers in filelib: >>>> >>>> https://github.com/erlang/otp/blob/maint/lib/stdlib/src/filelib.erl#L257 >>>> >>>> except if you have a better algorithm in mind i don't se the point of >>>> rewriting something that is aleaready existing ... >>>> >>>> On Sat, 10 Dec 2016 at 09:36, Sergej Jure?ko >>>> > wrote: >>>> >>>> Stop using filelib functions. Use file:read_file_info and >>>> file:list_dir. >>>> >>>> Sergej >>>> >>>> On Dec 10, 2016 9:29 AM, "Frank Muller" >>>> > wrote: >>>> >>>> Hi Stanislaw >>>> >>>> First, I don't care if I've to use documented/undocumented >>>> calls as long as I can achieve my goal: faster dir walking. >>>> >>>> And you're right, here is a detailed comparison with other >>>> scripting languages: >>>> >>>> In my /usr/share, there?s: >>>> 2580 directories >>>> 28953 files >>>> >>>> 1. Erlang (no io:format/1, just recurse): >>>> >>>> walk(Dir) -> >>>> {ok, Files} = file:list_dir(Dir), >>>> walk(Dir, Files). >>>> >>>> walk(Dir, [ Basename | Rest ]) -> >>>> Path = filename:join([ Dir, Basename ]), >>>> case filelib:is_dir(Path) of >>>> true -> >>>> walk(Path); >>>> false -> >>>> %% io:format("~s~n", [Path]), >>>> filelib:file_size(Path) >>>> end, >>>> walk(Dir, Rest); >>>> walk(_, []) -> >>>> ok. >>>> >>>> timer:tc(fun() -> directoy:walker("/usr/share") end). >>>> {4662361 ,ok} >>>> >>>> 2. Python (this code even count the size of dir): >>>> From: >>>> http://stackoverflow.com/questions/1392413/calculating-a-directory-size-using-python >>>> >>>> import os >>>> def get_size(start_path = '.'): >>>> total_size = 0 >>>> for dirpath, dirnames, filenames in os.walk(start_path): >>>> for f in filenames: >>>> fp = os.path.join(dirpath, f) >>>> total_size += os.path.getsize(fp) >>>> return total_size >>>> >>>> print get_size() >>>> >>>> $ cd /usr/share >>>> $ time dir_walker.py >>>> 432034130 >>>> 0.25 real 0.13 user 0.10 sys >>>> >>>> 2. Perl (same, count dir size) >>>> http://www.perlmonks.org/?node_id=168974 >>>> >>>> use File::Find; >>>> my $size = 0; >>>> find(sub { $size += -s if -f $_ }, "/usr/share"); >>>> >>>> $ time perl dir_walker.pl >>>> 432034130 >>>> 0.13 real 0.05 user 0.08 sys >>>> >>>> 3. Ruby (same, count dir size): >>>> >>>> def directory_size(path) >>>> path << '/' unless path.end_with?('/') >>>> raise RuntimeError, "#{path} is not a directory" unless >>>> File.directory?(path) >>>> total_size = 0 >>>> Dir["#{path}**/*"].each do |f| >>>> total_size += File.size(f) if File.file?(f) && >>>> File.size?(f) >>>> end >>>> total_size >>>> end >>>> puts directory_size '/usr/share? >>>> >>>> $ time walker.rb >>>> 432028422 >>>> >>>> 0.21 real 0.09 user 0.11 sys >>>> >>>> 4. Lua: >>>> From: http://lua-users.org/wiki/DirTreeIterator >>>> >>>> require "lfs" >>>> >>>> function dirtree(dir) >>>> assert(dir and dir ~= "", "directory parameter is missing >>>> or empty") >>>> if string.sub(dir, -1) == "/" then >>>> dir=string.sub(dir, 1, -2) >>>> end >>>> >>>> local function yieldtree(dir) >>>> for entry in lfs.dir(dir) do >>>> if entry ~= "." and entry ~= ".." then >>>> entry=dir.."/"..entry >>>> local attr=lfs.attributes(entry) >>>> coroutine.yield(entry,attr) >>>> if attr.mode == "directory" then >>>> yieldtree(entry) >>>> end >>>> end >>>> end >>>> end >>>> >>>> return coroutine.wrap(function() yieldtree(dir) end) >>>> end >>>> >>>> for filename, attr in dirtree("/usr/share") do >>>> print(attr.mode, filename) >>>> end >>>> >>>> $ luarocks install luafilesystem >>>> $ time lua walker.lua > /dev/null >>>> 0.30 real 0.16 user 0.14 sys >>>> >>>> Do you need more? >>>> >>>> Thanks for you help. >>>> /Frank >>>> >>>> Le sam. 10 d?c. 2016 ? 00:51, Stanislaw Klekot >>>> > a ?crit : >>>> >>>> On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller >>>> wrote: >>>> >>>> > I would like to improve the speed of my directory >>>> walker. >>>> >>>> > >>>> >>>> > walk(Dir) -> >>>> >>>> > {ok, Files} = prim_file:list_dir(Dir), >>>> >>>> > walk(Dir, Files). >>>> >>>> >>>> >>>> Why prim_file:list_dir() instead of file:list_dir()? The >>>> former is >>>> >>>> undocumented internal function. >>>> >>>> >>>> >>>> [...] >>>> >>>> > Compared to almost anything i found on the web, it?s >>>> still very slow: >>>> >>>> > > timer:tc(fun() -> dir:walk("/usr/share") end). >>>> >>>> > {4662361,ok} >>>> >>>> >>>> >>>> What is it this "anything you found on the web"? And how >>>> did you run >>>> >>>> your comparisons? There's a large difference between >>>> first and second >>>> >>>> consequent run caused by OS' directory cache, and there's >>>> large >>>> >>>> difference between simply walking through the directory >>>> and walking with >>>> >>>> printing something to the screen for every file. >>>> >>>> >>>> >>>> Then there's also your using filelib:is_dir() and then >>>> >>>> filelib:file_size(), which means two stat(2) calls, while >>>> you only need >>>> >>>> to do it once per file (file:read_file_info()). >>>> >>>> >>>> >>>> -- >>>> >>>> Stanislaw Klekot >>>> >>>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From max.lapshin@REDACTED Sat Dec 10 14:36:45 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 10 Dec 2016 16:36:45 +0300 Subject: [erlang-questions] Fast directory walker In-Reply-To: <01cbcb75-68dc-1d4d-eb69-49ae338e19f4@anydata.co.uk> References: <20161209235027.GA2814@jarowit.net> <97AE1B90-8D72-4C39-82F8-33FB127322C8@gmail.com> <3f1637b8-8f53-e2b7-dd0b-7255896cd32c@anydata.co.uk> <01cbcb75-68dc-1d4d-eb69-49ae338e19f4@anydata.co.uk> Message-ID: When you just call file:list_dir, you make gen_server:call to a singleton process. When you start driver and make prim_file calls to it, you are working on your own thread. Of course you must avoid lists if you do micro benchmarks: list_dir(Port, Dir) -> {ok, Entries} = prim_file:list_dir(H, Dir), [list_dir(Port, <>) || E <- Entries]. and of course, there should not be any magical performance. Tests on your laptop are usually void and useless. Take loaded server and make tests on it. How your software will work when HDD is responding during 30 seconds. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikpelinux@REDACTED Sat Dec 10 15:30:05 2016 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Sat, 10 Dec 2016 15:30:05 +0100 Subject: [erlang-questions] Fast directory walker In-Reply-To: <20161209235027.GA2814@jarowit.net> References: <20161209235027.GA2814@jarowit.net> Message-ID: <22604.4461.917426.414421@gargle.gargle.HOWL> Stanislaw Klekot writes: > On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller wrote: > > I would like to improve the speed of my directory walker. > > > > walk(Dir) -> > > {ok, Files} = prim_file:list_dir(Dir), > > walk(Dir, Files). > > Why prim_file:list_dir() instead of file:list_dir()? The former is > undocumented internal function. list_dir can be a very time-consuming operation, and in those cases using file:list_dir would block the single file server for everything else. We routinely use prim_file:list_dir to reduce the negative effects of accessing large directories. From frank.muller.erl@REDACTED Sat Dec 10 18:33:30 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sat, 10 Dec 2016 17:33:30 +0000 Subject: [erlang-questions] Fast directory walker In-Reply-To: <22604.4461.917426.414421@gargle.gargle.HOWL> References: <20161209235027.GA2814@jarowit.net> <22604.4461.917426.414421@gargle.gargle.HOWL> Message-ID: All in one file: -module(dir). -include_lib("kernel/include/file.hrl"). -export([ walker/1, walker1/1, walker2/1 ]). %% original walker(Path) -> %% io:format("~s~n", [Path]), case file:read_file_info(Path) of {ok, #file_info{type = regular}} -> 1; _ -> %% not care about symlink for nor, assume a directory Children = filelib:wildcard(Path ++ "/*"), lists:foldl(fun(P, N) -> N + walker(P) end, 0, Children) end. %% Sergej version + [raw] option walker1(Path) -> case prim_file:list_dir(Path) of {ok,L} -> walker1(Path,L,0); _ -> 0 end. walker1(Pth,["."|T],Sz) -> walker1(Pth,T,Sz); walker1(Pth,[".."|T],Sz) -> walker1(Pth,T,Sz); walker1(Pth,[H|T],Sz) -> Nm = Pth++"/"++H, case prim_file:read_file_info(Nm, [raw]) of {ok,#file_info{type = regular, size = FS}} -> walker1(Pth,T,Sz+FS); {ok,#file_info{type = directory}} -> case prim_file:list_dir(Nm) of {ok,L} -> walker1(Pth, T, walker1(Nm,L,Sz)); _ -> walker1(Pth, T, Sz) end; _ -> walker1(Pth,T,Sz) end; walker1(_,[],Sz) -> Sz. %% Sergej version + Max's hint walker2(Path) -> {ok, Port} = prim_file:start(), case prim_file:list_dir(Port, Path) of {ok,L} -> walker2(Port,Path,L,0); _ -> 0 end. walker2(Port,Pth,["."|T],Sz) -> walker2(Port,Pth,T,Sz); walker2(Port,Pth,[".."|T],Sz) -> walker2(Port,Pth,T,Sz); walker2(Port,Pth,[H|T],Sz) -> Nm = Pth++"/"++H, case prim_file:read_file_info(Nm, [raw]) of {ok,#file_info{type = regular, size = FS}} -> walker2(Port,Pth,T,Sz+FS); {ok,#file_info{type = directory}} -> case prim_file:list_dir(Port,Nm) of {ok,L} -> walker2(Port,Pth,T,walker2(Port,Nm,L,Sz)); _ -> walker2(Port,Pth, T, Sz) end; _ -> walker2(Port,Pth,T,Sz) end; walker2(_,_,[],Sz) -> Sz. 1> timer:tc(fun() -> dir:walker("/usr/share") end). {1538933,28941} 2> timer:tc(fun() -> dir:walker1("/usr/share") end). {1492408,447632520} 3> timer:tc(fun() -> dir:walker2("/usr/share") end). {1477578,447632520} Getting close to 1sec. Any other ideas for improvement? /Frank Le sam. 10 d?c. 2016 ? 15:30, Mikael Pettersson a ?crit : > Stanislaw Klekot writes: > > > On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller wrote: > > > > I would like to improve the speed of my directory walker. > > > > > > > > walk(Dir) -> > > > > {ok, Files} = prim_file:list_dir(Dir), > > > > walk(Dir, Files). > > > > > > Why prim_file:list_dir() instead of file:list_dir()? The former is > > > undocumented internal function. > > > > list_dir can be a very time-consuming operation, and in those cases > > using file:list_dir would block the single file server for everything > > else. We routinely use prim_file:list_dir to reduce the negative > > effects of accessing large directories. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dangud@REDACTED Sat Dec 10 19:46:34 2016 From: dangud@REDACTED (Dan Gudmundsson) Date: Sat, 10 Dec 2016 18:46:34 +0000 Subject: [erlang-questions] Fast directory walker In-Reply-To: References: <20161209235027.GA2814@jarowit.net> <22604.4461.917426.414421@gargle.gargle.HOWL> Message-ID: erl +A0 to turn off async fil? access? Or isn?t that used for list dir and fil? info? Den l?r 10 dec 2016 18:33Frank Muller skrev: > All in one file: > > -module(dir). > > -include_lib("kernel/include/file.hrl"). > > -export([ walker/1, > walker1/1, > walker2/1 ]). > > %% original > walker(Path) -> > %% io:format("~s~n", [Path]), > case file:read_file_info(Path) of > > {ok, #file_info{type = regular}} -> > 1; > _ -> %% not care about symlink for nor, assume a directory > > Children = filelib:wildcard(Path ++ "/*"), > lists:foldl(fun(P, N) -> N + walker(P) end, 0, Children) > end. > > %% Sergej version + [raw] option > > walker1(Path) -> > case prim_file:list_dir(Path) of > {ok,L} -> > walker1(Path,L,0); > _ -> > 0 > end. > walker1(Pth,["."|T],Sz) -> > walker1(Pth,T,Sz); > walker1(Pth,[".."|T],Sz) -> > walker1(Pth,T,Sz); > walker1(Pth,[H|T],Sz) -> > Nm = Pth++"/"++H, > case prim_file:read_file_info(Nm, [raw]) of > > {ok,#file_info{type = regular, size = FS}} -> > walker1(Pth,T,Sz+FS); > {ok,#file_info{type = directory}} -> > case prim_file:list_dir(Nm) of > {ok,L} -> > walker1(Pth, T, walker1(Nm,L,Sz)); > _ -> > walker1(Pth, T, Sz) > end; > _ -> > walker1(Pth,T,Sz) > end; > walker1(_,[],Sz) -> > Sz. > > > > %% Sergej version + Max's hint > walker2(Path) -> > {ok, Port} = prim_file:start(), > case prim_file:list_dir(Port, Path) of > {ok,L} -> > walker2(Port,Path,L,0); > _ -> > 0 > end. > > walker2(Port,Pth,["."|T],Sz) -> > walker2(Port,Pth,T,Sz); > walker2(Port,Pth,[".."|T],Sz) -> > walker2(Port,Pth,T,Sz); > walker2(Port,Pth,[H|T],Sz) -> > > Nm = Pth++"/"++H, > case prim_file:read_file_info(Nm, [raw]) of > > {ok,#file_info{type = regular, size = FS}} -> > walker2(Port,Pth,T,Sz+FS); > > {ok,#file_info{type = directory}} -> > case prim_file:list_dir(Port,Nm) of > {ok,L} -> > walker2(Port,Pth,T,walker2(Port,Nm,L,Sz)); > _ -> > walker2(Port,Pth, T, Sz) > end; > _ -> > walker2(Port,Pth,T,Sz) > end; > walker2(_,_,[],Sz) -> > Sz. > > > 1> timer:tc(fun() -> dir:walker("/usr/share") end). > {1538933,28941} > 2> timer:tc(fun() -> dir:walker1("/usr/share") end). > {1492408,447632520} > 3> timer:tc(fun() -> dir:walker2("/usr/share") end). > {1477578,447632520} > > Getting close to 1sec. Any other ideas for improvement? > > > /Frank > > Le sam. 10 d?c. 2016 ? 15:30, Mikael Pettersson a > ?crit : > > Stanislaw Klekot writes: > > > On Fri, Dec 09, 2016 at 11:15:58PM +0000, Frank Muller wrote: > > > > I would like to improve the speed of my directory walker. > > > > > > > > walk(Dir) -> > > > > {ok, Files} = prim_file:list_dir(Dir), > > > > walk(Dir, Files). > > > > > > Why prim_file:list_dir() instead of file:list_dir()? The former is > > > undocumented internal function. > > > > list_dir can be a very time-consuming operation, and in those cases > > using file:list_dir would block the single file server for everything > > else. We routinely use prim_file:list_dir to reduce the negative > > effects of accessing large directories. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t.greenwoodgeer@REDACTED Sun Dec 11 02:29:19 2016 From: t.greenwoodgeer@REDACTED (Todd Greenwood-Geer) Date: Sat, 10 Dec 2016 17:29:19 -0800 Subject: [erlang-questions] Erlang: searching for a convincing argument Message-ID: <1615e11e-760a-483e-c1ba-ddff49e476f1@gmail.com> Hello all, OVERVIEW I've been a long-time Erlang/OTP fan...but I'm caught in a catch-22. For years, I've wanted to write a substantial system in Erlang/OTP...but I've been stymied b/c none of my colleagues or managers wanted to risk investing in this unknown-to-them platform. W/O any significant personal experience, I have yet to convince anyone that this would be a great path... So I've watched, time and time again, as various portions of the Erlang platform be poorly implemented in Java, Python, etc. etc...only to wind up wading through the inevitable profusion of bugs and scalability issues. CHALLENGE So, I challenged a long-time colleague to come up with a problem that would convince him that he should have implemented some problem in Erlang. He came up with this problem from a previous company... Periodically, say once a month, his prev company had to send out a mass email (templated) to their customer base. This grew from thousands to millions over the course of a few years. As the number of emails increased, their simple script started to run from minutes to hours to days... Furthermore, the email providers impose throughput constraints such that you can only send X number of emails per hour in the first hour, Y in the second, etc. The ramp-ups were explicitly documented and not adhering to them could get you throttled or black-listed. TEST CASE To showcase why Erlang is so great, I suggested that we could model external and internal failures and show that the only end result was a change in throughput. Erlang Nodes Fake SMTP Relay Throughput ------------------|-------------------------|----------------------- [1...N] [1] FailureRate1 FailureRate2 emails/sec * Inputs: 1 million email addresses, read from a file. * The Erlang nodes have code/app that processes the email addresses. * The Fake SMTP Relay just receives the emails and writes them to a file or /dev/null, whatever. * FailureRate1 is the percentage of Nodes that are dead (killed, etc.), simulating hardware faults etc. * FailureRate2 is the percentage of errors that the Fake SMTP relay reports, simulating 3rd party endpoint failures. I also suggested that originally, he give me an incorrect address for his SMTP relay, and I'd perform a hot code update to correct this. Pretty sick (cool) right? DESIGN At this point, I have some design questions... # Design 1 : Use a database I could dump the 1Million email addresses into a database (ETS/mnesia, etc.) and have processes reading/writing state to the db as they process each email. But he was unimpressed, as this is so much like just using any old language that uses the db as a work queue (so long as the db is replicated). # Design 2 : Use erlang processes, all in memory at the same time I could create an Erlang process for each email address... but scaling is memory bound, so this doesn't seem right at all. # Design 3: Use erlang processes, but only read in a subset of the email addresses at a time I could read from the input file and create only M erlang processes at a time and then write to ETS to signify completion. But if I'm writing to ETS, I may as well read all the data into ETS/mnesia at the start, and use it as a work queue. Back to Design # 1. Ok, putting that question aside for a second... DISTRIBUTION # Distribution 3 : How to distribute the app for resiliency? I'd like to run this on N nodes and have a random reaper (chaos monkey, whatever) kill the Erlang nodes (or the underlying VM) randomly to simulate hardware errors. Again, my thinking feels constrained. I keep coming back to: stuff the state in a db, spin up a supervisor and a bunch of worker processes on a separate node. If the node with the worker processes dies, the supervisor creates worker processes on a different node, and so forth. Despite having read all the books I can find on Erlang and reading the list for years now... I still don't really know the best way to have supervisors living on separate nodes, reacting to node failures such that the application picks up where it left off on a new node. It might be simpler to just fan the workers out across all the nodes since the state is maintained in the db/queue. I'd still have to maintain state in the db to ensure that all the processes are adhering to the rate limits. But the central question remains, if I'm randomly killing my nodes, and if the node with the supervisor dies, what then? How do I replicate supervisors? What's the pattern that I'm missing here? FINAL So, when my friend first described his problem, I thought, "I can do this in like, 4 lines of Erlang!" Then he started adding constraints like rate limitations, etc... and I thought, "Ok, 20-30 lines...". Now, looking at the problem, I've spent a couple hours just writing it down and trying to consider how to solve it... I have more questions than when I started. I'd love to hear thoughts about solving this (or similar) problem(s). I've come to the conclusion that I cannot evangelize Erlang if I don't know how to solve even simple problems with it. -Todd BTW - over the years I've read: https://joearms.github.io/index.html Programming Erlang (ed 2 is on order) Learn You Some Erlang For Great Good Erlang and OTP In Action Erlang Programming From mononcqc@REDACTED Sun Dec 11 03:52:45 2016 From: mononcqc@REDACTED (Fred Hebert) Date: Sat, 10 Dec 2016 21:52:45 -0500 Subject: [erlang-questions] Erlang: searching for a convincing argument In-Reply-To: <1615e11e-760a-483e-c1ba-ddff49e476f1@gmail.com> References: <1615e11e-760a-483e-c1ba-ddff49e476f1@gmail.com> Message-ID: <20161211025244.GK684@ferdmbp.local> On 12/10, Todd Greenwood-Geer wrote: >Periodically, say once a month, his prev company had to send out a mass >email (templated) to their customer base. This grew from thousands to >millions over the course of a few years. As the number of emails >increased, their simple script started to run from minutes to hours to >days... Furthermore, the email providers impose throughput constraints >such that you can only send X number of emails per hour in the first >hour, Y in the second, etc. The ramp-ups were explicitly documented and >not adhering to them could get you throttled or black-listed. > This sounds like a perfect example for the techniques I recently mentioned in http://ferd.ca/handling-overload.html Specifically, you could use libraries such as safetyvalve[1] or jobs[2] to schedule all your tasks according to the required limits of the system, or use circuit breakers of all kinds[3][4][5] to regulate that load as a reaction from the systems you communicate with. Some of the strengths of Erlang don't come from its semantics, but on the strong operational focus of its community, and its dedication to writing solving problems in a space where such challenges are common. I'm a bit short on time tonight to respond with relation with distribution (it's a complex topic!); if I remember I'll try to come back to it later. [1] https://github.com/jlouis/safetyvalve [2] https://github.com/uwiger/jobs [3] https://github.com/klarna/circuit_breaker [4] https://github.com/jlouis/fuse [5] https://github.com/mmzeeman/breaky From mjtruog@REDACTED Sun Dec 11 04:51:20 2016 From: mjtruog@REDACTED (Michael Truog) Date: Sat, 10 Dec 2016 19:51:20 -0800 Subject: [erlang-questions] Erlang: searching for a convincing argument In-Reply-To: <1615e11e-760a-483e-c1ba-ddff49e476f1@gmail.com> References: <1615e11e-760a-483e-c1ba-ddff49e476f1@gmail.com> Message-ID: <584CCD38.60701@gmail.com> On 12/10/2016 05:29 PM, Todd Greenwood-Geer wrote: > Hello all, > > OVERVIEW > > I've been a long-time Erlang/OTP fan...but I'm caught in a catch-22. For years, I've wanted to write a substantial system in Erlang/OTP...but I've been stymied b/c none of my colleagues or managers wanted to risk investing in this unknown-to-them platform. W/O any significant personal experience, I have yet to convince anyone that this would be a great path... So I've watched, time and time again, as various portions of the Erlang platform be poorly implemented in Java, Python, etc. etc...only to wind up wading through the inevitable profusion of bugs and scalability issues. Generally, the main argument for Erlang use is for its ability to provide fault-tolerance for source code. The scalability advantage can be provided in various programming languages with an actor model library. It is important to notice Erlang is a functional programming language that attempts to avoid side-effects (errors managing state lead to system instability). Avoiding instability on the server-side is important, due to many clients depending on the server. > > CHALLENGE > > So, I challenged a long-time colleague to come up with a problem that would convince him that he should have implemented some problem in Erlang. He came up with this problem from a previous company... Periodically, say once a month, his prev company had to send out a mass email (templated) to their customer base. This grew from thousands to millions over the course of a few years. As the number of emails increased, their simple script started to run from minutes to hours to days... Furthermore, the email providers impose throughput constraints such that you can only send X number of emails per hour in the first hour, Y in the second, etc. The ramp-ups were explicitly documented and not adhering to them could get you throttled or black-listed. > > TEST CASE > > To showcase why Erlang is so great, I suggested that we could model external and internal failures and show that the only end result was a change in throughput. > > Erlang Nodes Fake SMTP Relay Throughput > ------------------|-------------------------|----------------------- > [1...N] [1] > FailureRate1 FailureRate2 emails/sec > > * Inputs: 1 million email addresses, read from a file. > * The Erlang nodes have code/app that processes the email addresses. > * The Fake SMTP Relay just receives the emails and writes them to a file or /dev/null, whatever. > * FailureRate1 is the percentage of Nodes that are dead (killed, etc.), simulating hardware faults etc. > * FailureRate2 is the percentage of errors that the Fake SMTP relay reports, simulating 3rd party endpoint failures. > > I also suggested that originally, he give me an incorrect address for his SMTP relay, and I'd perform a hot code update to correct this. Pretty sick (cool) right? > > DESIGN > > At this point, I have some design questions... > > # Design 1 : Use a database > I could dump the 1Million email addresses into a database (ETS/mnesia, etc.) and have processes reading/writing state to the db as they process each email. But he was unimpressed, as this is so much like just using any old language that uses the db as a work queue (so long as the db is replicated). > > # Design 2 : Use erlang processes, all in memory at the same time > I could create an Erlang process for each email address... but scaling is memory bound, so this doesn't seem right at all. > > # Design 3: Use erlang processes, but only read in a subset of the email addresses at a time > I could read from the input file and create only M erlang processes at a time and then write to ETS to signify completion. But if I'm writing to ETS, I may as well read all the data into ETS/mnesia at the start, and use it as a work queue. Back to Design # 1. > > Ok, putting that question aside for a second... Design #1 without ETS or mnesia would be a good approach. A SQL or NoSQL database would be picked based on the usage patterns and operational concerns. The impressing part is having a system that can survive failure scenarios independent of the database, so runtime problems related to the source code that were unanticipated by the developers. I would choose to use http://cloudi.org/ due to it saving me development time. I would probably have 2 CloudI services, 1 for periodically reading from the database (ServiceA) and 1 for sending an email based on the contents of a received service request (ServiceB) where ServiceA sends to ServiceB. That allows the concurrency concerns and throughput concerns to be service configuration settings, due to the various features in CloudI. > > DISTRIBUTION > > # Distribution 3 : How to distribute the app for resiliency? > I'd like to run this on N nodes and have a random reaper (chaos monkey, whatever) kill the Erlang nodes (or the underlying VM) randomly to simulate hardware errors. Again, my thinking feels constrained. I keep coming back to: stuff the state in a db, spin up a supervisor and a bunch of worker processes on a separate node. If the node with the worker processes dies, the supervisor creates worker processes on a different node, and so forth. > > Despite having read all the books I can find on Erlang and reading the list for years now... I still don't really know the best way to have supervisors living on separate nodes, reacting to node failures such that the application picks up where it left off on a new node. It might be simpler to just fan the workers out across all the nodes since the state is maintained in the db/queue. I'd still have to maintain state in the db to ensure that all the processes are adhering to the rate limits. > > But the central question remains, if I'm randomly killing my nodes, and if the node with the supervisor dies, what then? How do I replicate supervisors? What's the pattern that I'm missing here? CloudI provides node auto-discovery with LAN multicast and AWS EC2 API usage, so that can help simplify managing a group of nodes, with the same services on each so that failover can occur based on the routing of CloudI service requests. For system testing, the service configuration options monkey_chaos and monkey_latency exist, so that means Chaos Monkey testing and/or Latency Monkey testing can occur with a tweak to the configuration of existing services, for a separate environment, automated tests, or whatever is required. > > FINAL > > So, when my friend first described his problem, I thought, "I can do this in like, 4 lines of Erlang!" Then he started adding constraints like rate limitations, etc... and I thought, "Ok, 20-30 lines...". Now, looking at the problem, I've spent a couple hours just writing it down and trying to consider how to solve it... I have more questions than when I started. > > I'd love to hear thoughts about solving this (or similar) problem(s). I've come to the conclusion that I cannot evangelize Erlang if I don't know how to solve even simple problems with it. Not sure about the line count, since it depends on the details, like what templating needs to be supported for creating the email data. I am also assuming that the DB is modified elsewhere, and that the Erlang-side is updating the DB as it processes the DB entries that require emails to be sent. CloudI allows services in any supported programming language, so: C/C++, Erlang/Elixir, Java, JavaScript/node.js, Perl, PHP, Python and/or Ruby, which helps avoid risk people perceive from usage of Erlang (difficulty finding people with Erlang knowledge, expensive to train people, easy to find Java developers, etc.). CloudI allows you to utilize Erlang's advantages even if you are unable to develop with Erlang (for example, the Java CloudI tutorial is implemented only in Java). I created CloudI due to being in similar situations as you in the past, and I understand that CloudI saves me development time, so I would naturally use it to solve a problem like this. However, everyone approaches a problem differently, so there are many ways of approaching this with Erlang. Best Regards, Michael > > -Todd > > BTW - over the years I've read: > https://joearms.github.io/index.html > Programming Erlang (ed 2 is on order) > Learn You Some Erlang For Great Good > Erlang and OTP In Action > Erlang Programming > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From mark@REDACTED Mon Dec 12 03:39:22 2016 From: mark@REDACTED (Mark Steele) Date: Sun, 11 Dec 2016 21:39:22 -0500 Subject: [erlang-questions] Erlang for an audio streaming server Message-ID: Here's an audio streaming solution I wrote: https://github.com/marksteele/emusak Streams to a browser based UI using ffmpeg to transcode to mp3 (due to limitations in formats supported by browsers). Cheers, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Mon Dec 12 12:54:40 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Mon, 12 Dec 2016 11:54:40 +0000 Subject: [erlang-questions] Excludes apps from a rebar release Message-ID: Hi I'm able to generate a release with rebar and all's good. Except that wx kept being added even if I'm not using it at all. Same for observer, and mnesia. How I can set my reltool.config file to excluded these unused apps? Thank you. /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Mon Dec 12 15:00:55 2016 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Mon, 12 Dec 2016 15:00:55 +0100 Subject: [erlang-questions] R17 - Possible wedged scheduler In-Reply-To: References: Message-ID: First advice is: try a newer release. 1. If this is a bug then it might have been fixed in newer releases as quite a lot have changed in the purging machinery, especially in OTP-19. 2. The probablity to get help with troubleshooting from the Erlang/OTP team increases with the OTP version number. Having said that; It looks like the code_server is doing a code purge operation. If it is indeed hanging in code_server:cpc_recv/4 then a crash dump would be helpful to debug. erlang:halt("Crash it!"). /Sverker, Erlang/OTP On 12/09/2016 05:35 PM, Matthew Evans wrote: > Happened again, it appears that code_server is wedged: > > > admin@REDACTED:~$ doErlangFun "erlang:process_info(whereis(code_server))." > > [{registered_name,code_server}, > > {current_function,{code_server,cpc_recv,4}}, > > {initial_call,{erlang,apply,2}}, > > {status,waiting}, > > {message_queue_len,23}, > > {messages,[{code_call,<6805.4097.0>,{ensure_loaded,switch_type_module}}, > > {code_call,<6805.4146.0>,{ensure_loaded,switch_type_module}}, > > {code_call,<6805.941.0>,{ensure_loaded,pc_port_autoneg}}, > > {code_call,<6805.541.0>,{ensure_loaded,plexxiStatistics_types}}, > > {code_call,<6805.520.0>,{ensure_loaded,switch_type_module}}, > > {code_call,<6805.5123.0>,{ensure_loaded,secondary_erlang_node}}, > > {code_call,<6805.5122.0>,{ensure_loaded,secondary_erlang_node}}, > > {code_call,<6805.5162.0>,{ensure_loaded,icmp}}, > > {code_call,<6805.5321.0>, > > {ensure_loaded,mac_entries_record_handler}}, > > {code_call,<6805.5483.0>,{ensure_loaded,icmp}}, > > {code_call,<6805.6647.0>,{ensure_loaded,icmp}}, > > {code_call,<6805.7232.0>,{ensure_loaded,icmp}}, > > {code_call,<6805.7274.0>,{ensure_loaded,icmp}}, > > {code_call,<6805.7304.0>,{ensure_loaded,icmp}}, > > {code_call,<6805.8889.0>, > > {ensure_loaded,mac_entries_record_handler}}, > > {code_call,<6805.8951.0>, > > {ensure_loaded,mac_entries_record_handler}}, > > {code_call,<6805.576.0>, > > {ensure_loaded,cross_connect_unicast_utils}}, > > {code_call,<6805.19300.12>,{ensure_loaded,shell}}, > > {code_call,<6805.20313.12>,{ensure_loaded,shell}}, > > {code_call,<6805.21339.12>,{ensure_loaded,dbg}}, > > {code_call,<6805.31109.13>,get_mode}, > > {code_call,<6805.1255.14>,get_mode}, > > {system,{<6805.2521.14>,#Ref<6805.0.23.35356>},get_status}]}, > > {links,[<6805.11.0>]}, > > {dictionary,[{any_native_code_loaded,false}]}, > > {trap_exit,true}, > > {error_handler,error_handler}, > > {priority,normal}, > > {group_leader,<6805.9.0>}, > > {total_heap_size,86071}, > > {heap_size,10958}, > > {stack_size,25}, > > {reductions,13172282}, > > {garbage_collection,[{min_bin_vheap_size,46422}, > > {min_heap_size,233}, > > {fullsweep_after,65535}, > > {minor_gcs,71}]}, > > {suspending,[]}] > > admin@REDACTED:~$ > > > > ________________________________ > From:erlang-questions-bounces@REDACTED on behalf of Matthew Evans > Sent: Friday, December 9, 2016 9:56 AM > To: Erlang/OTP discussions > Subject: [erlang-questions] R17 - Possible wedged scheduler > > > Hi, > > > We just hit a situation where it appeared that 1 scheduler was wedged. Some parts of our application were working, but others appeared to be stuck. I could connect via a cnode application and an escript, but I couldn't connect via the Erlang shell. We have an escript that does rpc calls, some worked, others (e.g. anything to the code server or tracing failed) failed. > > > CPU load was minimal at the time, and heart didn't complain. We only have a single NIF, but this is not called on this hardware variant. We do use CNODE to talk to C applications. > > > We are running R17, Intel quad core CPU on Debian. > > > This is the first time this has been seen, so the questions are: > > > 1. Has anyone seen this before? > > 2. What can we do if we hit this condition in the future to debug? > > 3. Since heart doesn't detect this can anyone think of any alternative mechanisms? > > > Thanks > > > Matt > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From list1@REDACTED Mon Dec 12 20:18:25 2016 From: list1@REDACTED (Grzegorz Junka) Date: Mon, 12 Dec 2016 19:18:25 +0000 Subject: [erlang-questions] Excludes apps from a rebar release In-Reply-To: References: Message-ID: On 12/12/2016 11:54, Frank Muller wrote: > Hi > > I'm able to generate a release with rebar and all's good. > > Except that wx kept being added even if I'm not using it at all. Same > for observer, and mnesia. > > How I can set my reltool.config file to excluded these unused apps? > > Thank you. > > /Frank > Hi Frank, I am excluding all applications by default and then including only those I specifically want in the release. Have a look here: https://github.com/builderl/ex1_simple_cache/blob/master/etc/reltool.config#L48 Greg From frank.muller.erl@REDACTED Mon Dec 12 21:40:09 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Mon, 12 Dec 2016 20:40:09 +0000 Subject: [erlang-questions] Excludes apps from a rebar release In-Reply-To: References: Message-ID: Hi Junka You made my day. Actually, these two are needed: {incl_cond, exclude}, {exclude_archive_filters,[".*"]}, Then, add your app dependencies as described in your link. Thank you!!! /Frank Le lun. 12 d?c. 2016 ? 20:18, Grzegorz Junka a ?crit : > > > On 12/12/2016 11:54, Frank Muller wrote: > > > Hi > > > > > > I'm able to generate a release with rebar and all's good. > > > > > > Except that wx kept being added even if I'm not using it at all. Same > > > for observer, and mnesia. > > > > > > How I can set my reltool.config file to excluded these unused apps? > > > > > > Thank you. > > > > > > /Frank > > > > > > > Hi Frank, > > I am excluding all applications by default and then including only those > > I specifically want in the release. Have a look here: > > > > > https://github.com/builderl/ex1_simple_cache/blob/master/etc/reltool.config#L48 > > > > Greg > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From garryh@REDACTED Mon Dec 12 22:31:42 2016 From: garryh@REDACTED (Garry Hodgson) Date: Mon, 12 Dec 2016 16:31:42 -0500 Subject: [erlang-questions] running centos release tarball on debian Message-ID: <3a1ae603-5024-63ca-b1dd-73968a0332c4@att.com> We are using a jenkins build server on centos to create our releases, using rebar3. The deployed release tarball includes the erlang runtime. Recently we needed to deploy it on a remote system that runs Ubuntu. The developers used a tool called alien to convert the .rpm that installs the release tarball into a .deb package that Ubuntu expects. The application won't start, complaining that it can't find libcrypto, though OpenSSL is installed. I'm guessing that the compiled erlang runtime, having been built for centos, is looking for libs in different places, perhaps with different names. Is there a way to make this work by working around with env vars, symlinks, or other post-buld kludgery? Or do we need to just bite the bullet and set up a Jenkins build server for ubuntu? Are others deploying releases on different Linux distributions? Any suggestions or lessons learned would be appreciated. Thanks From erlang.org@REDACTED Mon Dec 12 22:53:58 2016 From: erlang.org@REDACTED (Stanislaw Klekot) Date: Mon, 12 Dec 2016 22:53:58 +0100 Subject: [erlang-questions] running centos release tarball on debian In-Reply-To: <3a1ae603-5024-63ca-b1dd-73968a0332c4@att.com> References: <3a1ae603-5024-63ca-b1dd-73968a0332c4@att.com> Message-ID: <20161212215358.GA18189@jarowit.net> On Mon, Dec 12, 2016 at 04:31:42PM -0500, Garry Hodgson wrote: > We are using a jenkins build server on centos to create our > releases, using rebar3. The deployed release tarball includes the > erlang runtime. Recently we needed to deploy it on a remote system > that runs Ubuntu. The developers used a tool called alien to convert > the .rpm that installs the release tarball into a .deb package that > Ubuntu expects. Erlang programmers, meet Linux sysadmins. Linux sysadmins, meet Erlang programmers. You can _finally_ make acquaintance. > The application won't start, complaining that it can't find > libcrypto, though OpenSSL is installed. I'm guessing that the > compiled erlang runtime, having been built for centos, is looking > for libs in different places, perhaps with different names. Of course it is. Even if it could find OpenSSL libs, there are high chances it would die on missing symbols or some other linking error. There's also the thing of host OS' architecture (i386 vs. amd64 (Debian) or i386 vs x86_64 (Red Hat)), though I doubt it somehow. > Is there a way to make this work by working around with env vars, > symlinks, or other post-buld kludgery? Or do we need to just bite > the bullet and set up a Jenkins build server for ubuntu? Either you bite the bullet or build a release without including ERTS, leaving the task of providing it for deployment phase. Then receivers of your release would need to compile Erlang at most. > Are others > deploying releases on different Linux distributions? Any suggestions > or lessons learned would be appreciated. I don't think many of Erlang programmers pay attention to working along with their OS. Otherwise we wouldn't have that poor cooperation between Erlang and package systems with initscripts. -- Stanislaw Klekot From s.safarov@REDACTED Tue Dec 13 10:20:10 2016 From: s.safarov@REDACTED (Sergey Safarov) Date: Tue, 13 Dec 2016 09:20:10 +0000 Subject: [erlang-questions] How to math list of record In-Reply-To: References: Message-ID: I want check type of passed arguments. In following example i can check record type -record(person, {first_name, last_name}). my_func(#person{} = Arg) -> Arg. It works. But how to do check of list of records? Like this -record(person, {first_name, last_name}). my_func([#person{} | ... ] = Arg) -> Arg. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Tue Dec 13 10:42:09 2016 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 13 Dec 2016 10:42:09 +0100 Subject: [erlang-questions] How to math list of record In-Reply-To: References: Message-ID: <20161213094209.GA20933@erix.ericsson.se> On Tue, Dec 13, 2016 at 09:20:10AM +0000, Sergey Safarov wrote: > I want check type of passed arguments. In following example i can check > record type The term "check type" is misleading. In Erlang you select a code path depending on the type i.e checking when doing something with it. So if you have a function that does nothing but crash for the wrong type that might be regarded as a type checking function, but in Erlang you in general do not check the type until you do something with the term. Unless you want to check the type in an API to report errors as early as possible... > > -record(person, {first_name, last_name}). > my_func(#person{} = Arg) -> Arg. > > It works. > But how to do check of list of records? Like this > > -record(person, {first_name, last_name}). > my_func([#person{} | ... ] = Arg) -> Arg. You will have to traverse the list to check all elements of the list. This is O(1) so in general you do not check a whole list just for fun. Instead you let it crash on invalid types while processing. Since processing a list is O(1) you loose just a constant factor by processing the list twice, so you could add a type checking pass over the list, but that is mostly just adding complexity... Just checking a record type still does not check the record fields nor contradictions between fields or any other problems with the records. my_func_check([#person{} | Persons]) -> my_func_check(Persons); my_func_check([]) -> ok. my_func(Arg) -> my_func_check(Arg), ... > > Thanks -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From dm.klionsky@REDACTED Tue Dec 13 10:44:36 2016 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Tue, 13 Dec 2016 12:44:36 +0300 Subject: [erlang-questions] running centos release tarball on debian In-Reply-To: <3a1ae603-5024-63ca-b1dd-73968a0332c4@att.com> References: <3a1ae603-5024-63ca-b1dd-73968a0332c4@att.com> Message-ID: <584FC304.5030407@gmail.com> To build for different OSes I use different docker images. Here's https://github.com/ten0s/alley-docker docker setup I developed at my previous job, but admins didn't want(know?) to put it under Jenkins so you will see a lot of other stuff. Look at centos{5,6} dirs for examples. Now almost the same docker setup, but for centos7 and centos5.i386 runs under Jenkins on AWS. On success I put tarballs to S3. Your task would be to create a new image for Ubuntu and install needed build deps. I like this approach because it allows to have a simple Jenkins slave with only docker and Jenkins agent on it. On 12/13/2016 12:31 AM, Garry Hodgson wrote: > We are using a jenkins build server on centos to create our releases, > using rebar3. The deployed release tarball includes the erlang > runtime. Recently we needed to deploy it on a remote system that runs > Ubuntu. The developers used a tool called alien to convert the .rpm > that installs the release tarball into a .deb package that Ubuntu > expects. > > The application won't start, complaining that it can't find libcrypto, > though OpenSSL is installed. I'm guessing that the compiled erlang > runtime, having been built for centos, is looking for libs in > different places, perhaps with different names. > > Is there a way to make this work by working around with env vars, > symlinks, or other post-buld kludgery? Or do we need to just bite the > bullet and set up a Jenkins build server for ubuntu? Are others > deploying releases on different Linux distributions? Any suggestions > or lessons learned would be appreciated. > > Thanks > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- BR, Dmitry From dmkolesnikov@REDACTED Tue Dec 13 10:53:03 2016 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Tue, 13 Dec 2016 11:53:03 +0200 Subject: [erlang-questions] running centos release tarball on debian In-Reply-To: <3a1ae603-5024-63ca-b1dd-73968a0332c4@att.com> References: <3a1ae603-5024-63ca-b1dd-73968a0332c4@att.com> Message-ID: <22769D43-E1DB-4171-ACFD-6B1A153E0DF7@gmail.com> Hello, You can solve the issue if you compile OpenSSL and Erlang yourself at build machine, disable-dynamic-ssl-lib. The resulting relx image would not require libcrypto anymore so that you can convert it. You can get hint on build step here https://github.com/docker-file/erlang/blob/master/Dockerfile Best Regards, Dmitry > On Dec 12, 2016, at 11:31 PM, Garry Hodgson wrote: > > We are using a jenkins build server on centos to create our releases, using rebar3. The deployed release tarball includes the erlang runtime. Recently we needed to deploy it on a remote system that runs Ubuntu. The developers used a tool called alien to convert the .rpm that installs the release tarball into a .deb package that Ubuntu expects. > > The application won't start, complaining that it can't find libcrypto, though OpenSSL is installed. I'm guessing that the compiled erlang runtime, having been built for centos, is looking for libs in different places, perhaps with different names. > > Is there a way to make this work by working around with env vars, symlinks, or other post-buld kludgery? Or do we need to just bite the bullet and set up a Jenkins build server for ubuntu? Are others deploying releases on different Linux distributions? Any suggestions or lessons learned would be appreciated. > > Thanks > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From sweden.feng@REDACTED Tue Dec 13 10:54:39 2016 From: sweden.feng@REDACTED (Alex Feng) Date: Tue, 13 Dec 2016 10:54:39 +0100 Subject: [erlang-questions] Number of threads are doubled when enable smp ? Message-ID: Hi, When I was doing some performance testing, I found that the number of threads just doubled if I enabled the smp (erl -smp auto). I haven't started to run anything, just wondering what the extra threads are doing with smp enabled ? %% with smp option enabled, we have 22 thread running under beam.smp 484: ~/erlang/test > top -H -p 699 top - 10:41:40 up 205 days, 22:31, 72 users, load average: 1.22, 1.65, 1.97 Tasks: 22 total, 0 running, 22 sleeping, 0 stopped, 0 zombie Cpu(s): 6.0%us, 4.7%sy, 0.0%ni, 88.8%id, 0.0%wa, 0.0%hi, 0.4%si, 0.0%st Mem: 32241M total, 28556M used, 3684M free, 842M buffers Swap: 0M total, 0M used, 0M free, 14000M cached PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 699 20 0 1757m 20m 2980 S 0 0.1 0:00.05 beam.smp 703 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 704 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 705 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 706 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 707 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 708 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 709 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 710 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 711 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 712 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 713 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 714 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 716 20 0 1757m 20m 2980 S 0 0.1 0:00.03 beam.smp 717 20 0 1757m 20m 2980 S 0 0.1 0:00.22 beam.smp 718 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 719 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 720 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 721 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp 722 20 0 1757m 20m 2980 S 0 0.1 0:00.02 beam.smp 723 20 0 1757m 20m 2980 S 0 0.1 0:00.02 beam.smp 724 20 0 1757m 20m 2980 S 0 0.1 0:00.00 beam.smp %% with smp option disabled, we have 11 threads running under beam 484: ~/erlang/test > top -H -p 31599 top - 10:39:35 up 205 days, 22:29, 72 users, load average: 1.21, 1.88, 2.08 Tasks: 11 total, 0 running, 11 sleeping, 0 stopped, 0 zombie Cpu(s): 50.0%us, 0.0%sy, 0.0%ni, 50.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 32241M total, 28549M used, 3691M free, 842M buffers Swap: 0M total, 0M used, 0M free, 14000M cached PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31599 20 0 1579m 15m 2836 S 0 0.0 0:00.32 beam 31607 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam 31608 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam 31609 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam 31610 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam 31611 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam 31612 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam 31613 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam 31614 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam 31615 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam 31616 20 0 1579m 15m 2836 S 0 0.0 0:00.00 beam Br, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From zkessin@REDACTED Tue Dec 13 11:32:43 2016 From: zkessin@REDACTED (Zachary Kessin) Date: Tue, 13 Dec 2016 12:32:43 +0200 Subject: [erlang-questions] Regexp Matching on Unicode Message-ID: Hi All I am hitting a bit of a wall here, I am building a lexer with leex and I really want to match on unicode chars, there is a regex class \p{Letter} but that does not seem to work in erlang. I really want is a way to say "Match a letter, but not a digit". So the \w would not work. Any ideas? -- Zach Kessin SquareTarget Twitter: @zkessin Skype: zachkessin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Tue Dec 13 11:40:04 2016 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Tue, 13 Dec 2016 11:40:04 +0100 Subject: [erlang-questions] Regexp Matching on Unicode In-Reply-To: References: Message-ID: Make sure to escape the property escape character and to also pass the [unicode] flag when compiling and it should be good to go: 28> {ok, Reg} = re:compile("\\p{L}{5}", []). {ok,{re_pattern,0,0,0, <<69,82,67,80,77,0,0,0,0,0,0,0,1,0,0,0,255,255,255,255, 255,255,...>>}} 29> re:run(<<"?????"/utf8>>, Reg). nomatch 30> {ok, RegUni} = re:compile("\\p{L}{5}", [unicode]). {ok,{re_pattern,0,1,0, <<69,82,67,80,77,0,0,0,0,8,0,0,1,0,0,0,255,255,255,255, 255,255,...>>}} 31> re:run(<<"?????"/utf8>>, RegUni). {match,[{0,15}]} *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D On Tue, Dec 13, 2016 at 11:32 AM, Zachary Kessin wrote: > Hi All > > I am hitting a bit of a wall here, I am building a lexer with leex and I > really want to match on unicode chars, there is a regex class \p{Letter} > but that does not seem to work in erlang. I really want is a way to say > "Match a letter, but not a digit". So the \w would not work. Any ideas? > > -- > Zach Kessin > SquareTarget > Twitter: @zkessin > Skype: zachkessin > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo@REDACTED Tue Dec 13 11:41:50 2016 From: hugo@REDACTED (Hugo Mills) Date: Tue, 13 Dec 2016 10:41:50 +0000 Subject: [erlang-questions] Regexp Matching on Unicode In-Reply-To: References: Message-ID: <20161213104150.GA7447@carfax.org.uk> On Tue, Dec 13, 2016 at 12:32:43PM +0200, Zachary Kessin wrote: > Hi All > > I am hitting a bit of a wall here, I am building a lexer with leex and I > really want to match on unicode chars, there is a regex class \p{Letter} > but that does not seem to work in erlang. I really want is a way to say > "Match a letter, but not a digit". So the \w would not work. Any ideas? I think if you want unicode support, you need to write your own lexer, or use something other than leex. It's a bit limited in what it supports. I went through this earlier this year, and ended up writing my own -- partly for that reason, and partly to do with the way I wanted to process block comments. Hugo. -- Hugo Mills | "There's more than one way to do it" is not a hugo@REDACTED carfax.org.uk | commandment. It is a dire warning. http://carfax.org.uk/ | PGP: E2AB1DE4 | -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From jose.valim@REDACTED Tue Dec 13 12:14:00 2016 From: jose.valim@REDACTED (=?UTF-8?Q?Jos=C3=A9_Valim?=) Date: Tue, 13 Dec 2016 12:14:00 +0100 Subject: [erlang-questions] Regexp Matching on Unicode In-Reply-To: References: Message-ID: Apologies, just after Hugo Mills' reply I noticed your question was related to leex and not re. leex does not support unicode character classes, such as \p or \w. It does accept unicode as its input as well as unicode characters as literals in your rules, such as [?-?], the pound sign, etc. *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D On Tue, Dec 13, 2016 at 11:40 AM, Jos? Valim < jose.valim@REDACTED> wrote: > Make sure to escape the property escape character and to also pass the > [unicode] flag when compiling and it should be good to go: > > 28> {ok, Reg} = re:compile("\\p{L}{5}", []). > {ok,{re_pattern,0,0,0, > <<69,82,67,80,77,0,0,0,0,0,0,0,1,0,0,0,255,255,255,255, > 255,255,...>>}} > 29> re:run(<<"?????"/utf8>>, Reg). > nomatch > > 30> {ok, RegUni} = re:compile("\\p{L}{5}", [unicode]). > {ok,{re_pattern,0,1,0, > <<69,82,67,80,77,0,0,0,0,8,0,0,1,0,0,0,255,255,255,255, > 255,255,...>>}} > 31> re:run(<<"?????"/utf8>>, RegUni). > {match,[{0,15}]} > > > > > *Jos? Valim* > www.plataformatec.com.br > Skype: jv.ptec > Founder and Director of R&D > > On Tue, Dec 13, 2016 at 11:32 AM, Zachary Kessin > wrote: > >> Hi All >> >> I am hitting a bit of a wall here, I am building a lexer with leex and I >> really want to match on unicode chars, there is a regex class \p{Letter} >> but that does not seem to work in erlang. I really want is a way to say >> "Match a letter, but not a digit". So the \w would not work. Any ideas? >> >> -- >> Zach Kessin >> SquareTarget >> Twitter: @zkessin >> Skype: zachkessin >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.safarov@REDACTED Tue Dec 13 11:17:17 2016 From: s.safarov@REDACTED (Sergey Safarov) Date: Tue, 13 Dec 2016 10:17:17 +0000 Subject: [erlang-questions] How to math list of record In-Reply-To: <20161213094209.GA20933@erix.ericsson.se> References: <20161213094209.GA20933@erix.ericsson.se> Message-ID: Thank you Raimo for datailed responce. ??, 13 ???. 2016 ?. ? 12:42, Raimo Niskanen < raimo+erlang-questions@REDACTED>: > On Tue, Dec 13, 2016 at 09:20:10AM +0000, Sergey Safarov wrote: > > I want check type of passed arguments. In following example i can check > > record type > > The term "check type" is misleading. In Erlang you select a code path > depending on the type i.e checking when doing something with it. So if you > have a function that does nothing but crash for the wrong type that might > be regarded as a type checking function, but in Erlang you in general do > not check the type until you do something with the term. > > Unless you want to check the type in an API to report errors as early as > possible... > > > > > -record(person, {first_name, last_name}). > > my_func(#person{} = Arg) -> Arg. > > > > It works. > > But how to do check of list of records? Like this > > > > -record(person, {first_name, last_name}). > > my_func([#person{} | ... ] = Arg) -> Arg. > > You will have to traverse the list to check all elements of the list. This > is O(1) so in general you do not check a whole list just for fun. Instead > you let it crash on invalid types while processing. > > Since processing a list is O(1) you loose just a constant factor by > processing the list twice, so you could add a type checking pass over the > list, but that is mostly just adding complexity... Just checking a record > type still does not check the record fields nor contradictions between > fields or any other problems with the records. > > my_func_check([#person{} | Persons]) -> > my_func_check(Persons); > my_func_check([]) -> > ok. > > my_func(Arg) -> > my_func_check(Arg), > ... > > > > > Thanks > > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.ansell@REDACTED Tue Dec 13 11:38:06 2016 From: lawrence.ansell@REDACTED (Lawrence Ansell) Date: Tue, 13 Dec 2016 10:38:06 +0000 Subject: [erlang-questions] Anyone Buy VEB #EEFSF2017 Yet? Message-ID: Has anyone purchased Very Early Bird Ticket for Erlang Elixir San Francisco Bay Area 2017 Conference yet? Very Early Bird tickets are on sale now and you guys get to save $400 compared to standard price. Here: http://www.erlang-factory.com/sfbay2017/ EEF SF Bay Area 2017 conference 23 - 24 March. Training 20 - 22 and 27 - 30 March. Tutorials 25 March. LE -------------- next part -------------- An HTML attachment was scrubbed... URL: From unix1@REDACTED Tue Dec 13 16:52:32 2016 From: unix1@REDACTED (Unix One) Date: Tue, 13 Dec 2016 15:52:32 +0000 Subject: [erlang-questions] Anyone Buy VEB #EEFSF2017 Yet? In-Reply-To: References: Message-ID: On 12/13/2016 02:38 AM, Lawrence Ansell wrote: > Has anyone purchased Very Early Bird Ticket for Erlang Elixir San > Francisco Bay Area 2017 Conference yet? > > Very Early Bird tickets are on sale now and you guys get to save $400 > compared to standard price. > > Here: http://www.erlang-factory.com/sfbay2017/ No. My current employer won't sponsor this and it doesn't quite fit in my personal budget :(. Otherwise, yeah, I've been to several of them, and it's a great a great deal and worth every cent. From eric.pailleau@REDACTED Tue Dec 13 18:19:47 2016 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Tue, 13 Dec 2016 18:19:47 +0100 Subject: [erlang-questions] running centos release tarball on debian In-Reply-To: <22769D43-E1DB-4171-ACFD-6B1A153E0DF7@gmail.com> References: <3a1ae603-5024-63ca-b1dd-73968a0332c4@att.com> <22769D43-E1DB-4171-ACFD-6B1A153E0DF7@gmail.com> Message-ID: <4eifu2vkb1ps0etn2uu6t7j1.1481649587032@email.android.com> And once you have a portable release, you can avoid using alien and create debian package on centos using https://github.com/crownedgrouse/debut in rebar overlay or use https://github.com/crownedgrouse/debbie module in your own escript. Regards "Envoy? depuis mon mobile " Eric ---- Dmitry Kolesnikov a ?crit ---- >Hello, > >You can solve the issue if you compile OpenSSL and Erlang yourself at build machine, disable-dynamic-ssl-lib. The resulting relx image would not require libcrypto anymore so that you can convert it. > >You can get hint on build step here https://github.com/docker-file/erlang/blob/master/Dockerfile > >Best Regards, >Dmitry > > >> On Dec 12, 2016, at 11:31 PM, Garry Hodgson wrote: >> >> We are using a jenkins build server on centos to create our releases, using rebar3. The deployed release tarball includes the erlang runtime. Recently we needed to deploy it on a remote system that runs Ubuntu. The developers used a tool called alien to convert the .rpm that installs the release tarball into a .deb package that Ubuntu expects. >> >> The application won't start, complaining that it can't find libcrypto, though OpenSSL is installed. I'm guessing that the compiled erlang runtime, having been built for centos, is looking for libs in different places, perhaps with different names. >> >> Is there a way to make this work by working around with env vars, symlinks, or other post-buld kludgery? Or do we need to just bite the bullet and set up a Jenkins build server for ubuntu? Are others deploying releases on different Linux distributions? Any suggestions or lessons learned would be appreciated. >> >> Thanks >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Tue Dec 13 18:59:11 2016 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Tue, 13 Dec 2016 18:59:11 +0100 Subject: [erlang-questions] running centos release tarball on debian In-Reply-To: <22769D43-E1DB-4171-ACFD-6B1A153E0DF7@gmail.com> References: <3a1ae603-5024-63ca-b1dd-73968a0332c4@att.com> <22769D43-E1DB-4171-ACFD-6B1A153E0DF7@gmail.com> Message-ID: I forgot to say that another way is to create a debian package with only the beam files that depends to another debian package installing the good Erlang VM for Ubuntu. This imply to build it on Ubuntu or use ubuntu 's Erlang VM in official repositories, but they are probably not up-to-date with last Erlang VM. "Envoy? depuis mon mobile " Eric ---- Dmitry Kolesnikov a ?crit ---- >Hello, > >You can solve the issue if you compile OpenSSL and Erlang yourself at build machine, disable-dynamic-ssl-lib. The resulting relx image would not require libcrypto anymore so that you can convert it. > >You can get hint on build step here https://github.com/docker-file/erlang/blob/master/Dockerfile > >Best Regards, >Dmitry > > >> On Dec 12, 2016, at 11:31 PM, Garry Hodgson wrote: >> >> We are using a jenkins build server on centos to create our releases, using rebar3. The deployed release tarball includes the erlang runtime. Recently we needed to deploy it on a remote system that runs Ubuntu. The developers used a tool called alien to convert the .rpm that installs the release tarball into a .deb package that Ubuntu expects. >> >> The application won't start, complaining that it can't find libcrypto, though OpenSSL is installed. I'm guessing that the compiled erlang runtime, having been built for centos, is looking for libs in different places, perhaps with different names. >> >> Is there a way to make this work by working around with env vars, symlinks, or other post-buld kludgery? Or do we need to just bite the bullet and set up a Jenkins build server for ubuntu? Are others deploying releases on different Linux distributions? Any suggestions or lessons learned would be appreciated. >> >> Thanks >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From garry@REDACTED Tue Dec 13 19:20:00 2016 From: garry@REDACTED (Garry Hodgson) Date: Tue, 13 Dec 2016 13:20:00 -0500 Subject: [erlang-questions] running centos release tarball on debian In-Reply-To: <22769D43-E1DB-4171-ACFD-6B1A153E0DF7@gmail.com> References: <3a1ae603-5024-63ca-b1dd-73968a0332c4@att.com> <22769D43-E1DB-4171-ACFD-6B1A153E0DF7@gmail.com> Message-ID: <54def611-7994-811e-9455-5f6600034c5b@att.com> Many thanks to all who replied. I expect we'll address this at the source, and upgrade our automated builds to handle both Centos and Ubunu. The Docker stuff looks intriguing as well. On 12/13/16 4:53 AM, Dmitry Kolesnikov wrote: > Hello, > > You can solve the issue if you compile OpenSSL and Erlang yourself at build machine, disable-dynamic-ssl-lib. The resulting relx image would not require libcrypto anymore so that you can convert it. > > You can get hint on build step here https://github.com/docker-file/erlang/blob/master/Dockerfile > > Best Regards, > Dmitry > > >> On Dec 12, 2016, at 11:31 PM, Garry Hodgson wrote: >> >> We are using a jenkins build server on centos to create our releases, using rebar3. The deployed release tarball includes the erlang runtime. Recently we needed to deploy it on a remote system that runs Ubuntu. The developers used a tool called alien to convert the .rpm that installs the release tarball into a .deb package that Ubuntu expects. >> >> The application won't start, complaining that it can't find libcrypto, though OpenSSL is installed. I'm guessing that the compiled erlang runtime, having been built for centos, is looking for libs in different places, perhaps with different names. >> >> Is there a way to make this work by working around with env vars, symlinks, or other post-buld kludgery? Or do we need to just bite the bullet and set up a Jenkins build server for ubuntu? Are others deploying releases on different Linux distributions? Any suggestions or lessons learned would be appreciated. >> >> Thanks >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From s.safarov@REDACTED Tue Dec 13 20:52:48 2016 From: s.safarov@REDACTED (Sergey Safarov) Date: Tue, 13 Dec 2016 19:52:48 +0000 Subject: [erlang-questions] why not works command from shell Message-ID: When executed following command from erl shell all works as expected 1> inet_res:nslookup('example.org','in','a'). {ok,{dns_rec,{dns_header,2,true,query,false,false,true,true, false,0}, [{dns_query,"example.org",a,in}], [{dns_rr,"example.org",a,in,0,86400, {93,184,216,34}, undefined,[],false}], [{dns_rr,"example.org",ns,in,0,86399,"a.iana-servers.net", undefined,[],false}, {dns_rr,"example.org",ns,in,0,86399,"b.iana-servers.net", undefined,[],false}], [{dns_rr,"a.iana-servers.net",a,in,0,172799, {199,43,135,53}, undefined,[],false}, {dns_rr,"a.iana-servers.net",aaaa,in,0,172799, {8193,1280,143,0,0,0,0,83}, undefined,[],false}, {dns_rr,"b.iana-servers.net",a,in,0,172799, {199,43,133,53}, undefined,[],false}, {dns_rr,"b.iana-servers.net",aaaa,in,0,172799, {8193,1280,141,0,0,0,0,83}, undefined,[],false}]}} But when i try executed following command from bash i got error [root@REDACTED ~]# erl -noshell -s inet_res nslookup 'example.org' 'in' 'a' {"init terminating in do_boot",{undef,[{inet_res,nslookup,[['example.org ',in,a]],[]},{init,start_it,1,[]},{init,start_em,1,[]}]}} init terminating in do_boot () Crash dump is being written to: erl_crash.dump...done Why it take place? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo@REDACTED Tue Dec 13 21:47:49 2016 From: hugo@REDACTED (Hugo Mills) Date: Tue, 13 Dec 2016 20:47:49 +0000 Subject: [erlang-questions] why not works command from shell In-Reply-To: References: Message-ID: <20161213204749.GE7447@carfax.org.uk> On Tue, Dec 13, 2016 at 07:52:48PM +0000, Sergey Safarov wrote: > When executed following command from erl shell all works as expected > > 1> inet_res:nslookup('example.org','in','a'). inet_res:nslookup/3 > [root@REDACTED ~]# erl -noshell -s inet_res nslookup 'example.org' 'in' 'a' > {"init terminating in do_boot",{undef,[{inet_res,nslookup,[['example.org > ',in,a]],[]},{init,start_it,1,[]},{init,start_em,1,[]}]}} > init terminating in do_boot () " -s Mod [Func [Arg1, Arg2, ...]](init flag): Makes init call the specified function. Func defaults to start. If no arguments are provided, the function is assumed to be of arity 0. Otherwise it is assumed to be of arity 1, taking the list [Arg1,Arg2,...] as argument. All arguments are passed as atoms. See init(3erl). " Note the fourth sentence: "Otherwise it is assumed to be of arity 1 [...]" so you're actually calling inet_res:nslookup(['example.org', in, a]) (which doesn't exist) Hugo. -- Hugo Mills | Two things came out of Berkeley in the 1960s: LSD hugo@REDACTED carfax.org.uk | and Unix. This is not a coincidence. http://carfax.org.uk/ | PGP: E2AB1DE4 | -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From s.safarov@REDACTED Tue Dec 13 22:07:56 2016 From: s.safarov@REDACTED (Sergey Safarov) Date: Tue, 13 Dec 2016 21:07:56 +0000 Subject: [erlang-questions] why not works command from shell In-Reply-To: <20161213204749.GE7447@carfax.org.uk> References: <20161213204749.GE7447@carfax.org.uk> Message-ID: Thank you Hugo I created wrapper [root@REDACTED test]# cat test.erl -module(test). -export([lookup/0, lookup_wait/0]). lookup() -> io:format("Called lookup~n"), Result = inet_res:nslookup('example.org',in,a), io:format("Finished lookup. Result: ~p~n", [Result]). lookup_wait() -> timer:sleep(5000), io:format("Called lookup_wait~n"), Result = inet_res:nslookup('example.org',in,a), io:format("Finished lookup_wait. Result: ~p~n", [Result]). And get following results. NXDOMAIN for lookup call. [root@REDACTED test]# erl -noshell -run test lookup Called lookup Finished lookup. Result: {error,nxdomain} Correct resolv when called lookup_wait. [root@REDACTED test]# erl -noshell -run test lookup_wait Called lookup_wait Finished lookup_wait. Result: {ok, {dns_rec, {dns_header,1,true,query,false,false,true, true,false,0}, [{dns_query,"example.org",a,in}], [{dns_rr,"example.org",a,in,0,81792, {93,184,216,34}, undefined,[],false}], [{dns_rr,"example.org",ns,in,0,81791, "a.iana-servers.net",undefined,[],false}, {dns_rr,"example.org",ns,in,0,81791, "b.iana-servers.net",undefined,[],false}], [{dns_rr,"a.iana-servers.net",a,in,0,168191, {199,43,135,53}, undefined,[],false}, {dns_rr,"a.iana-servers.net",aaaa,in,0, 168191, {8193,1280,143,0,0,0,0,83}, undefined,[],false}, {dns_rr,"b.iana-servers.net",a,in,0,168191, {199,43,133,53}, undefined,[],false}, {dns_rr,"b.iana-servers.net",aaaa,in,0, 168191, {8193,1280,141,0,0,0,0,83}, undefined,[],false}]}} Why resolv not work first five second of program start? What to do to resolv this issue? Sergey ??, 13 ???. 2016 ?. ? 23:47, Hugo Mills : > On Tue, Dec 13, 2016 at 07:52:48PM +0000, Sergey Safarov wrote: > > When executed following command from erl shell all works as expected > > > > 1> inet_res:nslookup('example.org','in','a'). > > inet_res:nslookup/3 > > > [root@REDACTED ~]# erl -noshell -s inet_res nslookup 'example.org' 'in' > 'a' > > {"init terminating in do_boot",{undef,[{inet_res,nslookup,[['example.org > > ',in,a]],[]},{init,start_it,1,[]},{init,start_em,1,[]}]}} > > init terminating in do_boot () > > " -s Mod [Func [Arg1, Arg2, ...]](init flag): > > Makes init call the specified function. Func defaults to > start. If no arguments are provided, the function is > assumed to be of arity 0. Otherwise it is assumed to be of > arity 1, taking the list [Arg1,Arg2,...] as argument. All > arguments are passed as atoms. See init(3erl). > " > > Note the fourth sentence: "Otherwise it is assumed to be of arity 1 > [...]" > > so you're actually calling > > inet_res:nslookup(['example.org', in, a]) (which doesn't exist) > > Hugo. > > -- > Hugo Mills | Two things came out of Berkeley in the 1960s: LSD > hugo@REDACTED carfax.org.uk | and Unix. This is not a coincidence. > http://carfax.org.uk/ | > PGP: E2AB1DE4 | > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Tue Dec 13 23:35:24 2016 From: lloyd@REDACTED (lloyd@REDACTED) Date: Tue, 13 Dec 2016 17:35:24 -0500 (EST) Subject: [erlang-questions] Erlang Factory Conference S.F. Message-ID: <1481668524.3913134@apps.rackspace.com> Ah, it's approaching time again for the Erlang Factory Conference S.F. I attended several years ago and found it educational and inspiring. But $890 for limited Early Bird tickets? $1290 for laggards? I suppose if you have corporate funds behind you it may feel of little matter--- particularly if expense accounts cover travel and accommodation. But if you're an army-of-one Erlang developer it's quite impossible. Pity. LRP From aschultz@REDACTED Wed Dec 14 11:00:36 2016 From: aschultz@REDACTED (Andreas Schultz) Date: Wed, 14 Dec 2016 11:00:36 +0100 (CET) Subject: [erlang-questions] Erlang 19.2 ? Message-ID: <1945182666.100459.1481709636330.JavaMail.zimbra@tpip.net> Hi, There is a tag with OTP-19.2 on github (https://github.com/erlang/otp/releases/tag/OTP-19.2). Does that mean 19.2 is officially released? I haven't seen (or missed) an announcement and erlang.org does not have 19.2, yet. Regards Andreas From kennethlakin@REDACTED Wed Dec 14 11:20:21 2016 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Wed, 14 Dec 2016 02:20:21 -0800 Subject: [erlang-questions] why not works command from shell In-Reply-To: References: <20161213204749.GE7447@carfax.org.uk> Message-ID: <7215a879-4d5b-68f4-8636-c10a55c2fdfb@gmail.com> On 12/13/2016 01:07 PM, Sergey Safarov wrote: > Correct resolv when called lookup_wait. Yeah, this old bug! What version of Erlang are you using? I remember that I tracked this down to some bad math somewhere in the bowels of the resolver management code that meant that you couldn't do lookups until the node has been up for five seconds. I _think_ it has been fixed in recent versions of Erlang, but I'm not sure. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From sweden.feng@REDACTED Wed Dec 14 11:28:33 2016 From: sweden.feng@REDACTED (Alex Feng) Date: Wed, 14 Dec 2016 11:28:33 +0100 Subject: [erlang-questions] Hot code replacement issues ? Message-ID: Hi Erlangers, I am trying to understand how hot patch upgrade is done in Erlang, I got this piece of code from stackoverflow, but I ran into 2 issues when I played around with it. Issue 1) In the piece of code, I suppose to send a message 'upgrade' to the running process, then it will finish the upgrade. It is working fine with "?MODULE:loop();", but if I remove the prefix ?MODULE in the code, then the upgrade will be failed, I mean when I send message 'run' after upgrade neither the new message nor the old message are printed. Issue 2) I was told if I recompile and reload the updated file from shell, then it will be done as well. So, I did this from shell, update the source file -> compile the file -> reload it ->send msg 'run' to the same process -> 'new version' is not printed out. What could be the mistake? -module(test_hot_swap). -export([loop/0]). loop() -> receive upgrade -> code:purge(?MODULE), compile:file(?MODULE), code:load_file(?MODULE), ?MODULE:loop(); %%without ?MODULE, it has problem. run -> io:format("This is a new version ~n"), %% before upgrade, the text is 'old version'. loop(); _ -> loop() end. Thank you. Br, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.safarov@REDACTED Wed Dec 14 11:32:25 2016 From: s.safarov@REDACTED (Sergey Safarov) Date: Wed, 14 Dec 2016 10:32:25 +0000 Subject: [erlang-questions] why not works command from shell In-Reply-To: <7215a879-4d5b-68f4-8636-c10a55c2fdfb@gmail.com> References: <20161213204749.GE7447@carfax.org.uk> <7215a879-4d5b-68f4-8636-c10a55c2fdfb@gmail.com> Message-ID: Thank you Kennett for your confirmation. I think like you and created ticket https://bugs.erlang.org/browse/ERL-317 At present time I use 18.3.4.4 version from Fedora repo. ??, 14 ???. 2016, 13:20 Kenneth Lakin : > On 12/13/2016 01:07 PM, Sergey Safarov wrote: > > Correct resolv when called lookup_wait. > > Yeah, this old bug! > What version of Erlang are you using? I remember that I tracked this > down to some bad math somewhere in the bowels of the resolver management > code that meant that you couldn't do lookups until the node has been up > for five seconds. > I _think_ it has been fixed in recent versions of Erlang, but I'm not sure. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From taavi@REDACTED Wed Dec 14 11:34:57 2016 From: taavi@REDACTED (Taavi Talvik) Date: Wed, 14 Dec 2016 12:34:57 +0200 Subject: [erlang-questions] Hot code replacement issues ? In-Reply-To: References: Message-ID: Switch to new version of code happens, when fully qualified function is called: 1) Switch to new code version happens: loop(..) -> modulename:loop(). 2) Old code will remain in use: loop(..) -> loop(). best regards, taavi > On 14 Dec 2016, at 12:28, Alex Feng wrote: > > Hi Erlangers, > > I am trying to understand how hot patch upgrade is done in Erlang, I got this piece of code from stackoverflow, but I ran into 2 issues when I played around with it. > > Issue 1) In the piece of code, I suppose to send a message 'upgrade' to the running process, then it will finish the upgrade. It is working fine with "?MODULE:loop();", but if I remove the prefix ?MODULE in the code, then the upgrade will be failed, I mean when I send message 'run' after upgrade neither the new message nor the old message are printed. > > Issue 2) I was told if I recompile and reload the updated file from shell, then it will be done as well. > So, I did this from shell, update the source file -> compile the file -> reload it ->send msg 'run' to the same process -> 'new version' is not printed out. What could be the mistake? > > > > -module(test_hot_swap). > > -export([loop/0]). > > loop() -> > receive > upgrade -> > code:purge(?MODULE), > compile:file(?MODULE), > code:load_file(?MODULE), > ?MODULE:loop(); %%without ?MODULE, it has problem. > run -> > io:format("This is a new version ~n"), %% before upgrade, the text is 'old version'. > loop(); > _ -> > loop() > end. > > Thank you. > > Br, > Alex > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions ? The single biggest problem in communication is the illusion that it has taken place. (George Bernard Shaw) From bjorn-egil.xb.dahlberg@REDACTED Wed Dec 14 11:40:29 2016 From: bjorn-egil.xb.dahlberg@REDACTED (=?UTF-8?Q?Bj=c3=b6rn-Egil_Dahlberg_XB?=) Date: Wed, 14 Dec 2016 11:40:29 +0100 Subject: [erlang-questions] Erlang/OTP 19.2 has been released Message-ID: <5105752c-8238-7ba0-3c50-3b5cfc5d9c8b@ericsson.com> Erlang/OTP 19.2 is the second service release for the 19 major release. The service release contains mostly bug fixes and characteristics improvements. Some highlights for 19.2 * STDLIB: The new behaviour gen_statem has been improved with 3 new features: the possibility to use old style non-proxy timeouts for gen_statem:call/2,3, state entry code, and state timeouts. These are backwards compatible. Minor code and documentation improvements has been performed including a borderline semantics correction of timeout zero handling. * SSL: Experimental version of DTLS. It is runnable but not complete and cannot be considered reliable for production usage. To use DTLS add the option {protocol, dtls} to ssl:connect and ssl:listen. * SSH: Extended the option silently_accept_hosts for ssh:connect to make it possible for the client to check the SSH host key fingerprint string. Se the reference manual for SSH. * ~40 contributions since the previous service release OTP 19.1 You can find the README and the full listing of changes for this service release at http://www.erlang.org/download/otp_src_19.2.readme The source distribution and binary distributions for Windows can be downloaded from http://www.erlang.org/download/otp_src_19.2.tar.gz http://www.erlang.org/download/otp_win32_19.2.exe http://www.erlang.org/download/otp_win64_19.2.exe Note: To unpack the TAR archive you need a GNU TAR compatible program. For installation instructions please consult the README file that is part of the distribution. The Erlang/OTP source can also be found at GitHub on the official Erlang repository, https://github.com/erlang/otp with tag OTP-19.2 The on-line documentation can be found at: http://www.erlang.org/doc/ You can also download the complete HTML documentation or the Unix manual files http://www.erlang.org/download/otp_doc_html_19.2.tar.gz http://www.erlang.org/download/otp_doc_man_19.2.tar.gz Please report any new issues via Erlang/OTPs public issue tracker https://bugs.erlang.org We want to thank all of those who sent us patches, suggestions and bug reports! Thank you! The Erlang/OTP Team at Ericsson From kennethlakin@REDACTED Wed Dec 14 11:43:07 2016 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Wed, 14 Dec 2016 02:43:07 -0800 Subject: [erlang-questions] why not works command from shell In-Reply-To: References: <20161213204749.GE7447@carfax.org.uk> <7215a879-4d5b-68f4-8636-c10a55c2fdfb@gmail.com> Message-ID: <1886d61b-631c-c071-07af-e56e8b726969@gmail.com> On 12/14/2016 02:32 AM, Sergey Safarov wrote: > Thank you Kennett for your confirmation. > I think like you and created ticket https://bugs.erlang.org/browse/ERL-317 For reference, the old post where I tracked down the issue is here: http://erlang.org/pipermail/erlang-questions/2015-November/086806.html I glanced at inet_db.erl in Github master, and it looks like that area of the file hasn't changed since I made the post, but I've not made a careful study of the rest of the code. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From raimo+erlang-questions@REDACTED Wed Dec 14 12:18:14 2016 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 14 Dec 2016 12:18:14 +0100 Subject: [erlang-questions] why not works command from shell In-Reply-To: References: <20161213204749.GE7447@carfax.org.uk> <7215a879-4d5b-68f4-8636-c10a55c2fdfb@gmail.com> Message-ID: <20161214111814.GA87918@erix.ericsson.se> On Wed, Dec 14, 2016 at 10:32:25AM +0000, Sergey Safarov wrote: > Thank you Kennett for your confirmation. > I think like you and created ticket https://bugs.erlang.org/browse/ERL-317 Already fixed in OTP-19.0. See https://github.com/erlang/otp/pull/949 > > At present time I use 18.3.4.4 version from Fedora repo. > > ??, 14 ???. 2016, 13:20 Kenneth Lakin : > > > On 12/13/2016 01:07 PM, Sergey Safarov wrote: > > > Correct resolv when called lookup_wait. > > > > Yeah, this old bug! > > What version of Erlang are you using? I remember that I tracked this > > down to some bad math somewhere in the bowels of the resolver management > > code that meant that you couldn't do lookups until the node has been up > > for five seconds. > > I _think_ it has been fixed in recent versions of Erlang, but I'm not sure. > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From arunp@REDACTED Wed Dec 14 12:18:50 2016 From: arunp@REDACTED (Arun) Date: Wed, 14 Dec 2016 16:48:50 +0530 Subject: [erlang-questions] Mnesia crashes and restarts the Erlang VM repeatedly In-Reply-To: <53894986-0aa9-bef6-2bf0-572c76602e04@utl.in> References: <53894986-0aa9-bef6-2bf0-572c76602e04@utl.in> Message-ID: <58512A9A.9030102@utl.in> Hi, I am facing some serious issues with mnesia database. We have an erlang application using mnesia database as back-end. The application runs on the hardware and the IP address of the hardware will be changing based on its connectivity and so the erlang node name. Some times as soon as the node changes am getting an error log as given below and the VM restarts repeatedly. But if I delete the mnesia related files and restart again, application runs smoothly. For me data reliability and persistence are the prime concern so I can not go with the above mentioned method. Can somebody kindly assist me how to resolve this issue with out deleting mnesia schema. Current working directory:/root/rel-3/galaxy/lib/galaxy-1.0.0/priv Release number as string:3 PHY: mdio@REDACTED:01 - Link is Up - 100/Full 36524 01999.282 2001391.0 71.4 532699753591315.4 11871.3 0 Exec: /usr/lib/erlang/erts-8.0/bin/erlexec -boot /root/rel-3/galaxy/releases/0.0.1/galaxy -mode embedded -boot_var ERTS_LIB_DIR /usr/lib/erlang/erts-8.0/../lib -config /root/rel-3/galaxy/releases/0.0.1/sys.config -args_file /root/rel-3/galaxy/vm.args -- console Root: /root/rel-3/galaxy /root/rel-3/galaxy Erlang/OTP 19 [erts-8.0] [source] [async-threads:10] [kernel-poll:false] =INFO REPORT==== 17-Nov-2016::12:42:59 === disk_log: repairing "/root/mnesia_database/DECISION_TAB.LOG" ... Mnesia('scu_plus@REDACTED'): Data may be missing, Corrupt logfile deleted: "/root/mnesia_database/LATEST.LOG", {not_a_log_file, "/root/mnesia_database/LATEST.LOG"} =ERROR REPORT==== 17-Nov-2016::12:43:00 === Mnesia('scu_plus@REDACTED'): ** ERROR ** (core dumped to file:"/root/rel-3/galaxy/MnesiaCore.scu_plus@REDACTED") ** FATAL ** Failed to merge schema: Bad cookie in table definition configuration_table: 'scu_plus@REDACTED' = {cstruct,configuration_table,set,[],['scu_plus@REDACTED'],[],[],0,read_write,false,[],[],false,gen_record,[key,value,time_stamp],[],[],[],{{1479494951549325042,-134217623,1},'scu_plus@REDACTED'},{{2,0},[]}},'scu_plus@REDACTED' = {cstruct,configuration_table,set,[],['scu_plus@REDACTED'],[],[],0,read_write,false,[],[],false,gen_record,[key,value,time_stamp],[],[],[],{{1479575314787900797,-134217651,1},'scu_plus@REDACTED'},{{2,0},[]}} =ERROR REPORT==== 17-Nov-2016::12:43:10 === ** Generic server mnesia_monitor terminating ** Last message in was {'EXIT',<0.411.0>,killed} ** When Server state == {state,<0.411.0>,[],[],true,[],undefined,[],[]} ** Reason for termination == ** killed =ERROR REPORT==== 17-Nov-2016::12:43:10 === ** Generic server mnesia_recover terminating ** Last message in was {'EXIT',<0.411.0>,killed} ** When Server state == {state,<0.411.0>,undefined,undefined,undefined,0, false,true,[]} ** Reason for termination == ** killed =ERROR REPORT==== 17-Nov-2016::12:43:10 === ** Generic server mnesia_snmp_sup terminating ** Last message in was {'EXIT',<0.411.0>,killed} ** When Server state == {state, {local,mnesia_snmp_sup}, simple_one_for_one, [{child,undefined,mnesia_snmp_sup, {mnesia_snmp_hook,start,[]}, transient,3000,worker, [mnesia_snmp_sup,mnesia_snmp_hook, supervisor]}], undefined,0,86400000,[],0,mnesia_snmp_sup,[]} ** Reason for termination == ** killed =ERROR REPORT==== 17-Nov-2016::12:43:10 === ** Generic server mnesia_subscr terminating ** Last message in was {'EXIT',<0.411.0>,killed} ** When Server state == {state,<0.411.0>,12307} ** Reason for termination == ** {{aborted,{node_not_running,'scu_plus@REDACTED'}}, [{mnesia,abort,1,[{file,"mnesia.erl"},{line,318}]}, {mnesia_lib,del,2,[{file,"mnesia_lib.erl"},{line,485}]}, {mnesia_subscr,do_handle_exit,1,[{file,"mnesia_subscr.erl"},{line,508}]}, {mnesia_subscr,handle_exit,2,[{file,"mnesia_subscr.erl"},{line,502}]}, {mnesia_subscr,do_prepare_stop,2,[{file,"mnesia_subscr.erl"},{line,526}]}, {mnesia_subscr,terminate,2,[{file,"mnesia_subscr.erl"},{line,304}]}, {gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,629}]}, {gen_server,terminate,7,[{file,"gen_server.erl"},{line,795}]}]} =ERROR REPORT==== 17-Nov-2016::12:43:10 === Mnesia('scu_plus@REDACTED'): ** ERROR ** mnesia_event got unexpected event: {'EXIT', <0.413.0>, {aborted, {node_not_running, 'scu_plus@REDACTED'}}} =INFO REPORT==== 17-Nov-2016::12:43:10 === application: mnesia exited: {{shutdown,{failed_to_start_child,mnesia_kernel_sup,killed}}, {mnesia_sup,start,[normal,[]]}} type: permanent {"Kernel pid terminated",application_controller,"{application_start_failure,mnesia,{{shutdown,{failed_to_start_child,mnesia_kernel_sup,killed}},{mnesia_sup,start,[normal,[]]}}}"} Crash dump is being written to: erl_crash.dump... Thanks in advance, Arun -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Wed Dec 14 12:19:37 2016 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 14 Dec 2016 12:19:37 +0100 Subject: [erlang-questions] why not works command from shell In-Reply-To: <1886d61b-631c-c071-07af-e56e8b726969@gmail.com> References: <20161213204749.GE7447@carfax.org.uk> <7215a879-4d5b-68f4-8636-c10a55c2fdfb@gmail.com> <1886d61b-631c-c071-07af-e56e8b726969@gmail.com> Message-ID: <20161214111937.GB87918@erix.ericsson.se> On Wed, Dec 14, 2016 at 02:43:07AM -0800, Kenneth Lakin wrote: > On 12/14/2016 02:32 AM, Sergey Safarov wrote: > > Thank you Kennett for your confirmation. > > I think like you and created ticket https://bugs.erlang.org/browse/ERL-317 > > For reference, the old post where I tracked down the issue is here: > > http://erlang.org/pipermail/erlang-questions/2015-November/086806.html Already fixed in OTP-19.0. See https://github.com/erlang/otp/pull/949 > > I glanced at inet_db.erl in Github master, and it looks like that area > of the file hasn't changed since I made the post, but I've not made a > careful study of the rest of the code. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From walter.weinmann@REDACTED Wed Dec 14 14:18:28 2016 From: walter.weinmann@REDACTED (Walter Weinmann) Date: Wed, 14 Dec 2016 14:18:28 +0100 Subject: [erlang-questions] Windows 10 x64 - basho - ebloom - compilation environment Message-ID: I'm running Windows 10 x64 with Cygwin x64. *make* is. $ make -v GNU Make 4.2.1 Built for x86_64-unknown-cygwin *gcc *is: $ gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-cygwin/5.4.0/lto-wrapper.exe Target: x86_64-pc-cygwin Configured with: /cygdrive/i/szsz/tmpp/gcc/gcc-5.4.0-1.x86_64/src/gcc-5.4.0/configure --srcdir=/cygdrive/i/szsz/tmpp/gcc/gcc-5.4.0-1.x86_64/src/gcc-5.4.0 --prefix=/usr --exec-prefix=/usr --localstatedir=/var --sysconfdir=/etc --docdir=/usr/share/doc/gcc --htmldir=/usr/share/doc/gcc/html -C --build=x86_64-pc-cygwin --host=x86_64-pc-cygwin --target=x86_64-pc-cygwin --without-libiconv-prefix --without-libintl-prefix --libexecdir=/usr/lib --enable-shared --enable-shared-libgcc --enable-static --enable-version-specific-runtime-libs --enable-bootstrap --enable-__cxa_atexit --with-dwarf2 --with-tune=generic --enable-languages=ada,c,c++,fortran,lto,objc,obj-c++ --enable-graphite --enable-threads=posix --enable-libatomic --enable-libcilkrts --enable-libgomp --enable-libitm --enable-libquadmath --enable-libquadmath-support --enable-libssp --enable-libada --enable-libgcj-sublibs --disable-java-awt --disable-symvers --with-ecj-jar=/usr/share/java/ecj.jar --with-gnu-ld --with-gnu-as --with-cloog-include=/usr/include/cloog-isl --without-libiconv-prefix --without-libintl-prefix --with-system-zlib --enable-linker-build-id --with-default-libstdcxx-abi=gcc4-compatible Thread model: posix gcc version 5.4.0 (GCC) Running *make inside the ebloom directory* (Cygwin shell) gives: walte@REDACTED ~/ORIGINALE/ebloom $ make ==> Entering directory `d:/SoftDevelopment/Projects/ORIGINALE/ebloom' ==> ebloom (compile) Compiled src/ebloom.erl Compiling c_src/ebloom_nifs.cpp c_src/ebloom_nifs.cpp:1:0: warning: -fPIC ignored for target (all code is position independent) // ------------------------------------------------------------------- ^ 'cc' is not recognized as an internal or external command, operable program or batch file. ERROR: cmd /q /c cc c_src/ebloom_nifs.o -shared -Lc:/Software/erl7.3/lib/erl_interface-3.8.2/lib -lerl_interface -lei -o priv/ebloom_nifs.dll failed with error: 1 and output: 'cc' is not recognized as an internal or external command, operable program or batch file. make: *** [Makefile:6: compile] Error 1 *How can I fix it?* -------------- next part -------------- An HTML attachment was scrubbed... URL: From walter.weinmann@REDACTED Wed Dec 14 14:19:45 2016 From: walter.weinmann@REDACTED (Walter Weinmann) Date: Wed, 14 Dec 2016 14:19:45 +0100 Subject: [erlang-questions] Windows 10 x64 - basho - eleveldb - compilation environment Message-ID: I'm running Windows 10 x64 with Cygwin x64. *make* is. $ make -v GNU Make 4.2.1 Built for x86_64-unknown-cygwin *gcc *is: $ gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-cygwin/5.4.0/lto-wrapper.exe Target: x86_64-pc-cygwin Configured with: /cygdrive/i/szsz/tmpp/gcc/gcc-5.4.0-1.x86_64/src/gcc-5.4.0/configure --srcdir=/cygdrive/i/szsz/tmpp/gcc/gcc-5.4.0-1.x86_64/src/gcc-5.4.0 --prefix=/usr --exec-prefix=/usr --localstatedir=/var --sysconfdir=/etc --docdir=/usr/share/doc/gcc --htmldir=/usr/share/doc/gcc/html -C --build=x86_64-pc-cygwin --host=x86_64-pc-cygwin --target=x86_64-pc-cygwin --without-libiconv-prefix --without-libintl-prefix --libexecdir=/usr/lib --enable-shared --enable-shared-libgcc --enable-static --enable-version-specific-runtime-libs --enable-bootstrap --enable-__cxa_atexit --with-dwarf2 --with-tune=generic --enable-languages=ada,c,c++,fortran,lto,objc,obj-c++ --enable-graphite --enable-threads=posix --enable-libatomic --enable-libcilkrts --enable-libgomp --enable-libitm --enable-libquadmath --enable-libquadmath-support --enable-libssp --enable-libada --enable-libgcj-sublibs --disable-java-awt --disable-symvers --with-ecj-jar=/usr/share/java/ecj.jar --with-gnu-ld --with-gnu-as --with-cloog-include=/usr/include/cloog-isl --without-libiconv-prefix --without-libintl-prefix --with-system-zlib --enable-linker-build-id --with-default-libstdcxx-abi=gcc4-compatible Thread model: posix gcc version 5.4.0 (GCC) Running *make inside the eleveldb directory* (Cygwin shell) gives: walte@REDACTED ~/eleveldb_idea/eleveldb $ make ./rebar get-deps ==> eleveldb (get-deps) ./rebar compile ==> eleveldb (compile) Compiling d:/SoftDevelopment/Projects/eleveldb_idea/eleveldb/c_src/eleveldb.cc Microsoft (R) C/C++ Optimizing Compiler Version 16.00.40219.01 for x64 Copyright (C) Microsoft Corporation. All rights reserved. cl : Command line warning D9002 : ignoring unknown option '-O3' cl : Command line warning D9002 : ignoring unknown option '-fPIC' cl : Command line warning D9002 : ignoring unknown option '-O3' eleveldb.cc d:/SoftDevelopment/Projects/eleveldb_idea/eleveldb/c_src/eleveldb.cc(23) : fatal error C1083: Cannot open include file: 'syslog.h': No such file or directory ERROR: compile failed while processing D:/SoftDevelopment/Projects/eleveldb_idea/eleveldb: rebar_abort make: *** [Makefile:15: compile] Error 1 *How can I fix it?* -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.safarov@REDACTED Wed Dec 14 15:25:06 2016 From: s.safarov@REDACTED (Sergey Safarov) Date: Wed, 14 Dec 2016 14:25:06 +0000 Subject: [erlang-questions] why not works command from shell In-Reply-To: <20161214111814.GA87918@erix.ericsson.se> References: <20161213204749.GE7447@carfax.org.uk> <7215a879-4d5b-68f4-8636-c10a55c2fdfb@gmail.com> <20161214111814.GA87918@erix.ericsson.se> Message-ID: Thank you Raimo for your reference for pull request where issue solution is described. Sergey ??, 14 ???. 2016 ?. ? 14:18, Raimo Niskanen < raimo+erlang-questions@REDACTED>: > On Wed, Dec 14, 2016 at 10:32:25AM +0000, Sergey Safarov wrote: > > Thank you Kennett for your confirmation. > > I think like you and created ticket > https://bugs.erlang.org/browse/ERL-317 > > Already fixed in OTP-19.0. See https://github.com/erlang/otp/pull/949 > > > > > At present time I use 18.3.4.4 version from Fedora repo. > > > > ??, 14 ???. 2016, 13:20 Kenneth Lakin : > > > > > On 12/13/2016 01:07 PM, Sergey Safarov wrote: > > > > Correct resolv when called lookup_wait. > > > > > > Yeah, this old bug! > > > What version of Erlang are you using? I remember that I tracked this > > > down to some bad math somewhere in the bowels of the resolver > management > > > code that meant that you couldn't do lookups until the node has been up > > > for five seconds. > > > I _think_ it has been fixed in recent versions of Erlang, but I'm not > sure. > > > > > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From netch@REDACTED Wed Dec 14 15:57:06 2016 From: netch@REDACTED (Valentin Nechayev) Date: Wed, 14 Dec 2016 16:57:06 +0200 Subject: [erlang-questions] [18.3] Random node crash after many days running Message-ID: <20161214145706.GA17430@netch.kiev.ua> Hi, I have a few systems running a small web app under Cowboy. The webs are used everyday but nodes are not used to restarting for long times. After ~36 days of running, the nodes started crashing with messages like 1> erl_child_setup: Select failed: -1 (10) The value of 36 days is suspiciously similar to 3/4 * 2**32 milliseconds. Is it possible that some time counter overflows with whole node crash? Environment: Erlang 18.3 locally built with `kerl' on Centos 7.2/x86_64. -netch- From chandrashekhar.mullaparthi@REDACTED Wed Dec 14 23:10:33 2016 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Wed, 14 Dec 2016 22:10:33 +0000 Subject: [erlang-questions] Mnesia crashes and restarts the Erlang VM repeatedly In-Reply-To: <58512A9A.9030102@utl.in> References: <53894986-0aa9-bef6-2bf0-572c76602e04@utl.in> <58512A9A.9030102@utl.in> Message-ID: Hi Arun, You can start your erlang node with an explicit name such as scu_plus@REDACTED or scu_plus@REDACTED which will prevent it from crashing whenever the IP address changes. If you are using something like relx to build your release, then you should be able to specify the node name in the $ROOTDIR/releases//vm.args file. cheers, Chandru On 14 December 2016 at 11:18, Arun wrote: > Hi, > > I am facing some serious issues with mnesia database. We have an > erlang application using mnesia database as back-end. The application runs > on the hardware and the IP address of the hardware will be changing based > on its connectivity and so the erlang node name. Some times as soon as the > node changes am getting an error log as given below and the VM restarts > repeatedly. But if I delete the mnesia related files and restart again, > application runs smoothly. For me data reliability and persistence are the > prime concern so I can not go with the above mentioned method. Can somebody > kindly assist me how to resolve this issue with out deleting mnesia schema. > > Current working directory:/root/rel-3/galaxy/lib/galaxy-1.0.0/priv > Release number as string:3 > PHY: mdio@REDACTED:01 - Link is Up - 100/Full > 36524 01999.282 2001391.0 71.4 532699753591315.4 11871.3 0 > Exec: /usr/lib/erlang/erts-8.0/bin/erlexec -boot /root/rel-3/galaxy/releases/0.0.1/galaxy -mode embedded -boot_var ERTS_LIB_DIR /usr/lib/erlang/erts-8.0/../lib -config /root/rel-3/galaxy/releases/0.0.1/sys.config -args_file /root/rel-3/galaxy/vm.args -- console > Root: /root/rel-3/galaxy > /root/rel-3/galaxy > Erlang/OTP 19 [erts-8.0] [source] [async-threads:10] [kernel-poll:false] > > > =INFO REPORT==== 17-Nov-2016::12:42:59 === > disk_log: repairing "/root/mnesia_database/DECISION_TAB.LOG" ... > Mnesia('scu_plus@REDACTED'): Data may be missing, Corrupt logfile deleted: "/root/mnesia_database/LATEST.LOG", {not_a_log_file, > "/root/mnesia_database/LATEST.LOG"} > > =ERROR REPORT==== 17-Nov-2016::12:43:00 === > Mnesia('scu_plus@REDACTED'): ** ERROR ** (core dumped to file: "/root/rel-3/galaxy/MnesiaCore.scu_plus@REDACTED" ) > ** FATAL ** Failed to merge schema: Bad cookie in table definition configuration_table: 'scu_plus@REDACTED' = {cstruct,configuration_table,set,[],['scu_plus@REDACTED'],[],[],0,read_write,false,[],[],false,gen_record,[key,value,time_stamp],[],[],[],{{1479494951549325042,-134217623,1},'scu_plus@REDACTED'},{{2,0},[]}}, 'scu_plus@REDACTED' = {cstruct,configuration_table,set,[],['scu_plus@REDACTED'],[],[],0,read_write,false,[],[],false,gen_record,[key,value,time_stamp],[],[],[],{{1479575314787900797,-134217651,1},'scu_plus@REDACTED'},{{2,0},[ > ]}} > > > =ERROR REPORT==== 17-Nov-2016::12:43:10 === > ** Generic server mnesia_monitor terminating > ** Last message in was {'EXIT',<0.411.0>,killed} > ** When Server state == {state,<0.411.0>,[],[],true,[],undefined,[],[]} > ** Reason for termination == > ** killed > > =ERROR REPORT==== 17-Nov-2016::12:43:10 === > ** Generic server mnesia_recover terminating > ** Last message in was {'EXIT',<0.411.0>,killed} > ** When Server state == {state,<0.411.0>,undefined,undefined,undefined,0, > false,true,[]} > ** Reason for termination == > ** killed > > =ERROR REPORT==== 17-Nov-2016::12:43:10 === > ** Generic server mnesia_snmp_sup terminating > ** Last message in was {'EXIT',<0.411.0>,killed} > ** When Server state == {state, > {local,mnesia_snmp_sup}, > simple_one_for_one, > [{child,undefined,mnesia_snmp_sup, > {mnesia_snmp_hook,start,[]}, > transient,3000,worker, > [mnesia_snmp_sup,mnesia_snmp_hook, > supervisor]}], > undefined,0,86400000,[],0,mnesia_snmp_sup,[]} > ** Reason for termination == > ** killed > > =ERROR REPORT==== 17-Nov-2016::12:43:10 === > ** Generic server mnesia_subscr terminating > ** Last message in was {'EXIT',<0.411.0>,killed} > ** When Server state == {state,<0.411.0>,12307} > ** Reason for termination == > ** {{aborted,{node_not_running,'scu_plus@REDACTED'}}, > [{mnesia,abort,1,[{file,"mnesia.erl"},{line,318}]}, > {mnesia_lib,del,2,[{file,"mnesia_lib.erl"},{line,485}]}, > {mnesia_subscr,do_handle_exit,1,[{file,"mnesia_subscr.erl"},{line,508}]}, > {mnesia_subscr,handle_exit,2,[{file,"mnesia_subscr.erl"},{line,502}]}, > {mnesia_subscr,do_prepare_stop,2,[{file,"mnesia_subscr.erl"},{line,526}]}, > {mnesia_subscr,terminate,2,[{file,"mnesia_subscr.erl"},{line,304}]}, > {gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,629}]}, > {gen_server,terminate,7,[{file,"gen_server.erl"},{line,795}]}]} > > =ERROR REPORT==== 17-Nov-2016::12:43:10 === > Mnesia('scu_plus@REDACTED'): ** ERROR ** mnesia_event got unexpected event: {'EXIT', > <0.413.0>, > {aborted, > {node_not_running, > 'scu_plus@REDACTED'}}} > > =INFO REPORT==== 17-Nov-2016::12:43:10 === > application: mnesia > exited: {{shutdown,{failed_to_start_child,mnesia_kernel_sup,killed}}, > {mnesia_sup,start,[normal,[]]}} > type: permanent > {"Kernel pid terminated",application_controller,"{application_start_failure,mnesia,{{shutdown,{failed_to_start_child,mnesia_kernel_sup,killed}},{mnesia_sup,start,[normal,[]]}}}"} > > Crash dump is being written to: erl_crash.dump... > > > Thanks in advance, > Arun > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.safarov@REDACTED Thu Dec 15 09:28:03 2016 From: s.safarov@REDACTED (Sergey Safarov) Date: Thu, 15 Dec 2016 08:28:03 +0000 Subject: [erlang-questions] Shared query Message-ID: Could you please share example how to implement shared access to query? I wants store in query atoms of hostnames and then pool them from poolboy worker. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.fredheim@REDACTED Thu Dec 15 13:04:45 2016 From: eric.fredheim@REDACTED (IRLeif) Date: Thu, 15 Dec 2016 12:04:45 +0000 Subject: [erlang-questions] How to think and reason when designing process- and message-centric systems? Message-ID: Dear Erlang community, This is my first email to the mailing list. I apologize in advance if this is odd or off-topic. Coming from an object-oriented and data-centric background, I have cognitive difficulties when it comes to conceptualizing, thinking about and designing systems consisting of modules, processes and key-value data stores. My brain reverts to thinking about classes, objects, inheritance trees, encapsulation and SQL-style relational data models. I'm afraid this could lead to unidiomatic Erlang system architectures and implementations, which would be undesirable. Here are some of the essential complexities I have difficulties grasping: A) Identifying discrete modules and processes and finding good names for them. B) Appointing supervisor and worker modules; defining process hierarchies. C) Deciding which processes should communicate with each other and how. D) Designing a sensible persistent data model with Mnesia or other NoSQL data models (e.g. using CouchDB). E) Deciding which processes should read and write persistent data records. F) Incorporating global modules/"shared facilities" like event handlers, loggers, etc. G) Visualizing the system architecture, processes and communication lines; what kind of graphics to use. H) Organizing source code files into separate projects and directory structures. Questions: 1) How do you unlearn "bad habits" from object-oriented way of thinking? 2) How do you think and reason about process-centric systems designs? 3) When designing a new system, how do you approach the above activities? I would appreciate any practical tips, examples, "mind hacks" and resources that might help. Kind regards, Leif Eric Fredheim -------------- next part -------------- An HTML attachment was scrubbed... URL: From Oliver.Korpilla@REDACTED Thu Dec 15 15:02:16 2016 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Thu, 15 Dec 2016 15:02:16 +0100 Subject: [erlang-questions] How to think and reason when designing process- and message-centric systems? In-Reply-To: References: Message-ID: Hello, Leif. My biggest hurdle was understanding how an OTP callback module works and I resolved that for myself by implementing a behavior along similar lines to understand it. Somehow my brain did not click when reading Erlang books about it. I never much worried about B) and E) as Erlang is so flexible when it comes to rearranging things later on. Similarly, H) evolved over time. C) is message passing in most cases. I occasionally employ calls but I'm relying mostly on message interfaces. Regarding F)... I still have found no use case for an event handler in my project. And once I understood (with the help of the mailing lists and forums) how the loggers work it was not hard. It was an interesting last year. Wouldn't miss it in the world. :) Best regards, Oliver ? ? Gesendet:?Donnerstag, 15. Dezember 2016 um 13:04 Uhr Von:?IRLeif An:?erlang-questions@REDACTED Betreff:?[erlang-questions] How to think and reason when designing process- and message-centric systems? Dear Erlang community, ? This is my first email to the mailing list. I apologize in advance if this is odd or off-topic. ? Coming from an object-oriented and data-centric background, I have cognitive difficulties when it comes to conceptualizing, thinking about and designing systems consisting of modules, processes and key-value data stores. ? My brain reverts to thinking about classes, objects, inheritance trees, encapsulation and SQL-style relational data models. I'm afraid this could lead to unidiomatic Erlang system architectures and implementations, which would be undesirable. ? Here are some of the essential complexities I have difficulties grasping: ? A) Identifying discrete modules and processes and finding good names for them. B) Appointing supervisor and worker modules; defining process hierarchies. C) Deciding which processes should communicate with each other and how. D) Designing a sensible persistent data model with Mnesia or other NoSQL data models (e.g. using CouchDB). E) Deciding which processes should read and write persistent data records. F) Incorporating global modules/"shared facilities" like event handlers, loggers, etc. G) Visualizing the system architecture, processes and communication lines; what kind of graphics to use. H) Organizing source code files into separate projects and directory structures. ? Questions: 1) How do you unlearn "bad habits" from object-oriented way of thinking? 2) How do you think and reason about process-centric systems designs? 3) When designing a new system, how do you approach the above activities? ? I would appreciate any practical tips, examples, "mind hacks" and resources that might help. ? Kind regards, ? Leif Eric Fredheim_______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From alex0player@REDACTED Thu Dec 15 15:13:05 2016 From: alex0player@REDACTED (Alex S.) Date: Thu, 15 Dec 2016 17:13:05 +0300 Subject: [erlang-questions] How to think and reason when designing process- and message-centric systems? In-Reply-To: References: Message-ID: Hello, what helps me personally when designing an Erlang system (as with any modular system, probably?..) is thinking in terms of responsibilities. E.g. each module is a provider of a certain service to everything else, whether it be a computational service (library module), interfacing service (db driver module, for example, or message codec), and/or stateful service (gen_server of some sorts). I?ve yet to write an event handler although I?ve certainly used an excellent one that ships with lager ;). One thing that helps mapping OO approach onto Erlang is thinking one module=one class (as in, a set of operations a state/actor responds to), and instances are created and passed around explicitly, be they server PIDs or other kinds of state. Do abstain, however, from creating ?objects? which have no behaviour (so-called dumb objects), as you can just use data types for that; and from creating a multitude of ?objects?, as you?re then better off thinking of a whole collection as an object. The process boundaries should generally be drawn along the lines of ?What parts of my system wouldn?t notice if the other parts would die and be seamlessly reborn??, although of course simplicity is also to keep in mind, and rest_for_one and one_for_all supervisor strategy is there for you to introduce explicit dependency between parts of the whole that are split between processes. Hope that helps! ^_^ > 15 ???. 2016 ?., ? 15:04, IRLeif ???????(?): > > Dear Erlang community, > > This is my first email to the mailing list. I apologize in advance if this is odd or off-topic. > > Coming from an object-oriented and data-centric background, I have cognitive difficulties when it comes to conceptualizing, thinking about and designing systems consisting of modules, processes and key-value data stores. > > My brain reverts to thinking about classes, objects, inheritance trees, encapsulation and SQL-style relational data models. I'm afraid this could lead to unidiomatic Erlang system architectures and implementations, which would be undesirable. > > Here are some of the essential complexities I have difficulties grasping: > > A) Identifying discrete modules and processes and finding good names for them. > B) Appointing supervisor and worker modules; defining process hierarchies. > C) Deciding which processes should communicate with each other and how. > D) Designing a sensible persistent data model with Mnesia or other NoSQL data models (e.g. using CouchDB). > E) Deciding which processes should read and write persistent data records. > F) Incorporating global modules/"shared facilities" like event handlers, loggers, etc. > G) Visualizing the system architecture, processes and communication lines; what kind of graphics to use. > H) Organizing source code files into separate projects and directory structures. > > Questions: > > 1) How do you unlearn "bad habits" from object-oriented way of thinking? > 2) How do you think and reason about process-centric systems designs? > 3) When designing a new system, how do you approach the above activities? > > I would appreciate any practical tips, examples, "mind hacks" and resources that might help. > > Kind regards, > > Leif Eric Fredheim > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mfidelman@REDACTED Thu Dec 15 15:44:56 2016 From: mfidelman@REDACTED (Miles Fidelman) Date: Thu, 15 Dec 2016 09:44:56 -0500 Subject: [erlang-questions] How to think and reason when designing process- and message-centric systems? In-Reply-To: References: Message-ID: It might help to read "Erlang & OTP in Action" and work through the exercises. It provides a pretty thorough, step-by-step view of building a meaningful application in Erlang. On 12/15/16 7:04 AM, IRLeif wrote: > Dear Erlang community, > > This is my first email to the mailing list. I apologize in advance if > this is odd or off-topic. > > Coming from an object-oriented and data-centric background, I have > cognitive difficulties when it comes to conceptualizing, thinking > about and designing systems consisting of modules, processes and > key-value data stores. > > My brain reverts to thinking about classes, objects, inheritance > trees, encapsulation and SQL-style relational data models. I'm afraid > this could lead to unidiomatic Erlang system architectures and > implementations, which would be undesirable. > > Here are some of the essential complexities I have difficulties grasping: > > A) Identifying discrete modules and processes and finding good names > for them. > B) Appointing supervisor and worker modules; defining process hierarchies. > C) Deciding which processes should communicate with each other and how. > D) Designing a sensible persistent data model with Mnesia or other > NoSQL data models (e.g. using CouchDB). > E) Deciding which processes should read and write persistent data records. > F) Incorporating global modules/"shared facilities" like event > handlers, loggers, etc. > G) Visualizing the system architecture, processes and communication > lines; what kind of graphics to use. > H) Organizing source code files into separate projects and directory > structures. > > Questions: > > 1) How do you unlearn "bad habits" from object-oriented way of thinking? > 2) How do you think and reason about process-centric systems designs? > 3) When designing a new system, how do you approach the above activities? > > I would appreciate any practical tips, examples, "mind hacks" and > resources that might help. > > Kind regards, > > Leif Eric Fredheim > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Dec 15 16:32:52 2016 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Thu, 15 Dec 2016 16:32:52 +0100 Subject: [erlang-questions] How to think and reason when designing process- and message-centric systems? In-Reply-To: References: Message-ID: <23b8e572-ff8c-880d-dfa0-8e2535a405b1@ninenines.eu> Welcome, On 12/15/2016 01:04 PM, IRLeif wrote: > Coming from an object-oriented and data-centric background, I have > cognitive difficulties when it comes to conceptualizing, thinking about > and designing systems consisting of modules, processes and key-value > data stores. > > My brain reverts to thinking about classes, objects, inheritance trees, > encapsulation and SQL-style relational data models. I'm afraid this > could lead to unidiomatic Erlang system architectures and > implementations, which would be undesirable. That's what brains do, unfortunately. Let me stop you there by correcting one small thing: SQL-style relational data models can still apply depending on the system you are building. PostgreSQL usage is not uncommon in the Erlang world, at least for small to medium systems. Classes, objects, and so on, on the other hand, do not exist, and thinking in terms of objects will have a negative impact on the design of your system. How do you solve this? Well it depends on your experience outside of OO. Here are a few early tips, apply the relevant ones: - Have you used a non-OO imperative language? C for example. In these languages, you only have functions. In Erlang, you only have functions. Erlang is closer to C than OO, so switch your brain to C-style programming instead of OO. If necessary, spend a few hours writing a short C program (or another language as long as you don't use/write OO code) and then switch to Erlang. This will set your mind on the right path. - Have you ever heard of the best practice that dictates that variables should not be reused? Erlang basically enforces that. When writing that short program, make sure you do not reuse variables. - Have you ever written code that recurses over directories to for example file files matching a pattern? You can't do that with only loops, regardless of the language. Have the short program use recursion instead of loops. After doing this your mind will be very close to how it needs to be to write sequential Erlang programs. The rest is minor syntax differences like commas and semicolons or uppercase first letter for variable names. 95% of the Erlang code you write is short sequential programs. If you have experience with microservices, then think of Erlang processes as microservices. If you have experience with client/server development, then think of them all as servers that communicate to each other in a mostly client/server fashion. The only difference is that they communicate using Erlang messages. It definitely helps to reimplement something like a gen_server or gen_statem as you will better grasp links, monitors and timeouts. These are useful to have solid systems; although you can do mostly without and still get results. > Here are some of the essential complexities I have difficulties grasping: > > A) Identifying discrete modules and processes and finding good names for > them. There's basically two kinds of modules in Erlang: those implementing a process, and those containing useful functions. For the latter it should be obvious: group all the related functions together. For the former, you are almost required to put them in a module as you implement behaviours, so no big trouble there. As for naming, well, good luck with that. I generally follow this pattern: _[_] where is a namespace I put in all modules of my application (often the name of the application), is the descriptive name (for example 'router' or 'http') and is the type of the module, for example 'server', 'sup' for supervisor, 'h' for Cowboy handler, 't' for a module containing functions for converting from/to and manipulating a type, and so on. > D) Designing a sensible persistent data model with Mnesia or other NoSQL > data models (e.g. using CouchDB). Depending on what you are doing you might not need those. And if you do, I would advise not jumping into it early. Start with what you know. Erlang is a big enough change as it is. > E) Deciding which processes should read and write persistent data records. Start simple, measure, then decide what to do about it. Pools of connections are not uncommon, and already written for you in most cases. > F) Incorporating global modules/"shared facilities" like event handlers, > loggers, etc. Don't know what you mean with event handlers exactly, but as far as logging goes, use either error_logger or lager depending on your needs. Again not much to think about there. > H) Organizing source code files into separate projects and directory > structures. A few simple rules: - A module covers one small topic - A process does one thing - An application covers one topic and does one thing that involves multiple modules and processes > B) Appointing supervisor and worker modules; defining process hierarchies. > C) Deciding which processes should communicate with each other and how. > G) Visualizing the system architecture, processes and communication > lines; what kind of graphics to use. I'll leave that one for others. > Questions: > > 1) How do you unlearn "bad habits" from object-oriented way of thinking? The only way to unlearn something is to not use it for extended periods of time. I'm afraid that's not much help. Instead, try manipulating your brain into finding the right state of mind when you work on Erlang projects. It's the same problem you have when you play tennis and then switch to ping pong. The only difference is that you don't play them in the same environment or with the same tools so it's easier to switch from one to the other. Actually changing environment could help you into switching your OO brain off: change desk, change editor, and so on. > 2) How do you think and reason about process-centric systems designs? > 3) When designing a new system, how do you approach the above activities? Start small and naive, measure, then take decisions as to how the processes should be organized. Erlang is very easy to refactor compared to OO, so take advantage of that. Go for a working prototype, then improve upon it. How long you take to go from prototype to production will mostly depend on your experience. I've been interrupted about a thousand times when writing this, so hopefully I make some sort of sense. :-) Cheers, -- Lo?c Hoguin https://ninenines.eu From s.safarov@REDACTED Thu Dec 15 17:56:07 2016 From: s.safarov@REDACTED (Sergey Safarov) Date: Thu, 15 Dec 2016 16:56:07 +0000 Subject: [erlang-questions] How to think and reason when designing process- and message-centric systems? In-Reply-To: <23b8e572-ff8c-880d-dfa0-8e2535a405b1@ninenines.eu> References: <23b8e572-ff8c-880d-dfa0-8e2535a405b1@ninenines.eu> Message-ID: As guide for first step in erlang I use http://learnyousomeerlang.com/contents Think it will help you. ??, 15 ???. 2016, 18:33 Lo?c Hoguin : > Welcome, > > On 12/15/2016 01:04 PM, IRLeif wrote: > > Coming from an object-oriented and data-centric background, I have > > cognitive difficulties when it comes to conceptualizing, thinking about > > and designing systems consisting of modules, processes and key-value > > data stores. > > > > My brain reverts to thinking about classes, objects, inheritance trees, > > encapsulation and SQL-style relational data models. I'm afraid this > > could lead to unidiomatic Erlang system architectures and > > implementations, which would be undesirable. > > That's what brains do, unfortunately. > > Let me stop you there by correcting one small thing: SQL-style > relational data models can still apply depending on the system you are > building. PostgreSQL usage is not uncommon in the Erlang world, at least > for small to medium systems. > > Classes, objects, and so on, on the other hand, do not exist, and > thinking in terms of objects will have a negative impact on the design > of your system. > > How do you solve this? Well it depends on your experience outside of OO. > Here are a few early tips, apply the relevant ones: > > - Have you used a non-OO imperative language? C for example. In these > languages, you only have functions. In Erlang, you only have functions. > Erlang is closer to C than OO, so switch your brain to C-style > programming instead of OO. > > If necessary, spend a few hours writing a short C program (or another > language as long as you don't use/write OO code) and then switch to > Erlang. This will set your mind on the right path. > > - Have you ever heard of the best practice that dictates that variables > should not be reused? Erlang basically enforces that. > > When writing that short program, make sure you do not reuse variables. > > - Have you ever written code that recurses over directories to for > example file files matching a pattern? You can't do that with only > loops, regardless of the language. > > Have the short program use recursion instead of loops. > > After doing this your mind will be very close to how it needs to be to > write sequential Erlang programs. The rest is minor syntax differences > like commas and semicolons or uppercase first letter for variable names. > > 95% of the Erlang code you write is short sequential programs. If you > have experience with microservices, then think of Erlang processes as > microservices. If you have experience with client/server development, > then think of them all as servers that communicate to each other in a > mostly client/server fashion. The only difference is that they > communicate using Erlang messages. > > It definitely helps to reimplement something like a gen_server or > gen_statem as you will better grasp links, monitors and timeouts. These > are useful to have solid systems; although you can do mostly without and > still get results. > > > Here are some of the essential complexities I have difficulties grasping: > > > > A) Identifying discrete modules and processes and finding good names for > > them. > > There's basically two kinds of modules in Erlang: those implementing a > process, and those containing useful functions. For the latter it should > be obvious: group all the related functions together. For the former, > you are almost required to put them in a module as you implement > behaviours, so no big trouble there. > > As for naming, well, good luck with that. > > I generally follow this pattern: _[_] where is a > namespace I put in all modules of my application (often the name of the > application), is the descriptive name (for example 'router' or > 'http') and is the type of the module, for example 'server', > 'sup' for supervisor, 'h' for Cowboy handler, 't' for a module > containing functions for converting from/to and manipulating a type, and > so on. > > > D) Designing a sensible persistent data model with Mnesia or other NoSQL > > data models (e.g. using CouchDB). > > Depending on what you are doing you might not need those. And if you do, > I would advise not jumping into it early. Start with what you know. > Erlang is a big enough change as it is. > > > E) Deciding which processes should read and write persistent data > records. > > Start simple, measure, then decide what to do about it. Pools of > connections are not uncommon, and already written for you in most cases. > > > F) Incorporating global modules/"shared facilities" like event handlers, > > loggers, etc. > > Don't know what you mean with event handlers exactly, but as far as > logging goes, use either error_logger or lager depending on your needs. > Again not much to think about there. > > > H) Organizing source code files into separate projects and directory > > structures. > > A few simple rules: > > - A module covers one small topic > - A process does one thing > - An application covers one topic and does one thing that involves > multiple modules and processes > > > B) Appointing supervisor and worker modules; defining process > hierarchies. > > C) Deciding which processes should communicate with each other and how. > > G) Visualizing the system architecture, processes and communication > > lines; what kind of graphics to use. > > I'll leave that one for others. > > > Questions: > > > > 1) How do you unlearn "bad habits" from object-oriented way of thinking? > > The only way to unlearn something is to not use it for extended periods > of time. I'm afraid that's not much help. Instead, try manipulating your > brain into finding the right state of mind when you work on Erlang > projects. > > It's the same problem you have when you play tennis and then switch to > ping pong. The only difference is that you don't play them in the same > environment or with the same tools so it's easier to switch from one to > the other. > > Actually changing environment could help you into switching your OO > brain off: change desk, change editor, and so on. > > > 2) How do you think and reason about process-centric systems designs? > > 3) When designing a new system, how do you approach the above activities? > > Start small and naive, measure, then take decisions as to how the > processes should be organized. Erlang is very easy to refactor compared > to OO, so take advantage of that. Go for a working prototype, then > improve upon it. How long you take to go from prototype to production > will mostly depend on your experience. > > I've been interrupted about a thousand times when writing this, so > hopefully I make some sort of sense. :-) > > Cheers, > > -- > Lo?c Hoguin > https://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dszoboszlay@REDACTED Thu Dec 15 22:22:14 2016 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Thu, 15 Dec 2016 21:22:14 +0000 Subject: [erlang-questions] How to think and reason when designing process- and message-centric systems? In-Reply-To: References: <23b8e572-ff8c-880d-dfa0-8e2535a405b1@ninenines.eu> Message-ID: Hi, There's been a lot of good advice in this thread so far. I would like to throw in my two cents regarding designing and thinking in process structures. Erlang is known as a highly concurrent language, every tutorial talks a lot about processes and so on; so it's no surprise working with processes sounds very exciting (and maybe even scary sometimes) for newcomers. But I believe in most cases you should ignore them as long as you can. A typical server application would serve incoming requests. Signing up a new user for example. Your system will be concurrent by simply allowing each request to be served in parallel. So there will be one process per request. This is something you would get typically out of the box from your Erlang web server - no need to think much about processes so far. Your job would be to code how to handle a request. How to sign up a new user for example. This is often just simple sequential code: add the user to the DB, send a welcome email, whatever. You don't need to throw in processes just because you're working in Erlang. It's not like programming a GPU, where you have to think hard how to allow all those tiny processors crunch through your initially very sequential looking calculation. If your algorithm is best expressed as sequential code, just write sequential code. Your server will be fast not because it could use all CPU cores to serve that single sign up request. It will be fast because it will be able to serve different sign up requests on all those cores. As you go on with your sequential code, you will sometimes encounter problems where the need for new processes arise naturally. Then you can add these processes, and it will all feel very obvious and easy. Some typical situations that call for processes: - You have some stateful component that needs atomic updates. A process can hold the state and its message queue will serialise the update requests. - You have to perform some task as you serve the request, but it doesn't have to complete before sending a response to the user. Just delegate this asynchronous task to a different process. Typical example: logging. - You need to do 2 or more tasks that don't depend on each other and can take a long time. Just spawn a process for each task and let them run in parallel. - You have a resource like an ETS table or port that has to be around for a long time. You will need a long living process to own this resource. - You need to do some heavy (CPU or memory wise) computation. Like resizing the avatar the new user has uploaded. You are afraid that if too many concurrent sign ups would happen at the same time you would overload the system. This is a point where you concurrent processes need to synchronise with each other (maybe via an other process or just an ETS table) to limit the concurrency in your system. (Yes, this is kind of the opposite of the above points...) And there are of course advanced situations when you can (or even need to) code process-aware. There are nice tricks like offloading work from processes with a possibly long message queue to a different process with guaranteed no messages, spawning new processes in a library to protect the global state of the process calling into the library and so on. But in the beginning just concentrate on the simple cases. Once you start using more and more processes, you will have to answer the question of what to do if one of them dies? Shall asynchronous tasks still be carried on if the initial request crashed? You will quickly learn a lot about links and supervisors when you get to this point. So my advise is to start writing the sequential part of the code, and see what other processes emerge naturally. If you want to first design the process structure and supervisor tree of your application, than you risk implementing a lot of functionality that could be plain sequential library code as gen_servers. Which would only introduce unnecessary bottlenecks into your system. Cheers, Daniel On Thu, 15 Dec 2016 at 17:56 Sergey Safarov wrote: > As guide for first step in erlang I use > http://learnyousomeerlang.com/contents > > Think it will help you. > > ??, 15 ???. 2016, 18:33 Lo?c Hoguin : > > Welcome, > > On 12/15/2016 01:04 PM, IRLeif wrote: > > Coming from an object-oriented and data-centric background, I have > > cognitive difficulties when it comes to conceptualizing, thinking about > > and designing systems consisting of modules, processes and key-value > > data stores. > > > > My brain reverts to thinking about classes, objects, inheritance trees, > > encapsulation and SQL-style relational data models. I'm afraid this > > could lead to unidiomatic Erlang system architectures and > > implementations, which would be undesirable. > > That's what brains do, unfortunately. > > Let me stop you there by correcting one small thing: SQL-style > relational data models can still apply depending on the system you are > building. PostgreSQL usage is not uncommon in the Erlang world, at least > for small to medium systems. > > Classes, objects, and so on, on the other hand, do not exist, and > thinking in terms of objects will have a negative impact on the design > of your system. > > How do you solve this? Well it depends on your experience outside of OO. > Here are a few early tips, apply the relevant ones: > > - Have you used a non-OO imperative language? C for example. In these > languages, you only have functions. In Erlang, you only have functions. > Erlang is closer to C than OO, so switch your brain to C-style > programming instead of OO. > > If necessary, spend a few hours writing a short C program (or another > language as long as you don't use/write OO code) and then switch to > Erlang. This will set your mind on the right path. > > - Have you ever heard of the best practice that dictates that variables > should not be reused? Erlang basically enforces that. > > When writing that short program, make sure you do not reuse variables. > > - Have you ever written code that recurses over directories to for > example file files matching a pattern? You can't do that with only > loops, regardless of the language. > > Have the short program use recursion instead of loops. > > After doing this your mind will be very close to how it needs to be to > write sequential Erlang programs. The rest is minor syntax differences > like commas and semicolons or uppercase first letter for variable names. > > 95% of the Erlang code you write is short sequential programs. If you > have experience with microservices, then think of Erlang processes as > microservices. If you have experience with client/server development, > then think of them all as servers that communicate to each other in a > mostly client/server fashion. The only difference is that they > communicate using Erlang messages. > > It definitely helps to reimplement something like a gen_server or > gen_statem as you will better grasp links, monitors and timeouts. These > are useful to have solid systems; although you can do mostly without and > still get results. > > > Here are some of the essential complexities I have difficulties grasping: > > > > A) Identifying discrete modules and processes and finding good names for > > them. > > There's basically two kinds of modules in Erlang: those implementing a > process, and those containing useful functions. For the latter it should > be obvious: group all the related functions together. For the former, > you are almost required to put them in a module as you implement > behaviours, so no big trouble there. > > As for naming, well, good luck with that. > > I generally follow this pattern: _[_] where is a > namespace I put in all modules of my application (often the name of the > application), is the descriptive name (for example 'router' or > 'http') and is the type of the module, for example 'server', > 'sup' for supervisor, 'h' for Cowboy handler, 't' for a module > containing functions for converting from/to and manipulating a type, and > so on. > > > D) Designing a sensible persistent data model with Mnesia or other NoSQL > > data models (e.g. using CouchDB). > > Depending on what you are doing you might not need those. And if you do, > I would advise not jumping into it early. Start with what you know. > Erlang is a big enough change as it is. > > > E) Deciding which processes should read and write persistent data > records. > > Start simple, measure, then decide what to do about it. Pools of > connections are not uncommon, and already written for you in most cases. > > > F) Incorporating global modules/"shared facilities" like event handlers, > > loggers, etc. > > Don't know what you mean with event handlers exactly, but as far as > logging goes, use either error_logger or lager depending on your needs. > Again not much to think about there. > > > H) Organizing source code files into separate projects and directory > > structures. > > A few simple rules: > > - A module covers one small topic > - A process does one thing > - An application covers one topic and does one thing that involves > multiple modules and processes > > > B) Appointing supervisor and worker modules; defining process > hierarchies. > > C) Deciding which processes should communicate with each other and how. > > G) Visualizing the system architecture, processes and communication > > lines; what kind of graphics to use. > > I'll leave that one for others. > > > Questions: > > > > 1) How do you unlearn "bad habits" from object-oriented way of thinking? > > The only way to unlearn something is to not use it for extended periods > of time. I'm afraid that's not much help. Instead, try manipulating your > brain into finding the right state of mind when you work on Erlang > projects. > > It's the same problem you have when you play tennis and then switch to > ping pong. The only difference is that you don't play them in the same > environment or with the same tools so it's easier to switch from one to > the other. > > Actually changing environment could help you into switching your OO > brain off: change desk, change editor, and so on. > > > 2) How do you think and reason about process-centric systems designs? > > 3) When designing a new system, how do you approach the above activities? > > Start small and naive, measure, then take decisions as to how the > processes should be organized. Erlang is very easy to refactor compared > to OO, so take advantage of that. Go for a working prototype, then > improve upon it. How long you take to go from prototype to production > will mostly depend on your experience. > > I've been interrupted about a thousand times when writing this, so > hopefully I make some sort of sense. :-) > > Cheers, > > -- > Lo?c Hoguin > https://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From list1@REDACTED Fri Dec 16 00:35:23 2016 From: list1@REDACTED (Grzegorz Junka) Date: Thu, 15 Dec 2016 23:35:23 +0000 Subject: [erlang-questions] How to think and reason when designing process- and message-centric systems? In-Reply-To: References: Message-ID: <1c7d8286-a354-2588-968a-dddb907144c3@gjunka.com> On 15/12/2016 12:04, IRLeif wrote: > Dear Erlang community, > > This is my first email to the mailing list. I apologize in advance if > this is odd or off-topic. > > Coming from an object-oriented and data-centric background, I have > cognitive difficulties when it comes to conceptualizing, thinking > about and designing systems consisting of modules, processes and > key-value data stores. > > My brain reverts to thinking about classes, objects, inheritance > trees, encapsulation and SQL-style relational data models. I'm afraid > this could lead to unidiomatic Erlang system architectures and > implementations, which would be undesirable. > > Here are some of the essential complexities I have difficulties grasping: > > A) Identifying discrete modules and processes and finding good names > for them. > B) Appointing supervisor and worker modules; defining process hierarchies. > C) Deciding which processes should communicate with each other and how. > D) Designing a sensible persistent data model with Mnesia or other > NoSQL data models (e.g. using CouchDB). > E) Deciding which processes should read and write persistent data records. > F) Incorporating global modules/"shared facilities" like event > handlers, loggers, etc. > G) Visualizing the system architecture, processes and communication > lines; what kind of graphics to use. > H) Organizing source code files into separate projects and directory > structures. > > Questions: > > 1) How do you unlearn "bad habits" from object-oriented way of thinking? > 2) How do you think and reason about process-centric systems designs? > 3) When designing a new system, how do you approach the above activities? > Well, for me Erlang is like a better version of OOP, it's how actually classes should have been implemented from the beginning. You simply have to mentally match actors with classes and method invocation with message passing. An actor is like a class that creates "living" entities instead of pieces of data. Having the definition of an actor (in an Erlang module), you can create multiple actors sharing the same code. Each actor maintains its own state exposing outside an API through which other actors can read or update the actor's state. In OOP an instance of a class is simply some piece of memory to keep the state - instance variables. If the responsibility for the state wasn't delegated to class or instance methods, every method could change the state in a haphazard way. You need to synchronize which methods update which variables, add setters and getters, restrict the variables with private/protected access modifiers, only to protect the state from becoming undefined or corrupted. Actor does the same but better. Whereas you can find a way to update an instance without invoking any methods (e.g. using pointers) it's not possible to update a state of an actor if it doesn't expose an API - Erlang doesn't provide any semantic to do that (you can only kill an actor). But the main difference is that whereas instance methods run sequentially in the order in which they are called, actors actually run in parallel. If you have ever tried to implement any concurrent computation in OOP you will know that those two don't get together very well. If you have 10 processors and 100 instances of classes you can't distribute the instances across the processors/servers to share the load. That even doesn't make sense. You would need to use some message passing library, e.g. MPI, to distribute such code across processors. However, in Erlang the distribution is done by the VM (BEAM) where schedulers can transparently exchange messages send between actors no matter on which processor those actors are running. They can even transparently distribute the actors across multiple nodes connected with network interfaces. In Erlang an actor is implemented using behaviours, e.g. gen_server , gen_fsm . During execution an Erlang application can have as many running processes/actors as an OOP application can have created instances of classes. You could start by trying to: - map global state (global variables, data held in static classes/istances used by the whole application) to ets or mnesia . - map singletons and classes that maintain state or provide access to resources to actors/processes. - map classes that only provide a collection of functions to Erlang modules. Many books discussing OOP try to map the world to classes and instances (e.g. a class 'washing machine' that has methods 'wash', 'dry', etc). For me it's easier to map the surrounding world to actors rather than classes because objects in reality don't wait for each other. They exist, respond to actions and can change in real time independently of each other - exactly like actors. I hope this helps Greg -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkbucc@REDACTED Fri Dec 16 02:39:53 2016 From: mkbucc@REDACTED (Mark Bucciarelli) Date: Thu, 15 Dec 2016 20:39:53 -0500 Subject: [erlang-questions] How to think and reason when designing process- and message-centric systems? In-Reply-To: References: Message-ID: <20161216013953.GA786@Marks-MBP-3> On Thu, Dec 15, 2016 at 12:04:45PM +0000, IRLeif wrote: > > This is my first email to the mailing list. I apologize in advance if this > is odd or off-topic. > It has generated some good discussion, thanks for kicking it off! I have been teaching myself Erlang since June and lurking on this list since then and have always found it to be welcoming and helpful. I suspect no one found your message off-topic. > > Coming from an object-oriented and data-centric background, I have > cognitive difficulties when it comes to conceptualizing, thinking about and > designing systems consisting of modules, processes and key-value data > stores. > Before Erlang, that was my background and how I still make a living. > ... > Questions: > > 1) How do you unlearn "bad habits" from object-oriented way of thinking? > I am not sure what you mean by "bad habits", but will hazard a guess: shared, mutable state and or deep stacks of inheritance and layers of abstraction. Erlang prohibits you from these "bad" behaviors by design. For example, once you assign a value to a variable it is bound to that value. I like to think of it like a math proof, where you say "Let x = 1. In this case, ..." In a proof, you would never then say that x changes it's value ... it is immutable state. It is interesting to read the original rational for OOP in Meyers orange book: that nouns change less frequenly in a system than the verbs. Since most software work is maintence, to minimize maintenance you should structure your system around the stable things: which Meyers hypothesize was the nouns. I think that approach has been replaced and that we are at the start of an industry shift towards a focus on verbs with things like event sourcing and CQRS. If you want to build a system that focusses on the messages and the contents of them, Erlang is a perfect fit. > > 2) How do you think and reason about process-centric systems designs? > Lot's of great responses on this topic already. For me, the simplest thing is to think of the verbs (messages) in your system instead of the nouns. > > I would appreciate any practical tips, examples, "mind hacks" and resources > that might help. > Here are a few that I have found particularly useful. "The Road to the Generic Server" Chapter in Joe Armstrong's Programming Erlang. * Builds up slowly the core idea behind OTP behaviors in a accessible way. (I highly recommend this book in general, but that chapter is gold.) "The Universal Server" (http://joearms.github.io/2013/11/21/My-favorite-erlang-program.html) * Such. an. awesome. hack. I smile each time I think of it. "Error Handling" (http://erlang.org/course/error_handling.html) * Concisely presents the primitives that underly any robust process tree. Planet Erlang (http://www.planeterlang.com) * Aggregator of Erlang blogs. Unfortunately, it is heavily diluted by Elixer content. Blogging about Erlang has actually helped me a lot. To start to write something for publication pretty quickly tells you how well you really understand it. :) Enjoy, Mark -- Blogging at markbucciarelli.com Tweeting @mbucc P.S. My next blog post (this Wednesday) will be a second post on event handling with gen_event. From zxq9@REDACTED Fri Dec 16 02:58:52 2016 From: zxq9@REDACTED (zxq9) Date: Fri, 16 Dec 2016 10:58:52 +0900 Subject: [erlang-questions] How to think and reason when designing process- and message-centric systems? In-Reply-To: References: Message-ID: <2403943.2jbFVTaLeE@changa> Off-list reply due to the incomplete nature of the reference below. On 2016?12?15? ??? 12:04:45 IRLeif wrote: > Dear Erlang community, > > This is my first email to the mailing list. I apologize in advance if this > is odd or off-topic. > > Coming from an object-oriented and data-centric background, I have > cognitive difficulties when it comes to conceptualizing, thinking about and > designing systems consisting of modules, processes and key-value data > stores. > > My brain reverts to thinking about classes, objects, inheritance trees, > encapsulation and SQL-style relational data models. I'm afraid this could > lead to unidiomatic Erlang system architectures and implementations, which > would be undesirable. I have been working on a book for intermediate Erlang that starts out with a conceptual architecture design section before getting into code. This section is already written and may provide some "how to think about X" type things in Erlang. http://zxq9.com/erlmud/ Beware of the code in the v0.1 section. It is deliberately written in an almost anti-OTP style. I simply have not had the time to cover how to factor out the common bits there, then expose how those common bits are already in OTP, and so on. My work on this has been (mostly) stalled for about a year. :-/ https://github.com/zxq9/erlmud If you want an example of a small-enough-to-digest but OTP-based project, there is a UUID generator that can require a process be started to track the current clock value in the case of very high-volume UUIDv1 generation. It is documented and written in a style that was developed on the list in collaboration with all the graybeards I could find: https://github.com/zxq9/zuuid The intent of the UUID project was to present a small example exactly for cases such as this. If you have time please check them out. Hopefully they can, in conjunction with other resources such as "Learn You Some Erlang", get you up to speed without too many hurdles. If you find them useful, feel free to pass them along. I wrote a post on SO a while back about the difference between Java threads and Erlang processes. The underlying point is that the differences between Java and Erlang go way beyond (and are entirely orthogonal to) the difference between OOP and FP. It would be entirely possible, for example, to implement a version of Java on top of the Erlang VM -- but we would require adding a few primitives to the language (some for concurrency, some for type completeness). http://stackoverflow.com/questions/32294367/erlang-process-vs-java-thread/32296577#32296577 Have fun! Remember the cardinal rule: If you're not confused then you're not learning anything! -Craig From zxq9@REDACTED Fri Dec 16 03:00:00 2016 From: zxq9@REDACTED (zxq9) Date: Fri, 16 Dec 2016 11:00 +0900 Subject: [erlang-questions] How to think and reason when designing process- and message-centric systems? In-Reply-To: <2403943.2jbFVTaLeE@changa> References: <2403943.2jbFVTaLeE@changa> Message-ID: <8200893.x4sgYOjGyk@changa> On 2016?12?16? ??? 10:58:52 zxq9 wrote: > Off-list reply due to the incomplete nature of the reference below. Well crap... that wasn't supposed to go to this address. :-/ BAH! -Craig From ok@REDACTED Fri Dec 16 05:30:15 2016 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 16 Dec 2016 17:30:15 +1300 Subject: [erlang-questions] How to think and reason when designing process- and message-centric systems? In-Reply-To: References: Message-ID: <522f43ac-db4a-0e9f-0c57-d8b69a3e49c2@cs.otago.ac.nz> There has been a lot of excellent stuff in replies so far. On 16/12/16 1:04 AM, IRLeif wrote: > Coming from an object-oriented and data-centric background, I have > cognitive difficulties when it comes to conceptualizing, thinking about > and designing systems consisting of modules, processes and key-value > data stores. > > My brain reverts to thinking about classes, objects, inheritance trees, > encapsulation and SQL-style relational data models. The conventional way of thinking about OOP is that objects are ways of encapsulating mutable state. The mutability of that state is *why* it needs encapsulation. What's really been fascinating to watch over the years has been the way that when OO languages like Java and C# met concurrency for real, people began to realise that you want as much of your data as makes sense to be IMMUTABLE. There are some really great blog articles about Microsoft's experimental Midori operating system, and one of the themes is the way they kept on trying to improve (and exploit!) support for immutable data. The way to enapsulate data in Erlang is for a process to own it. (Even in Java or C#, it's not good enough to know which OBJECT might change a datum, you really want to know which THREADS might change a datum. Here again, Midori warped C# into something that could *prove* that there were no data races, by limiting access to shared mutable data.) There's nothing wrong with SQL data bases, except of course the fact that they are a painfully bad fit for OOP. They are very good at what they were designed to do, and if fine-grained access control to data is important to you -- like if you are holding data about other people! -- then you *shouldn't* try to think away from SQL. However, in the data base area there are many interesting alternatives, like Graph databases (neo4j amongst others) and Triple Stores supporting RDF OWL and SPARQL. (Scale? Got it. Performance? Got it. Fine grained security control? Not so much.) Inheritance has its uses. But even in OOP these days, inheritance of structure and/or code is deprecated by many opinionated people in favour of interfaces, and the idea of multiple data structures with similar interfaces is not alien to functional programming. (Again, a Midori datum. The C# analogue of Java's 'final' is 'sealed', and Midori ended up making classes sealed by default. Oh, and methods were non-virtual by default, even in unsealed classes. Apparently they loved that C# feature; they *really* didn't want any dynamic dispatch / overriding without fairly major reasons.) There are a number of things that you have learned that will carry over quite well. Starting with use cases is going to help. Using a testing framework is just as good an idea in Erlang as it is in Smalltalk (where the XUnit framework originated). Actually, most of the mental habits you describe sound like IMPLEMENTATION-ORIENTED thinking. I'm suggesting taking a higher level view. Here's something that happened today. My daughter is going to be doing tertiary study next year. She needed to apply for a student loan. So she needed to get a Tax File Number. For which you go to the AA. So she took - a paper form - her identification documents - me to make the payment What's the work-flow here? Is the paper form an object which can do things if you ask it nicely? Of course not! The information got copied onto another form. (The *information* matters, not the *object*.) Her identification documents were photocopied and the originals returned. (The *information* matters, not the *object*.) The copies were then scanned in again and electronic copies sent to the capital. (The *information* matters, not the *object*.) It should all have been doable on-line, using the identification numbers on the identification documents (like driver's licence), BUT it required a known human being to see them and verify that the pictures matched. Eventually, some data base is going to hold - an SQL encoding of - a form typed by a human being - reading a scanned copy - of a photocopy - of an original document issued by the same government, for which the data is already on file in some other data base(s). The *information* matters, but the *object* doesn't. This is a good thing, because in a distributed system, having multiple sites work on the same object is, well, it only makes sense for device-controlling objects that *can't* be copied or massive shared data objects that *shouldn't* be copied. You shouldn't be *deciding* which processes should communicate with each other; that should be *forced* on you by the nature of the information flow. The GRASP patterns from OO have some relevance. Perhaps the biggest problem I've had is unthinking sequential programming. I try not to think in terms of *adding* concurrency to something but of *not taking it away*. I don't succeed as often as I'd like. It's a bit like SQL data bases. You start out with a normalised design, and then you denormalise *carefully* when you need the efficiency. Again, it was a Midori lesson. LOTS of small processes. Heavy use of 'async/await': any time they wanted to do a long-running thing and didn't need the result right away, fork off a lightweight concurrent activity to do it and wait only when you really need the result before you can continue. As for graphics, people have proposed graphical notations for Erlang, and Erlang has its roots in a community who were thoroughly familiar with SDL. I'm not sure that there is one notation that is suitable for all people and all problems. (UML certainly isn't.) I'm afraid this > could lead to unidiomatic Erlang system architectures and > implementations, which would be undesirable. > > Here are some of the essential complexities I have difficulties grasping: > > A) Identifying discrete modules and processes and finding good names for > them. > B) Appointing supervisor and worker modules; defining process hierarchies. > C) Deciding which processes should communicate with each other and how. > D) Designing a sensible persistent data model with Mnesia or other NoSQL > data models (e.g. using CouchDB). > E) Deciding which processes should read and write persistent data records. > F) Incorporating global modules/"shared facilities" like event handlers, > loggers, etc. > G) Visualizing the system architecture, processes and communication > lines; what kind of graphics to use. > H) Organizing source code files into separate projects and directory > structures. > > Questions: > > 1) How do you unlearn "bad habits" from object-oriented way of thinking? > 2) How do you think and reason about process-centric systems designs? > 3) When designing a new system, how do you approach the above activities? > > I would appreciate any practical tips, examples, "mind hacks" and > resources that might help. > > Kind regards, > > Leif Eric Fredheim > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From tuanbeo0502@REDACTED Fri Dec 16 19:33:25 2016 From: tuanbeo0502@REDACTED (Tuan Cao) Date: Fri, 16 Dec 2016 10:33:25 -0800 Subject: [erlang-questions] Erlang ODBC App returns wrong for multi-statement request Message-ID: Hi Erlang community, When running multi-statement request in which the first statement returns zero rows, the Erlang ODBC App returns wrong result. To reproduce: 1> odbc:start(). ok 2> {ok, Ref}=odbc:connect("Driver=ODBC Driver 13 for SQL Server;Server=XXX;Database=YYY;Uid=ZZZ;Pwd=PPP;Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;", []). {ok,<0.40.0>} 3> odbc:sql_query(Ref, "create table z1(i integer);"). {updated,undefined} 4> odbc:sql_query(Ref, "insert into z1 select * from z1 where 1<0; select 1;"). {updated,0} As you can see, odbc_sql_query only returns the first result, and ignore all subsequence results. Looking at the source code, I found that this code block swallow subsequence result: /* OTP-5759, fails when 0 rows deleted */ if (result == SQL_NO_DATA_FOUND) { msg = encode_result(state); } else { /* Handle multiple result sets */ do { ei_x_encode_list_header(&dynamic_buffer(state), 1); msg = encode_result(state); /* We don't want to continue if an error occured */ if (msg.length != 0) { break; } msg = more_result_sets(state); /* We don't want to continue if an error occured */ if (msg.length != 0) { break; } } while (exists_more_result_sets(state)); ei_x_encode_empty_list(&dynamic_buffer(state)); } -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuanbeo0502@REDACTED Fri Dec 16 19:45:10 2016 From: tuanbeo0502@REDACTED (Tuan Cao) Date: Fri, 16 Dec 2016 10:45:10 -0800 Subject: [erlang-questions] Erlang ODBC App returns wrong for multi-statement request In-Reply-To: References: Message-ID: Oops, the previous email was sent when I pasted the source code. Continue .... I can't find any detailed information about the issue OTP-5759. IMHO, the else branch is good enough for this code block. In particular, we can replace that code block by: /* Handle multiple result sets */ do { ei_x_encode_list_header(&dynamic_buffer(state), 1); msg = encode_result(state); /* We don't want to continue if an error occured */ if (msg.length != 0) { break; } msg = more_result_sets(state); /* We don't want to continue if an error occured */ if (msg.length != 0) { break; } } while (exists_more_result_sets(state)); ei_x_encode_empty_list(&dynamic_buffer(state)); I am afraid that doing so will re-surface the issue reported by OTP-5759. If you have detailed information(for example: bug report, how it was fixed), please share with me. Thanks, Tuan On Fri, Dec 16, 2016 at 10:33 AM, Tuan Cao wrote: > Hi Erlang community, > > > When running multi-statement request in which the first statement returns > zero rows, the Erlang ODBC App returns wrong result. > > To reproduce: > > 1> odbc:start(). > > ok > > 2> {ok, Ref}=odbc:connect("Driver=ODBC Driver 13 for SQL > Server;Server=XXX;Database=YYY;Uid=ZZZ;Pwd=PPP;Encrypt= > yes;TrustServerCertificate=no;Connection Timeout=30;", []). > > {ok,<0.40.0>} > > 3> odbc:sql_query(Ref, "create table z1(i integer);"). > > {updated,undefined} > > 4> odbc:sql_query(Ref, "insert into z1 select * from z1 where 1<0; select > 1;"). > > {updated,0} > > > As you can see, odbc_sql_query only returns the first result, and ignore > all subsequence results. > > Looking at the source code, I found that this code block swallow > subsequence result: > > > > /* OTP-5759, fails when 0 rows deleted */ > if (result == SQL_NO_DATA_FOUND) { > msg = encode_result(state); > } else { > /* Handle multiple result sets */ > do { > ei_x_encode_list_header(&dynamic_buffer(state), 1); > msg = encode_result(state); > /* We don't want to continue if an error occured */ > if (msg.length != 0) { > break; > } > msg = more_result_sets(state); > /* We don't want to continue if an error occured */ > if (msg.length != 0) { > break; > } > } while (exists_more_result_sets(state)); > ei_x_encode_empty_list(&dynamic_buffer(state)); > } > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Mon Dec 19 07:17:44 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Mon, 19 Dec 2016 06:17:44 +0000 Subject: [erlang-questions] escript: faster startup and connect to TargetNode Message-ID: Hi everyone, I?ve an escript which connects to a TargetNode using this code (snippet from Basho's nodetool): https://github.com/basho/rebar/blob/develop/priv/templates/simplenode.nodetool#L34-L45 %% See if the node is currently running -- if it's not, we'll bail case {net_kernel:hidden_connect_node(TargetNode), net_adm:ping(TargetNode)} of {true, pong} -> ok; {false,pong} -> io:format("Failed to connect to node ~p .\n", [TargetNode]), halt(1); {_, pang} -> io:format("Node ~p not responding to pings.\n", [TargetNode]), halt(1) end, It works perfectly well, but it?s kind of slow. I mean the connection is a bit slow to be established even it the TargetNode is running on the same host. Can we make it faster? Is there another way to connect to a TargetNoode without using net_kernel module? Finally, what kind of escript's options can make the startup faster in general? I found this one: -compile(native). Thank you /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Mon Dec 19 09:51:34 2016 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 19 Dec 2016 09:51:34 +0100 Subject: [erlang-questions] escript: faster startup and connect to TargetNode In-Reply-To: References: Message-ID: You're not doing any cpu heavy work at all, so native compilation will not do anything for you - it might just add overhead. Measure the run time for a stripped down escript version that doesn't actually try to connect, to see what the actual startup time is. Most likely, all the time is spent in the call to connect node. /Richard 2016-12-19 7:17 GMT+01:00 Frank Muller : > Hi everyone, > > I?ve an escript which connects to a TargetNode using this code (snippet > from Basho's nodetool): > https://github.com/basho/rebar/blob/develop/priv/ > templates/simplenode.nodetool#L34-L45 > > %% See if the node is currently running -- if it's not, we'll bail > case {net_kernel:hidden_connect_node(TargetNode), > net_adm:ping(TargetNode)} of > {true, pong} -> > ok; > {false,pong} -> > io:format("Failed to connect to node ~p .\n", [TargetNode]), > halt(1); > {_, pang} -> > io:format("Node ~p not responding to pings.\n", [TargetNode]), > halt(1) > end, > > It works perfectly well, but it?s kind of slow. I mean the connection is a > bit slow to be established even it the TargetNode is running on the same > host. > Can we make it faster? Is there another way to connect to a TargetNoode > without using net_kernel module? > > Finally, what kind of escript's options can make the startup faster in > general? > I found this one: > -compile(native). > > Thank you > /Frank > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Mon Dec 19 10:33:51 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Mon, 19 Dec 2016 09:33:51 +0000 Subject: [erlang-questions] escript: faster startup and connect to TargetNode In-Reply-To: References: Message-ID: Hi Richard, You're right. Dropped the "native" flag and here are the measurements: 1. Stripped down escript (no connect): 0.15ms 2. Stripped down escript (with only connect): 0.79ms These confirm your assumptions. So the "connect" is where most of the time is spent. Q: is there another way to make the "connect" quicker (undocumented calls)? Or, any another way to connect to a node? Thank you ! /Frank Le lun. 19 d?c. 2016 ? 09:51, Richard Carlsson a ?crit : > You're not doing any cpu heavy work at all, so native compilation will not > do anything for you - it might just add overhead. > > Measure the run time for a stripped down escript version that doesn't > actually try to connect, to see what the actual startup time is. Most > likely, all the time is spent in the call to connect node. > > > /Richard > > > > 2016-12-19 7:17 GMT+01:00 Frank Muller : > > Hi everyone, > > I?ve an escript which connects to a TargetNode using this code (snippet > from Basho's nodetool): > > https://github.com/basho/rebar/blob/develop/priv/templates/simplenode.nodetool#L34-L45 > > %% See if the node is currently running -- if it's not, we'll bail > case {net_kernel:hidden_connect_node(TargetNode), > net_adm:ping(TargetNode)} of > {true, pong} -> > ok; > {false,pong} -> > io:format("Failed to connect to node ~p .\n", [TargetNode]), > halt(1); > {_, pang} -> > io:format("Node ~p not responding to pings.\n", [TargetNode]), > halt(1) > end, > > It works perfectly well, but it?s kind of slow. I mean the connection is a > bit slow to be established even it the TargetNode is running on the same > host. > Can we make it faster? Is there another way to connect to a TargetNoode > without using net_kernel module? > > Finally, what kind of escript's options can make the startup faster in > general? > I found this one: > -compile(native). > > Thank you > /Frank > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Mon Dec 19 10:37:44 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Mon, 19 Dec 2016 09:37:44 +0000 Subject: [erlang-questions] escript: faster startup and connect to TargetNode In-Reply-To: References: Message-ID: Sorry, hit the send button too fast. 1. Stripped down escript (no connect): ~0.150ms 2. Stripped down escript (with only connect): ~0.790ms /Frank Le lun. 19 d?c. 2016 ? 10:33, Frank Muller a ?crit : > Hi Richard, > > You're right. Dropped the "native" flag and here are the measurements: > > 1. Stripped down escript (no connect): > 0.15ms > > 2. Stripped down escript (with only connect): > 0.79ms > > These confirm your assumptions. So the "connect" is where most of the time > is spent. > > Q: is there another way to make the "connect" quicker (undocumented > calls)? Or, any another way to connect to a node? > > Thank you ! > > /Frank > > Le lun. 19 d?c. 2016 ? 09:51, Richard Carlsson > a ?crit : > > You're not doing any cpu heavy work at all, so native compilation will not > do anything for you - it might just add overhead. > > Measure the run time for a stripped down escript version that doesn't > actually try to connect, to see what the actual startup time is. Most > likely, all the time is spent in the call to connect node. > > > /Richard > > > > 2016-12-19 7:17 GMT+01:00 Frank Muller : > > Hi everyone, > > I?ve an escript which connects to a TargetNode using this code (snippet > from Basho's nodetool): > > https://github.com/basho/rebar/blob/develop/priv/templates/simplenode.nodetool#L34-L45 > > %% See if the node is currently running -- if it's not, we'll bail > case {net_kernel:hidden_connect_node(TargetNode), > net_adm:ping(TargetNode)} of > {true, pong} -> > ok; > {false,pong} -> > io:format("Failed to connect to node ~p .\n", [TargetNode]), > halt(1); > {_, pang} -> > io:format("Node ~p not responding to pings.\n", [TargetNode]), > halt(1) > end, > > It works perfectly well, but it?s kind of slow. I mean the connection is a > bit slow to be established even it the TargetNode is running on the same > host. > Can we make it faster? Is there another way to connect to a TargetNoode > without using net_kernel module? > > Finally, what kind of escript's options can make the startup faster in > general? > I found this one: > -compile(native). > > Thank you > /Frank > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Dinislam.Salikhov@REDACTED Mon Dec 19 13:29:18 2016 From: Dinislam.Salikhov@REDACTED (Salikhov Dinislam) Date: Mon, 19 Dec 2016 15:29:18 +0300 Subject: [erlang-questions] Garbage collection Message-ID: Hello! I've got a couple of questions about garbage collection: 1. AFAIU, a process's heap is garbage collected when the process is executed. A process creates and destroys new objects and when there is no enough room for the new object in the heap, GC is invoked. When the process is in scheduler's wait queue, then no new objects are created, so GC isn't called. The only exception, I see, is when new messages are sent to suspended process: they are to be kept in receiver's heap and it may lead to GC invoked. Is my understanding correct? Or there are other possibilities to GC suspended process? 2. In OTP-19.2 sources, erts/emulator/beam/erl_gc.c:sweep_off_heap() may reallocate ProcBins including the struct binary they point to. The list of ProcBins to realloc is received in link_live_proc_bin(). What prevents us to reallocate the binary that may be referenced by another process? Salikhov Dinislam From dmytro.lytovchenko@REDACTED Mon Dec 19 13:46:45 2016 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Mon, 19 Dec 2016 13:46:45 +0100 Subject: [erlang-questions] Garbage collection In-Reply-To: References: Message-ID: 1. When the process is suspended some external events may happen, such as a kill signal passed to a process. Its value has to be allocated on the process heap, so it is possible that GC is invoked. If i remember right, on heap overflow messages will land in heap fragments. Next GC will consolidate them into a single new heap. 2. ProcBins are (iirc) onheap binaries <64 bytes. Each larger binary stored on heap is a reference counted pointer to binary heap. Until refcount is >0 the binary remains locked. 2016-12-19 13:29 GMT+01:00 Salikhov Dinislam < Dinislam.Salikhov@REDACTED>: > Hello! > > I've got a couple of questions about garbage collection: > 1. AFAIU, a process's heap is garbage collected when the process is > executed. A process creates and destroys new objects and when there is no > enough room for the new object in the heap, GC is invoked. > When the process is in scheduler's wait queue, then no new objects are > created, so GC isn't called. > The only exception, I see, is when new messages are sent to suspended > process: they are to be kept in receiver's heap and it may lead to GC > invoked. > Is my understanding correct? Or there are other possibilities to GC > suspended process? > > 2. In OTP-19.2 sources, erts/emulator/beam/erl_gc.c:sweep_off_heap() may > reallocate ProcBins including the struct binary they point to. The list of > ProcBins to realloc is received in link_live_proc_bin(). What prevents us > to reallocate the binary that may be referenced by another process? > > Salikhov Dinislam > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmytro.lytovchenko@REDACTED Mon Dec 19 13:48:55 2016 From: dmytro.lytovchenko@REDACTED (Dmytro Lytovchenko) Date: Mon, 19 Dec 2016 13:48:55 +0100 Subject: [erlang-questions] Garbage collection In-Reply-To: References: Message-ID: on 1. Such GC on a suspended process sometimes is DELAYED. Depends which opcode caused this, some opcodes such as gc_bif have Live argument which allows running a GC inline, other will have to call GC but DELAY it. 2016-12-19 13:46 GMT+01:00 Dmytro Lytovchenko : > 1. When the process is suspended some external events may happen, such as > a kill signal passed to a process. Its value has to be allocated on the > process heap, so it is possible that GC is invoked. > If i remember right, on heap overflow messages will land in heap > fragments. Next GC will consolidate them into a single new heap. > > 2. ProcBins are (iirc) onheap binaries <64 bytes. Each larger binary > stored on heap is a reference counted pointer to binary heap. Until > refcount is >0 the binary remains locked. > > 2016-12-19 13:29 GMT+01:00 Salikhov Dinislam com>: > >> Hello! >> >> I've got a couple of questions about garbage collection: >> 1. AFAIU, a process's heap is garbage collected when the process is >> executed. A process creates and destroys new objects and when there is no >> enough room for the new object in the heap, GC is invoked. >> When the process is in scheduler's wait queue, then no new objects are >> created, so GC isn't called. >> The only exception, I see, is when new messages are sent to suspended >> process: they are to be kept in receiver's heap and it may lead to GC >> invoked. >> Is my understanding correct? Or there are other possibilities to GC >> suspended process? >> >> 2. In OTP-19.2 sources, erts/emulator/beam/erl_gc.c:sweep_off_heap() may >> reallocate ProcBins including the struct binary they point to. The list of >> ProcBins to realloc is received in link_live_proc_bin(). What prevents us >> to reallocate the binary that may be referenced by another process? >> >> Salikhov Dinislam >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Dinislam.Salikhov@REDACTED Mon Dec 19 14:15:05 2016 From: Dinislam.Salikhov@REDACTED (Salikhov Dinislam) Date: Mon, 19 Dec 2016 16:15:05 +0300 Subject: [erlang-questions] Garbage collection In-Reply-To: References: Message-ID: <85cd1e3d-a9fa-265b-17b1-6969ec8a9213@kaspersky.com> Thank you for quick response, Dmytro! I'm afraid you're a bit misled. > 2. ProcBins are (iirc) onheap binaries <64 bytes. Each larger binary stored on heap is a reference counted pointer to binary heap. Until refcount is >0 the binary remains locked. According to http://erlang.org/doc/efficiency_guide/binaryhandling.html, ProcBin is a wrapper for globally stored binary while the latter is refcounted. Heap binaries (<64 bytes) are different thing and are handled without ProcBins involved. On 12/19/2016 03:46 PM, Dmytro Lytovchenko wrote: > 1. When the process is suspended some external events may happen, such > as a kill signal passed to a process. Its value has to be allocated on > the process heap, so it is possible that GC is invoked. > If i remember right, on heap overflow messages will land in heap > fragments. Next GC will consolidate them into a single new heap. > > 2. ProcBins are (iirc) onheap binaries <64 bytes. Each larger binary > stored on heap is a reference counted pointer to binary heap. Until > refcount is >0 the binary remains locked. > > 2016-12-19 13:29 GMT+01:00 Salikhov Dinislam > >: > > Hello! > > I've got a couple of questions about garbage collection: > 1. AFAIU, a process's heap is garbage collected when the process > is executed. A process creates and destroys new objects and when > there is no enough room for the new object in the heap, GC is invoked. > When the process is in scheduler's wait queue, then no new objects > are created, so GC isn't called. > The only exception, I see, is when new messages are sent to > suspended process: they are to be kept in receiver's heap and it > may lead to GC invoked. > Is my understanding correct? Or there are other possibilities to > GC suspended process? > > 2. In OTP-19.2 sources, > erts/emulator/beam/erl_gc.c:sweep_off_heap() may reallocate > ProcBins including the struct binary they point to. The list of > ProcBins to realloc is received in link_live_proc_bin(). What > prevents us to reallocate the binary that may be referenced by > another process? > > Salikhov Dinislam > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Mon Dec 19 15:04:08 2016 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 19 Dec 2016 15:04:08 +0100 Subject: [erlang-questions] escript: faster startup and connect to TargetNode In-Reply-To: References: Message-ID: 2016-12-19 10:37 GMT+01:00 Frank Muller : > These confirm your assumptions. So the "connect" is where most of the time > is spent. > > Q: is there another way to make the "connect" quicker (undocumented > calls)? Or, any another way to connect to a node? > I'll leave that answer to someone trained in the dark arts of networks. But again, it might be useful to do some basic measurements to see if it's reasonable to think that the connection time is slow. What are the ping times between the machines? When already connected, how long does an Erlang-level net_adm:ping() roundtrip take? How fast can some other comparable tools do a request-reply roundtrip? Et cetera. /Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Mon Dec 19 15:41:22 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Mon, 19 Dec 2016 14:41:22 +0000 Subject: [erlang-questions] escript: faster startup and connect to TargetNode In-Reply-To: References: Message-ID: As I've explained before, the escript is running in the same machine as the TargetNode. This is why I've posted the question. They're both running locally. /Frank Le lun. 19 d?c. 2016 ? 15:04, Richard Carlsson a ?crit : > 2016-12-19 10:37 GMT+01:00 Frank Muller : > > These confirm your assumptions. So the "connect" is where most of the time > is spent. > > Q: is there another way to make the "connect" quicker (undocumented > calls)? Or, any another way to connect to a node? > > > I'll leave that answer to someone trained in the dark arts of networks. > But again, it might be useful to do some basic measurements to see if it's > reasonable to think that the connection time is slow. What are the ping > times between the machines? When already connected, how long does an > Erlang-level net_adm:ping() roundtrip take? How fast can some other > comparable tools do a request-reply roundtrip? Et cetera. > > /Richard > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennethlakin@REDACTED Mon Dec 19 16:26:49 2016 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Mon, 19 Dec 2016 07:26:49 -0800 Subject: [erlang-questions] snmpc: Error when compiling RFC 4669 MIB and others Message-ID: I'm running Erlang 19.1.3. I'm using snmpc to compile the MIBs from RFCs 4668, 4669, 4670, and 4671. (These are the updated RADIUS client/server and RADIUS Accounting client/server MIBs.) Attempts to compile any these MIBs fail with the same error (but with differing line numbers) : Eshell V8.1 (abort with ^G) 1> snmpc:compile("RADIUS-AUTH-SERVER-MIB", []). RADIUS-AUTH-SERVER-MIB.mib: 692: syntax error before: '{' {error,compilation_failed} 2> In all cases, the line causing the trouble is a SYNTAX line with a -er- type of InetAddressType inside an OBJECT block, inside a MODULE-COMPLIANCE block. If I change the SYNTAX type of that OBJECT to something else (like, say InetAddress), and update the constraints, the compilation succeeds with the following warnings and return value: [RADIUS-AUTH-SERVER-MIB.mib][WAR]: No RowStatus column in table radiusAuthClientTable => The default functions won't work properly [RADIUS-AUTH-SERVER-MIB.mib][WAR]: No RowStatus column in table radiusAuthClientExtTable => The default functions won't work properly {ok,"./RADIUS-AUTH-SERVER-MIB.bin"} Both InetAddressType and InetAddress appear to be defined in the same MIB. That MIB file appears to be part of the Erlang snmp application. The MIB files that I'm attempting to compile are in the current working directory of the Erlang shell. Three separate reputable-looking MIB syntax checkers can find no significant problem with the files I'm attempting to compile. Why is compilation failing? Is there something that I'm overlooking, or is this a bug? The RFC 4669 MIB is at [0]. See [1] for the plaintext version of the same file. [0] https://gist.github.com/kennethlakin/9bdefaeb7c7c8a4945c45ace184c658d [1] https://gist.githubusercontent.com/kennethlakin/9bdefaeb7c7c8a4945c45ace184c658d/raw/469a639ffff47a9ed47099af3c344a29c9fe35ba/RADIUS-AUTH-SERVER-MIB.mib -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From solvip@REDACTED Mon Dec 19 16:31:34 2016 From: solvip@REDACTED (=?UTF-8?B?U8O2bHZpIFDDoWxsIMOBc2dlaXJzc29u?=) Date: Mon, 19 Dec 2016 15:31:34 +0000 Subject: [erlang-questions] Type specifications for modules implementing a specific behaviour Message-ID: Hi (sorry if this is a duplicate, but I sent this message to the list last week, before subscribing to it, cannot find it anywhere and assumed it went to the bitbucket) I've searched, but I haven't found anything and I'm afraid of what I'm asking is impossible, but I thought I'd ask here as well. Is there a way for us to say that a type spec is describing a module that should implement a specific behaviour? That is, I want to do something like the following: -module(foo). -callback foo(bar) -> quux. Then I'll have a concrete implementation of this: -module(bar). -behaviour(foo). foo(bar) -> quux. In the third module, I'd like to spec a record so that I can state that a certain field of the record should contain an atom which is the name of a module implementing a behaviour: -module(quux). -record(state, { callback_mod = bar :: behaviour(foo) }). I'd then of course like Dialyzer to tell me if something is wrong. Can we do something like this? If not, would it be extremely difficult to add support for this? Where could I start? Thanks alot S?lvi P?ll ?sgeirsson From ali.sabil@REDACTED Mon Dec 19 17:07:31 2016 From: ali.sabil@REDACTED (Ali Sabil) Date: Mon, 19 Dec 2016 16:07:31 +0000 Subject: [erlang-questions] Type specifications for modules implementing a specific behaviour In-Reply-To: References: Message-ID: I don't think this is supported by dialyzer today, but it would indeed be a very valuable addition. It's a feature I would really appreciate myself. On Mon, Dec 19, 2016 at 4:40 PM S?lvi P?ll ?sgeirsson wrote: > Hi > > (sorry if this is a duplicate, but I sent this message to the list > last week, before subscribing to it, cannot find it anywhere and > assumed it went to the bitbucket) > > I've searched, but I haven't found anything and I'm afraid of what I'm > asking is impossible, but I thought I'd ask here as well. > > Is there a way for us to say that a type spec is describing a module > that should implement a specific behaviour? > > That is, I want to do something like the following: > > -module(foo). > -callback foo(bar) -> quux. > > Then I'll have a concrete implementation of this: > > -module(bar). > -behaviour(foo). > foo(bar) -> quux. > > In the third module, I'd like to spec a record so that I can state > that a certain field of the record > should contain an atom which is the name of a module implementing a > behaviour: > > -module(quux). > -record(state, { callback_mod = bar :: behaviour(foo) }). > > I'd then of course like Dialyzer to tell me if something is wrong. > Can we do something like this? If not, would it be extremely > difficult to add support for this? Where could I start? > > Thanks alot > S?lvi P?ll ?sgeirsson > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.goertzen@REDACTED Mon Dec 19 18:11:17 2016 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Mon, 19 Dec 2016 17:11:17 +0000 Subject: [erlang-questions] snmpc: Error when compiling RFC 4669 MIB and others In-Reply-To: References: Message-ID: On the lines that have errors, does it help if you change the type to INTEGER? (ie InetAddressType becomes INTEGER). On Mon, Dec 19, 2016 at 9:27 AM Kenneth Lakin wrote: > I'm running Erlang 19.1.3. I'm using snmpc to compile the MIBs from RFCs > 4668, 4669, 4670, and 4671. (These are the updated RADIUS client/server > and RADIUS Accounting client/server MIBs.) Attempts to compile any these > MIBs fail with the same error (but with differing line numbers) : > > Eshell V8.1 (abort with ^G) > 1> snmpc:compile("RADIUS-AUTH-SERVER-MIB", []). > RADIUS-AUTH-SERVER-MIB.mib: 692: syntax error before: '{' > {error,compilation_failed} > 2> > > In all cases, the line causing the trouble is a SYNTAX line with a -er- > type of InetAddressType inside an OBJECT block, inside a > MODULE-COMPLIANCE block. If I change the SYNTAX type of that OBJECT to > something else (like, say InetAddress), and update the constraints, the > compilation succeeds with the following warnings and return value: > > [RADIUS-AUTH-SERVER-MIB.mib][WAR]: No RowStatus column in table > radiusAuthClientTable => The default functions won't work properly > [RADIUS-AUTH-SERVER-MIB.mib][WAR]: No RowStatus column in table > radiusAuthClientExtTable => The default functions won't work properly > {ok,"./RADIUS-AUTH-SERVER-MIB.bin"} > > Both InetAddressType and InetAddress appear to be defined in the same > MIB. That MIB file appears to be part of the Erlang snmp application. > The MIB files that I'm attempting to compile are in the current working > directory of the Erlang shell. Three separate reputable-looking MIB > syntax checkers can find no significant problem with the files I'm > attempting to compile. Why is compilation failing? Is there something > that I'm overlooking, or is this a bug? > > The RFC 4669 MIB is at [0]. See [1] for the plaintext version of the > same file. > > [0] https://gist.github.com/kennethlakin/9bdefaeb7c7c8a4945c45ace184c658d > [1] > > https://gist.githubusercontent.com/kennethlakin/9bdefaeb7c7c8a4945c45ace184c658d/raw/469a639ffff47a9ed47099af3c344a29c9fe35ba/RADIUS-AUTH-SERVER-MIB.mib > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Mon Dec 19 22:52:12 2016 From: mjtruog@REDACTED (Michael Truog) Date: Mon, 19 Dec 2016 13:52:12 -0800 Subject: [erlang-questions] Remove behaviour checking from erl_lint (continued) Message-ID: <5858568C.6010306@gmail.com> Based on the discussion of https://github.com/erlang/otp/pull/1274 there needs to be a discussion on the mailing list to determine if we can remove behaviour checking in the Erlang compiler, and to determine where the behaviour checking will be done in the future. Since the atomic update unit for source code in Erlang is a module, not an Erlang/OTP application, it seems best (to me) to avoid having the compiler attempt to process Erlang/OTP applications, unless many things were changed so that all Erlang/OTP application modules were versioned and updated together, possibly allowing the coexistence of separate versions of the same Erlang/OTP application. Inlining across module boundaries has been discussed before (e.g., http://erlang.org/pipermail/erlang-questions/2015-December/086943.html https://github.com/erlang/otp/pull/1274 ), and pursuing inlining within an Erlang/OTP application seems best if the Erlang/OTP application is updated as one atomic update unit that is versioned together. The current Erlang behaviour checking is a simpler problem that is mainly about trying to determine where linting should take place. Much of the lint activity for Erlang source code occurs in dialyzer, so that seems like a natural place to move behaviour checking to (to me). The reason we are talking about this is to make sure the Erlang compiler is a dependable executable that can be part of a dependable toolchain, since we want a software stack that can work together in a way where everything is as reusable as possible. A compiler that sometimes fails due to a warning when ran with parallel execution would be undependable as part of a larger toolchain. Having a toolchain that avoids monolithic development helps to pursue more efficient development, since each monolithic development effort is effectively a dead-end due to the replicated effort in each monolithic result (similar to rebar2 and rebar3 coexisting, and both being difficult to use in a separate build-system). Having several smaller tools that are reusable helps to make sure everything can grow to fit any use-case (similar to how maven has grown in the Java ecosystem). So, this may not fit into one email thread, but it does appear to require more discussion to move further. Best Regards, Michael From t.greenwoodgeer@REDACTED Tue Dec 20 01:46:20 2016 From: t.greenwoodgeer@REDACTED (Todd Greenwood-Geer) Date: Mon, 19 Dec 2016 16:46:20 -0800 Subject: [erlang-questions] How to build erlang without wxWindows? Message-ID: <1d925e6c-7dd9-291e-1671-7fecbd4fa966@gmail.com> 1. Desktop builds fail on Ubuntu 16.04 I'm was trying to build Erlang 19.2 on my desktop (ubuntu 16.04.1 LTS) and ran into trouble with linking in various libraries/apps: * sometimes openssl is found, sometimes not... * turns out you specifying the path to the --with-javac=... isn't used at all. But updating your PATH to include the jdk directory works After spinning my wheels for a half day, I thought I'd try building something that worked, or at least used to work... vagrant-erlang. This way I could create repeatable builds, share them, and/or ask for help... 2. Vagrant builds fail on Ubuntu 12.04 vagrant-erlang, without modifications, fails b/c the paths to kerl have changed. I forked the repo to resolve these issues here: https://github.com/ToddG/vagrant-erlang This project is building R16B03 on Ubuntu 12.04... once I get this working, I'd like to move forward to 19.2 on Ubuntu 16x... I've updated the kerl paths, but I cannot successfully disable wx from the build, nor can I install an acceptable version of wx... 3. Failed work-arounds * Adding the wxWindows libraries to this VM Unfortunately, wx2.8 is available in the apt-repos, but the build requires >= 2.8.4. * Removing wx from the list of enabled applications I have tried to remove this via kerl options as: * KERL_CONFIGURE_DISABLE_APPLICATIONS="wx" or as: * KERL_CONFIGURE_OPTIONS="... --without-wx" But the build keeps failing trying to build wx... ??? The log file shows all: https://github.com/ToddG/vagrant-erlang/blob/master/otp_build_R16B03.log Note, the ./lib/wx/SKIP file was created, it just doesn't seem to matter to the build... Any help is appreciated. Feel free to try out the Vagrant file yourself... I'll keep iterating on this project, as I'd eventually like to have this building 19.2 with/without wxWindows... eventually a statically linked binary (no ext deps like openssl! Though the jdk will probably always be separate...). -Todd From g.a.vinogradov@REDACTED Tue Dec 20 03:33:56 2016 From: g.a.vinogradov@REDACTED (Gleb Vinogradov) Date: Tue, 20 Dec 2016 09:33:56 +0700 Subject: [erlang-questions] Remove behaviour checking from erl_lint (continued) In-Reply-To: <5858568C.6010306@gmail.com> References: <5858568C.6010306@gmail.com> Message-ID: Let's compare `-spec` and `-behaviour` declarations. Erlang compiler doesn't check `-spec` declarations and doesn't produce warnings if function called with wrong parameters. Dialyzer does. So, I think behaviour checks could be removed from compiler as well, and should be placed in dialyzer. So behaviours of compiler and dialyzer will be consistent - one tool compiles the code and doesn't care about declarations, but another does. Best Regards, Gleb 2016-12-20 4:52 GMT+07:00 Michael Truog : > Based on the discussion of https://github.com/erlang/otp/pull/1274 there > needs to be a discussion on the mailing list to determine if we can remove > behaviour checking in the Erlang compiler, and to determine where the > behaviour checking will be done in the future. > > Since the atomic update unit for source code in Erlang is a module, not an > Erlang/OTP application, it seems best (to me) to avoid having the compiler > attempt to process Erlang/OTP applications, unless many things were changed > so that all Erlang/OTP application modules were versioned and updated > together, possibly allowing the coexistence of separate versions of the > same Erlang/OTP application. Inlining across module boundaries has been > discussed before (e.g., http://erlang.org/pipermail/er > lang-questions/2015-December/086943.html https://github.com/erlang/otp/ > pull/1274 ), and pursuing inlining within an Erlang/OTP application seems > best if the Erlang/OTP application is updated as one atomic update unit > that is versioned together. > > The current Erlang behaviour checking is a simpler problem that is mainly > about trying to determine where linting should take place. Much of the lint > activity for Erlang source code occurs in dialyzer, so that seems like a > natural place to move behaviour checking to (to me). The reason we are > talking about this is to make sure the Erlang compiler is a dependable > executable that can be part of a dependable toolchain, since we want a > software stack that can work together in a way where everything is as > reusable as possible. A compiler that sometimes fails due to a warning > when ran with parallel execution would be undependable as part of a larger > toolchain. Having a toolchain that avoids monolithic development helps to > pursue more efficient development, since each monolithic development effort > is effectively a dead-end due to the replicated effort in each monolithic > result (similar to rebar2 and rebar3 coexisting, and both being difficult > to use in a separate build-system). Having several smaller tools that are > reusable helps to make sure everything can grow to fit any use-case > (similar to how maven has grown in the Java ecosystem). So, this may not > fit into one email thread, but it does appear to require more discussion to > move further. > > Best Regards, > Michael > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t.greenwoodgeer@REDACTED Tue Dec 20 07:18:50 2016 From: t.greenwoodgeer@REDACTED (Todd Greenwood-Geer) Date: Mon, 19 Dec 2016 22:18:50 -0800 Subject: [erlang-questions] How to build erlang without wxWindows? In-Reply-To: <1d925e6c-7dd9-291e-1671-7fecbd4fa966@gmail.com> References: <1d925e6c-7dd9-291e-1671-7fecbd4fa966@gmail.com> Message-ID: Successfully building 19.2 on Ubuntu 16.04 and 16.10 (vagrant + virtualbox). The key was to add the following: Add "--without-wx" to the KERL_CONFIGURE_OPTIONS, which in turn get passed to 'configure'. FWIW - the build still complains that wx is not enabled, but continues with the build. -Todd https://github.com/ToddG/vagrant-erlang On 12/19/2016 04:46 PM, Todd Greenwood-Geer wrote: > 1. Desktop builds fail on Ubuntu 16.04 > > I'm was trying to build Erlang 19.2 on my desktop (ubuntu 16.04.1 LTS) > and ran into trouble with linking in various libraries/apps: > > * sometimes openssl is found, sometimes not... > * turns out you specifying the path to the --with-javac=... isn't used > at all. But updating your PATH to include the jdk directory works > > After spinning my wheels for a half day, I thought I'd try building > something that worked, or at least used to work... vagrant-erlang. > This way I could create repeatable builds, share them, and/or ask for > help... > > 2. Vagrant builds fail on Ubuntu 12.04 > > vagrant-erlang, without modifications, fails b/c the paths to kerl > have changed. I forked the repo to resolve these issues here: > > https://github.com/ToddG/vagrant-erlang > > This project is building R16B03 on Ubuntu 12.04... once I get this > working, I'd like to move forward to 19.2 on Ubuntu 16x... > > I've updated the kerl paths, but I cannot successfully disable wx from > the build, nor can I install an acceptable version of wx... > > 3. Failed work-arounds > > * Adding the wxWindows libraries to this VM > > Unfortunately, wx2.8 is available in the apt-repos, but > the build requires >= 2.8.4. > > * Removing wx from the list of enabled applications > > I have tried to remove this via kerl options as: > > * KERL_CONFIGURE_DISABLE_APPLICATIONS="wx" > > or as: > > * KERL_CONFIGURE_OPTIONS="... --without-wx" > > But the build keeps failing trying to build wx... ??? > > The log file shows all: > > https://github.com/ToddG/vagrant-erlang/blob/master/otp_build_R16B03.log > > Note, the ./lib/wx/SKIP file was created, it just doesn't seem to > matter to the build... > > Any help is appreciated. Feel free to try out the Vagrant file > yourself... > > I'll keep iterating on this project, as I'd eventually like to have > this building 19.2 with/without wxWindows... eventually a statically > linked binary (no ext deps like openssl! Though the jdk will probably > always be separate...). > > -Todd From mjtruog@REDACTED Tue Dec 20 09:36:07 2016 From: mjtruog@REDACTED (Michael Truog) Date: Tue, 20 Dec 2016 00:36:07 -0800 Subject: [erlang-questions] Remove behaviour checking from erl_lint (continued) In-Reply-To: <19643ace-bc11-4329-aa97-6fa8e0aa0a27@Spark> References: <5858568C.6010306@gmail.com> <19643ace-bc11-4329-aa97-6fa8e0aa0a27@Spark> Message-ID: <5858ED77.4040009@gmail.com> On 12/19/2016 11:09 PM, Micha? Muska?a wrote: > Right now when I don't implement a callback I get the feedback from the compiler in a matter of seconds - the compiler is very quick even on large projects since it can be incremental. Moving the check to dialyzer will mean I need to launch a separate program and wait for its output, potentially couple minutes. This doesn't sound like an improvement to me. > > Micha?. > It may not seem like an improvement, but it is required due to the compiler needing to focus only on the module it is trying to compile. Any error checking that involves other modules needs to rely on something else, with dialyzer being the Erlang linting tool due to the dynamic typing in Erlang. With static typing in C/C++ the linting can happen along with compilation. Even if Erlang changes to process Erlang/OTP applications as chunks of modules, that doesn't handle when behaviours would be in other Erlang/OTP applications (also, that processing appears to only exist far in the future or just simply as a discussion topic, nothing concrete to talk/think about right now). Your main complaint is the speed of execution of dialyzer, and that is another valid problem. An easy way to try to fix the slowness in dialyzer is try to make it do less, where enabling more checks requires more flags. The slowness perceived in dialyzer appears to be the main reason given for manual usage of xref, so there are also people that would prefer using xref instead, just simply due to it doing less and executing quicker when doing less. My main concern is just being able to have a dependable toolchain for Erlang usage, so creating a new lint tool doesn't seem like a bad option for me. Worst case is that it would only do behaviour checking, and could grow to whatever is required that dialyzer doesn't need to cover (just easy checks that doesn't require dialyzer's use of memory or execution time). That approach probably wants something more like xref. However, xref seems to provide a long list of false negatives when it is used and it has tons of options to make it used in countless ways, not making it a simple tool that everyone wants to use for all their development. From essen@REDACTED Tue Dec 20 10:21:31 2016 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Tue, 20 Dec 2016 10:21:31 +0100 Subject: [erlang-questions] Remove behaviour checking from erl_lint (continued) In-Reply-To: <5858ED77.4040009@gmail.com> References: <5858568C.6010306@gmail.com> <19643ace-bc11-4329-aa97-6fa8e0aa0a27@Spark> <5858ED77.4040009@gmail.com> Message-ID: <0bc754aa-71a7-460f-57dc-890aae243944@ninenines.eu> On 12/20/2016 09:36 AM, Michael Truog wrote: > On 12/19/2016 11:09 PM, Micha? Muska?a wrote: >> Right now when I don't implement a callback I get the feedback from >> the compiler in a matter of seconds - the compiler is very quick even >> on large projects since it can be incremental. Moving the check to >> dialyzer will mean I need to launch a separate program and wait for >> its output, potentially couple minutes. This doesn't sound like an >> improvement to me. >> >> Micha?. >> > It may not seem like an improvement, but it is required due to the > compiler needing to focus only on the module it is trying to compile. > Any error checking that involves other modules needs to rely on > something else, with dialyzer being the Erlang linting tool due to the I believe Dialyzer already checks this, perhaps not exactly the same way, and with missing behavior modules not being considered errors. > Your main complaint is the speed of execution of dialyzer, and that is > another valid problem. An easy way to try to fix the slowness in > dialyzer is try to make it do less, where enabling more checks requires > more flags. The slowness perceived in dialyzer appears to be the main > reason given for manual usage of xref, so there are also people that > would prefer using xref instead, just simply due to it doing less and > executing quicker when doing less. My main concern is just being able > to have a dependable toolchain for Erlang usage, so creating a new lint > tool doesn't seem like a bad option for me. Worst case is that it would > only do behaviour checking, and could grow to whatever is required that > dialyzer doesn't need to cover (just easy checks that doesn't require > dialyzer's use of memory or execution time). That approach probably > wants something more like xref. However, xref seems to provide a long > list of false negatives when it is used and it has tons of options to > make it used in countless ways, not making it a simple tool that > everyone wants to use for all their development. I'm not sure what you mean. It's not hard to run xref once with warnings that provide no false positives, and if necessary have a more informative run with possible false positives. Both runs will be largely faster than Dialyzer (maybe not that much on trivial projects when the PLT is already built, but definitely on projects with many modules, or huge generated modules and so on). I believe xref should be made more approachable though, either in documentation or interface. It seems to be capable of a lot more than "rebar xref". Cheers, -- Lo?c Hoguin https://ninenines.eu From kennethlakin@REDACTED Tue Dec 20 13:48:18 2016 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Tue, 20 Dec 2016 04:48:18 -0800 Subject: [erlang-questions] snmpc: Error when compiling RFC 4669 MIB and others In-Reply-To: References: Message-ID: <3f45ad0b-db0a-be4c-5e50-bb3220aa40fc@gmail.com> On 12/19/2016 09:11 AM, Daniel Goertzen wrote: > On the lines that have errors, does it help if you change the type to > INTEGER? (ie InetAddressType becomes INTEGER). It does. If I change the type of that object to either INTEGER or InetAddress, the MIB is accepted by the compiler. I have a minimal MIB that the compiler chokes on. Other than complaints about unused imports and about how an object of type "InetAddress" isn't preceded by an object of type "InetAddressType", other MIB checkers don't appear to see any problems with it. See: https://gist.github.com/kennethlakin/c4b17c94ea1d9a97d4db1c35c0b64b78 or https://gist.githubusercontent.com/kennethlakin/c4b17c94ea1d9a97d4db1c35c0b64b78/raw/10fa8078f33fa402d07b6613445597b35e94564d/TEST-MIB.mib Switch the type of testObj from InetAddressType to InetAddress to get a file that'll pass through the snmpc MIB compiler. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From mononcqc@REDACTED Tue Dec 20 15:52:29 2016 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 20 Dec 2016 09:52:29 -0500 Subject: [erlang-questions] Remove behaviour checking from erl_lint (continued) In-Reply-To: <0bc754aa-71a7-460f-57dc-890aae243944@ninenines.eu> References: <5858568C.6010306@gmail.com> <19643ace-bc11-4329-aa97-6fa8e0aa0a27@Spark> <5858ED77.4040009@gmail.com> <0bc754aa-71a7-460f-57dc-890aae243944@ninenines.eu> Message-ID: <20161220145228.GB18549@fhebert-ltm2.internal.salesforce.com> I'm not taking position here; I do enjoy the compiler yelling at me for valid reasons when it can, and most tools out there provide good integration with my editor based on its output rather than the output of other tools. That being said, I'm not closed to the idea of people who care about the compiler's well-being to be able to clean it up and maintain it better. On 12/20, Lo?c Hoguin wrote: >I believe xref should be made more approachable though, either in >documentation or interface. It seems to be capable of a lot more than >"rebar xref". > Xref is a better candidate to move that stuff to than dialyzer, which requires minutes of runtime the first time around, but also requires a lot of memory. On my VPS, if I try to run dialyzer on the rebar3 code base, it will take minutes, yes, but will also hover between 1.0 and 1.5GB of RAM usage while pegging 3-4 of the 4 CPUs at 100%, until it eventually gets killed by the OS. That's for 62 source files, and 29,000 LoC (we get 59 files and 12637 lines of code if we stop analyzing certifi, which has CA cert bundles inlined -- but does not change the overall dialyzer behaviour of getting killed by the OS). I'm not saying this to disparage dialyzer, but the type of analysis it can do and that I use it for (and where I run it) does not include the basic checks that xref or the compiler does. xref usability also needs to be a bit better before I'd consider it to be an intuitive tool that does not require wrapping before running as an end user. From daniel.goertzen@REDACTED Tue Dec 20 16:20:58 2016 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Tue, 20 Dec 2016 15:20:58 +0000 Subject: [erlang-questions] snmpc: Error when compiling RFC 4669 MIB and others In-Reply-To: <3f45ad0b-db0a-be4c-5e50-bb3220aa40fc@gmail.com> References: <3f45ad0b-db0a-be4c-5e50-bb3220aa40fc@gmail.com> Message-ID: I think the relevant piece of compiler code is this: https://github.com/erlang/otp/blob/maint/lib/snmp/src/compile/snmpc_mib_gram.yrl#L385-L395 The way I read this is that size refinement is allowed on a usertype, but enumeration refinement is not implemented, but should be. The RFC talking about this is https://tools.ietf.org/html/rfc2578#section-9 I suggest filing this at https://bugs.erlang.org On Tue, Dec 20, 2016 at 6:48 AM Kenneth Lakin wrote: On 12/19/2016 09:11 AM, Daniel Goertzen wrote: > On the lines that have errors, does it help if you change the type to > INTEGER? (ie InetAddressType becomes INTEGER). It does. If I change the type of that object to either INTEGER or InetAddress, the MIB is accepted by the compiler. I have a minimal MIB that the compiler chokes on. Other than complaints about unused imports and about how an object of type "InetAddress" isn't preceded by an object of type "InetAddressType", other MIB checkers don't appear to see any problems with it. See: https://gist.github.com/kennethlakin/c4b17c94ea1d9a97d4db1c35c0b64b78 or https://gist.githubusercontent.com/kennethlakin/c4b17c94ea1d9a97d4db1c35c0b64b78/raw/10fa8078f33fa402d07b6613445597b35e94564d/TEST-MIB.mib Switch the type of testObj from InetAddressType to InetAddress to get a file that'll pass through the snmpc MIB compiler. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennethlakin@REDACTED Tue Dec 20 16:29:54 2016 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Tue, 20 Dec 2016 07:29:54 -0800 Subject: [erlang-questions] snmpc: Error when compiling RFC 4669 MIB and others In-Reply-To: References: <3f45ad0b-db0a-be4c-5e50-bb3220aa40fc@gmail.com> Message-ID: On 12/20/2016 07:20 AM, Daniel Goertzen wrote: > The way I read this is that size refinement is allowed on a usertype, but > enumeration refinement is not implemented, but should be. The RFC talking > about this is https://tools.ietf.org/html/rfc2578#section-9 Thank you _very_ much for digging in to this! Kudos! > I suggest filing this at https://bugs.erlang.org Will do. Thanks! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From frank.muller.erl@REDACTED Tue Dec 20 22:16:07 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Tue, 20 Dec 2016 21:16:07 +0000 Subject: [erlang-questions] simple_one_for_one supervisor: strange error when stopping node Message-ID: Hi, I?m seeing this intermittent error when stopping my Erlang node with init:stop(). 2016-12-20 22:00:11 =SUPERVISOR REPORT==== Supervisor: {local,server_sock_sup} Context: shutdown_error Reason: noproc Offender: [{nb_children,1},{name,t3_router},{mfargs,{client_sock,start_link,[]}},{restart_type,temporary},{shutdown,5000},{child_type,worker}] 2016-12-20 22:00:11.069 [error] <0.162.0> Supervisor server_sock_sup had child client_sock started with client_sock start_link() at undefined exit with reason noproc in context shutdown_error The client_sock is a gen_server which traps exit. It?s supervised by server_sock_sup as a simple_one_for_one. I?ve also tried without trapping exit in client_sock and was also able to notice this error. Can someone please explain me what I?m doing wrong? /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From justin.k.wood@REDACTED Wed Dec 21 01:56:50 2016 From: justin.k.wood@REDACTED (Justin Wood) Date: Tue, 20 Dec 2016 19:56:50 -0500 Subject: [erlang-questions] SSL handshake failure Message-ID: Hi there, I am attempting to use the ssl module in order to get a socket to a remote server (MongoDB). I am using the following line of code (OTP 19) ssl:connect("my-server.net", 27017, [binary, {active, false}, {packet, raw}], 5000). Whenever I use this, I get the following: =ERROR REPORT==== 20-Dec-2016::19:13:13 === SSL: certify: tls_connection.erl:603:Fatal error: handshake failure - malformed_handshake {error,{tls_alert,"handshake failure"}} Which lead me to look into erlang:get_stacktrace/0 erlang:get_stacktrace(). [{tls_connection,start_fsm,8, [{file,"tls_connection.erl"},{line,79}]}, {ssl_connection,connect,8, [{file,"ssl_connection.erl"},{line,84}]}, {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,674}]}, {shell,exprs,7,[{file,"shell.erl"},{line,686}]}, {shell,eval_exprs,7,[{file,"shell.erl"},{line,641}]}, {shell,eval_loop,3,[{file,"shell.erl"},{line,626}]}] My problem is that the certificate that the server is using should be legitimate. I can connect, without error, using other clients and languages. I took a look through Wireshark in order to try and figure out what is going on and this is what I see. * The Client says hello and presents a list of cipher suites. * The Server says hello and says that it wants to use TLS_RSA_WITH_AES_256_GCM_SHA384 (This cipher is in the list that the client sends) * The Server sends the Client the certificates it is using (these both appear to be signed by DigiCert). * The Server sends an encrypted handshake message. * The Client responds with a Fatal Alert stating a Handshake Failure. * The Server sends another encrypted handshake message. I have verified that ssl:connect/4 is working fine as I can connect to a number of different miscellaneous services (including a number of other MongoDB instances). I was wondering if there is anything else I can do to try and figure out why erlang does not allow this connection. Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From technion@REDACTED Wed Dec 21 09:38:20 2016 From: technion@REDACTED (Technion) Date: Wed, 21 Dec 2016 08:38:20 +0000 Subject: [erlang-questions] SSL handshake failure In-Reply-To: References: Message-ID: Hi Justin, Is this earlier discussion relevant: http://erlang.org/pipermail/erlang-questions/2016-November/090780.html ________________________________ From: erlang-questions-bounces@REDACTED on behalf of Justin Wood Sent: Wednesday, 21 December 2016 11:56 AM To: erlang-questions@REDACTED Subject: [erlang-questions] SSL handshake failure Hi there, I am attempting to use the ssl module in order to get a socket to a remote server (MongoDB). I am using the following line of code (OTP 19) ssl:connect("my-server.net", 27017, [binary, {active, false}, {packet, raw}], 5000). Whenever I use this, I get the following: =ERROR REPORT==== 20-Dec-2016::19:13:13 === SSL: certify: tls_connection.erl:603:Fatal error: handshake failure - malformed_handshake {error,{tls_alert,"handshake failure"}} Which lead me to look into erlang:get_stacktrace/0 erlang:get_stacktrace(). [{tls_connection,start_fsm,8, [{file,"tls_connection.erl"},{line,79}]}, {ssl_connection,connect,8, [{file,"ssl_connection.erl"},{line,84}]}, {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,674}]}, {shell,exprs,7,[{file,"shell.erl"},{line,686}]}, {shell,eval_exprs,7,[{file,"shell.erl"},{line,641}]}, {shell,eval_loop,3,[{file,"shell.erl"},{line,626}]}] My problem is that the certificate that the server is using should be legitimate. I can connect, without error, using other clients and languages. I took a look through Wireshark in order to try and figure out what is going on and this is what I see. * The Client says hello and presents a list of cipher suites. * The Server says hello and says that it wants to use TLS_RSA_WITH_AES_256_GCM_SHA384 (This cipher is in the list that the client sends) * The Server sends the Client the certificates it is using (these both appear to be signed by DigiCert). * The Server sends an encrypted handshake message. * The Client responds with a Fatal Alert stating a Handshake Failure. * The Server sends another encrypted handshake message. I have verified that ssl:connect/4 is working fine as I can connect to a number of different miscellaneous services (including a number of other MongoDB instances). I was wondering if there is anything else I can do to try and figure out why erlang does not allow this connection. Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From schneider@REDACTED Wed Dec 21 09:56:52 2016 From: schneider@REDACTED (Schneider) Date: Wed, 21 Dec 2016 09:56:52 +0100 Subject: [erlang-questions] Access control framework and method for data versioning Message-ID: <24dc9270-1778-c363-8e27-703b51864988@xs4all.nl> Dear list, I am looking for an (Erlang based) access control framework for an application for processing medical records. The ideal framework would offer policy based access control in a distributed environment. A rule based access control system would be the next best option. Role based access and things like ACLs are not really good solutions. For the same application, I need to implement versioning of data, i.e. data should not be changed after it is recorded but superseded by a new version offering a full history of the data. Data is stored in some NoSQL database system. I would love to hear some suggestions on implementing versioning of data. /Frans From S.J.Thompson@REDACTED Wed Dec 21 15:08:18 2016 From: S.J.Thompson@REDACTED (Simon Thompson) Date: Wed, 21 Dec 2016 14:08:18 +0000 Subject: [erlang-questions] =?utf-8?b?V2UncmUgcmVjcnVpdGluZyDigKbCoHdoeSBu?= =?utf-8?q?ot_join_us=3F?= Message-ID: We?re recruiting, and if you?re interested in programming languages, types, logics, or more practical aspects like refactoring and other tools, we?d really welcome an application from you. Simon T. Lecturer in Computing (2 posts) University of Kent - School of Computing We are seeking to appoint two Lecturers in Computing (the equivalent of tenure track, assistant professors), based at our Canterbury campus. We are particularly interested in appointments in Security, or Programming Languages but are happy to consider excellent candidates in other areas that would complement or enhance the School?s existing research strengths. Job description https://www11.i-grasp.com/fe/wrapdev_SendFile.asp?key=49170934&c=336212628234&pagestamp=dbycojjivswemxeujv&index=4 Additional Information https://www11.i-grasp.com/fe/wrapdev_SendFile.asp?key=49170934&c=336212628234&pagestamp=dbycojjivswemxeujv&index=2 Please apply at https://www11.i-grasp.com/fe/tpl_kent01.asp?newms=jj&id=40606&aid=14243 Closing date for applications: 6 February 2017 The School of Computing [1] is a welcoming and diverse environment whose commitment to gender equality has been recognised with a Bronze Athena SWAN [2] award. We are keen to enhance the balanced, inclusive and diverse nature of the community within our School and would particularly encourage female candidates to apply for these posts. We are committed to delivering high quality research and education. Our research is focused on five broad research areas: Programming Languages and Systems; Computer Security; Computational Intelligence; Data Science; and Computing Education. Full details can be found at: https://www.cs.kent.ac.uk/research. The 2014 Research Excellence Framework (REF) ranked our school 12th for research intensity [3], and we are well placed to improve on this in the next REF. The University of Kent has campuses in Canterbury and Medway, and specialist postgraduate centres in Athens, Brussels, Paris and Rome. It was ranked 16th in the UK in The Guardian University Guide 2016. It ranked 4th for overall student satisfaction (NSS, 2016) and 17th for the intensity of its excellent research (REF 2014). It has 20,000 students and 1,872 academic, teaching and research staff. Overlooking the city centre, and with 125 nationalities represented, the Canterbury campus has a very cosmopolitan feel. Canterbury is a small city that retains parts of its medieval walls. Famous for its heritage (Canterbury Cathedral; Chaucer?s Tales; etc), Canterbury is a vibrant community whose culture and leisure facilities are enhanced by hosting three universities. The city and surrounding region combines an attractive and affordable environment, good schools, and fast transport links to London and mainland Europe. Links: [1] https://cs.kent.ac.uk/ [2] http://www.ecu.ac.uk/equality-charters/athena-swan/ [3] https://www.timeshighereducation.com/sites/default/files/Attachments/2014/12/30/a/b/i/subject-ranking-on-intensity.pdf Simon Thompson | Professor of Logic and Computation School of Computing | University of Kent | Canterbury, CT2 7NF, UK s.j.thompson@REDACTED | M +44 7986 085754 | W www.cs.kent.ac.uk/~sjt From S.J.Thompson@REDACTED Wed Dec 21 15:15:09 2016 From: S.J.Thompson@REDACTED (Simon Thompson) Date: Wed, 21 Dec 2016 14:15:09 +0000 Subject: [erlang-questions] =?utf-8?b?V2UncmUgcmVjcnVpdGluZyDigKbCoHdoeSBu?= =?utf-8?q?ot_join_us=3F_LINKS_FIXED?= In-Reply-To: References: Message-ID: <23949407-E355-4D5B-8260-1CB7A3CD84EC@kent.ac.uk> Apologies: please ignore the first two links, and to go for all details to: https://www11.i-grasp.com/fe/tpl_kent01.asp?newms=jj&id=40606&aid=14243 We?re recruiting, and if you?re interested in programming languages, types, logics, or more practical aspects like refactoring and other tools, we?d really welcome an application from you. Simon T. Lecturer in Computing (2 posts) University of Kent - School of Computing We are seeking to appoint two Lecturers in Computing (the equivalent of tenure track, assistant professors), based at our Canterbury campus. We are particularly interested in appointments in Security, or Programming Languages but are happy to consider excellent candidates in other areas that would complement or enhance the School?s existing research strengths. Job description Additional Information Please apply at https://www11.i-grasp.com/fe/tpl_kent01.asp?newms=jj&id=40606&aid=14243 Closing date for applications: 6 February 2017 The School of Computing [1] is a welcoming and diverse environment whose commitment to gender equality has been recognised with a Bronze Athena SWAN [2] award. We are keen to enhance the balanced, inclusive and diverse nature of the community within our School and would particularly encourage female candidates to apply for these posts. We are committed to delivering high quality research and education. Our research is focused on five broad research areas: Programming Languages and Systems; Computer Security; Computational Intelligence; Data Science; and Computing Education. Full details can be found at: https://www.cs.kent.ac.uk/research. The 2014 Research Excellence Framework (REF) ranked our school 12th for research intensity [3], and we are well placed to improve on this in the next REF. The University of Kent has campuses in Canterbury and Medway, and specialist postgraduate centres in Athens, Brussels, Paris and Rome. It was ranked 16th in the UK in The Guardian University Guide 2016. It ranked 4th for overall student satisfaction (NSS, 2016) and 17th for the intensity of its excellent research (REF 2014). It has 20,000 students and 1,872 academic, teaching and research staff. Overlooking the city centre, and with 125 nationalities represented, the Canterbury campus has a very cosmopolitan feel. Canterbury is a small city that retains parts of its medieval walls. Famous for its heritage (Canterbury Cathedral; Chaucer?s Tales; etc), Canterbury is a vibrant community whose culture and leisure facilities are enhanced by hosting three universities. The city and surrounding region combines an attractive and affordable environment, good schools, and fast transport links to London and mainland Europe. Links: [1] https://cs.kent.ac.uk/ [2] http://www.ecu.ac.uk/equality-charters/athena-swan/ [3] https://www.timeshighereducation.com/sites/default/files/Attachments/2014/12/30/a/b/i/subject-ranking-on-intensity.pdf Simon Thompson | Professor of Logic and Computation School of Computing | University of Kent | Canterbury, CT2 7NF, UK s.j.thompson@REDACTED | M +44 7986 085754 | W www.cs.kent.ac.uk/~sjt Professor Simon Thompson | Director of Innovation School of Computing | University of Kent | Canterbury, CT2 7NF, UK s.j.thompson@REDACTED | M +44 7986 085754 | W www.cs.kent.ac.uk/~sjt From frank.muller.erl@REDACTED Wed Dec 21 17:57:14 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Wed, 21 Dec 2016 16:57:14 +0000 Subject: [erlang-questions] simple_one_for_one supervisor: strange error when stopping node In-Reply-To: References: Message-ID: Spent another day fighting with this, no luck. Even after applying this trick from angner?s repo, I?m still seeing these error message: https://github.com/agner/agner/blob/master/src/agner_app.erl#L17-L17 Can someone at least (OTP guys) explain me the ? why ? i?m getting these messages please? /Frank Le mar. 20 d?c. 2016 ? 22:16, Frank Muller a ?crit : > Hi, > > I?m seeing this intermittent error when stopping my Erlang node with > init:stop(). > > 2016-12-20 22:00:11 =SUPERVISOR REPORT==== > Supervisor: {local,server_sock_sup} > Context: shutdown_error > Reason: noproc > Offender: > [{nb_children,1},{name,t3_router},{mfargs,{client_sock,start_link,[]}},{restart_type,temporary},{shutdown,5000},{child_type,worker}] > > 2016-12-20 22:00:11.069 [error] <0.162.0> Supervisor server_sock_sup had > child client_sock started with client_sock start_link() at undefined exit > with reason noproc in context shutdown_error > > The client_sock is a gen_server which traps exit. > It?s supervised by server_sock_sup as a simple_one_for_one. > > I?ve also tried without trapping exit in client_sock and was also able to > notice this error. > > Can someone please explain me what I?m doing wrong? > > /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex0player@REDACTED Wed Dec 21 18:32:27 2016 From: alex0player@REDACTED (Alex S.) Date: Wed, 21 Dec 2016 20:32:27 +0300 Subject: [erlang-questions] simple_one_for_one supervisor: strange error when stopping node In-Reply-To: References: Message-ID: <9B2652BC-8384-4D90-855C-6EBCD20CA6A3@gmail.com> The only thing I can say about this is that the trick is dated; it?s been a long time since simple_one_for_one supervisors wait for their children synchronously. The reason for the observed behaviour might happen IF the children are unlinked (or indeed, never were linked) to the parent process. Then trying to monitor some of the long-dead children reports exit reason ?noproc?. Check if the children are linked properly. > 21 ???. 2016 ?., ? 19:57, Frank Muller ???????(?): > > Spent another day fighting with this, no luck. > > Even after applying this trick from angner?s repo, I?m still seeing these error message: > https://github.com/agner/agner/blob/master/src/agner_app.erl#L17-L17 > > Can someone at least (OTP guys) explain me the ? why ? i?m getting these messages please? > > /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex0player@REDACTED Wed Dec 21 18:39:25 2016 From: alex0player@REDACTED (Alex S.) Date: Wed, 21 Dec 2016 20:39:25 +0300 Subject: [erlang-questions] simple_one_for_one supervisor: strange error when stopping node In-Reply-To: References: Message-ID: As a followup, the following insights from OTP 19?s source code: %% If a naughty child did unlink and the child dies before %% monitor the result will be that shutdown/2 receives a %% 'DOWN'-message with reason noproc. %% If the child should die after the unlink there %% will be a 'DOWN'-message with a correct reason %% that will be handled in shutdown/2. > 21 ???. 2016 ?., ? 19:57, Frank Muller ???????(?): > > Spent another day fighting with this, no luck. > > Even after applying this trick from angner?s repo, I?m still seeing these error message: > https://github.com/agner/agner/blob/master/src/agner_app.erl#L17-L17 > > Can someone at least (OTP guys) explain me the ? why ? i?m getting these messages please? > > /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Wed Dec 21 18:47:31 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Wed, 21 Dec 2016 17:47:31 +0000 Subject: [erlang-questions] simple_one_for_one supervisor: strange error when stopping node In-Reply-To: References: Message-ID: Thank you Alex. Forgot to mention that I'm using 17.5 on Linux. I'm sure all children are linked because they're started using the supervisor. I'll try to narrow that my app and double check. /Frank Le mer. 21 d?c. 2016 ? 18:39, Alex S. a ?crit : > As a followup, the following insights from OTP 19?s source code: > > %% If a naughty child did unlink and the child dies before > %% monitor the result will be that shutdown/2 receives a > %% 'DOWN'-message with reason noproc. > %% If the child should die after the unlink there > %% will be a 'DOWN'-message with a correct reason > %% that will be handled in shutdown/2. > > 21 ???. 2016 ?., ? 19:57, Frank Muller > ???????(?): > > Spent another day fighting with this, no luck. > > Even after applying this trick from angner?s repo, I?m still seeing these > error message: > https://github.com/agner/agner/blob/master/src/agner_app.erl#L17-L17 > > Can someone at least (OTP guys) explain me the ? why ? i?m getting these > messages please? > > /Frank > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fernando.benavides@REDACTED Wed Dec 21 18:52:54 2016 From: fernando.benavides@REDACTED (Brujo Benavides) Date: Wed, 21 Dec 2016 14:52:54 -0300 Subject: [erlang-questions] simple_one_for_one supervisor: strange error when stopping node In-Reply-To: References: Message-ID: <5E85238C-6BA8-4553-840A-9166CB368BBA@inakanetworks.com> Frank: Maybe it?s just wording, but to clarify just in case: Starting children from within a supervisor doesn?t guarantee they?re linked. Linking to its parent supervisor is a responsibility of the child process. In your case, if client_sock:start_link/0 doesn?t actually start and link a process to its caller, then your supervisor won?t do it for you :) Cheers! > On Dec 21, 2016, at 14:47, Frank Muller wrote: > > Thank you Alex. > > Forgot to mention that I'm using 17.5 on Linux. > I'm sure all children are linked because they're started using the supervisor. > > I'll try to narrow that my app and double check. > > /Frank > > Le mer. 21 d?c. 2016 ? 18:39, Alex S. > a ?crit : > As a followup, the following insights from OTP 19?s source code: > > %% If a naughty child did unlink and the child dies before > %% monitor the result will be that shutdown/2 receives a > %% 'DOWN'-message with reason noproc. > %% If the child should die after the unlink there > %% will be a 'DOWN'-message with a correct reason > %% that will be handled in shutdown/2. > > >> 21 ???. 2016 ?., ? 19:57, Frank Muller > ???????(?): >> > >> Spent another day fighting with this, no luck. >> >> Even after applying this trick from angner?s repo, I?m still seeing these error message: >> https://github.com/agner/agner/blob/master/src/agner_app.erl#L17-L17 >> >> Can someone at least (OTP guys) explain me the ? why ? i?m getting these messages please? >> >> /Frank > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex0player@REDACTED Wed Dec 21 18:55:08 2016 From: alex0player@REDACTED (Alex S.) Date: Wed, 21 Dec 2016 20:55:08 +0300 Subject: [erlang-questions] simple_one_for_one supervisor: strange error when stopping node In-Reply-To: References: Message-ID: <8930B8EA-D28F-41D7-9414-48C71BBE42EF@gmail.com> No problem, I ran into much the same issue some time ago. The synchronous shutdown is present since at least OTP 16, I think, either way, OTP 17 has it too. Good luck with your search! You don?t get automatic linking by starting using the supervisor, the child process is supposed to link. If you?re using gen_server start_link or proc_lib start_link, or indeed bare spawn_link, it is done, on start. Otherwise, no dice. > 21 ???. 2016 ?., ? 20:47, Frank Muller ???????(?): > > Thank you Alex. > > Forgot to mention that I'm using 17.5 on Linux. > I'm sure all children are linked because they're started using the supervisor. > > I'll try to narrow that my app and double check. > > /Frank From frank.muller.erl@REDACTED Wed Dec 21 18:55:23 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Wed, 21 Dec 2016 17:55:23 +0000 Subject: [erlang-questions] simple_one_for_one supervisor: strange error when stopping node In-Reply-To: <5E85238C-6BA8-4553-840A-9166CB368BBA@inakanetworks.com> References: <5E85238C-6BA8-4553-840A-9166CB368BBA@inakanetworks.com> Message-ID: Hmm ... I see. Let me triple check then. Thanks Brujo. /Frank Le mer. 21 d?c. 2016 ? 18:52, Brujo Benavides < fernando.benavides@REDACTED> a ?crit : > Frank: > > Maybe it?s just wording, but to clarify just in case: > Starting children from within a supervisor doesn?t guarantee they?re > linked. > Linking to its parent supervisor is a responsibility of the child process. > In your case, if *client_sock:start_link/0* doesn?t actually start and > link a process to its caller*, *then your supervisor won?t do it for you > :) > > Cheers! > > On Dec 21, 2016, at 14:47, Frank Muller > wrote: > > Thank you Alex. > > Forgot to mention that I'm using 17.5 on Linux. > I'm sure all children are linked because they're started using the > supervisor. > > I'll try to narrow that my app and double check. > > /Frank > > Le mer. 21 d?c. 2016 ? 18:39, Alex S. a ?crit : > > As a followup, the following insights from OTP 19?s source code: > > %% If a naughty child did unlink and the child dies before > %% monitor the result will be that shutdown/2 receives a > %% 'DOWN'-message with reason noproc. > %% If the child should die after the unlink there > %% will be a 'DOWN'-message with a correct reason > %% that will be handled in shutdown/2. > > 21 ???. 2016 ?., ? 19:57, Frank Muller > ???????(?): > > Spent another day fighting with this, no luck. > > Even after applying this trick from angner?s repo, I?m still seeing these > error message: > https://github.com/agner/agner/blob/master/src/agner_app.erl#L17-L17 > > Can someone at least (OTP guys) explain me the ? why ? i?m getting these > messages please? > > /Frank > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From justin.k.wood@REDACTED Wed Dec 21 14:47:08 2016 From: justin.k.wood@REDACTED (Justin Wood) Date: Wed, 21 Dec 2016 08:47:08 -0500 Subject: [erlang-questions] SSL handshake failure In-Reply-To: References: Message-ID: After upgrading to Erlang OTP-19.2 and, in different requests, specified the version as one of tlsv1, tlsv1.1 or tlsv1.2. Every time I received the same error. I'm not sure if it is helpful at all, but I can replicate this on Windows, OSX 10.11 (El Capitan) and Ubuntu 14.04. I have not tried on other platforms. Is there any other information that I would be able to give that would be helpful? On Wed, Dec 21, 2016 at 3:38 AM, Technion wrote: > Hi Justin, > > > Is this earlier discussion relevant: > > > http://erlang.org/pipermail/erlang-questions/2016-November/090780.html > > > > > > ------------------------------ > *From:* erlang-questions-bounces@REDACTED erlang.org> on behalf of Justin Wood > *Sent:* Wednesday, 21 December 2016 11:56 AM > *To:* erlang-questions@REDACTED > *Subject:* [erlang-questions] SSL handshake failure > > Hi there, > > I am attempting to use the ssl module in order to get a socket to a remote > server (MongoDB). I am using the following line of code (OTP 19) > > ssl:connect("my-server.net", 27017, [binary, {active, false}, {packet, > raw}], 5000). > > Whenever I use this, I get the following: > > =ERROR REPORT==== 20-Dec-2016::19:13:13 === > SSL: certify: tls_connection.erl:603:Fatal error: handshake failure - > malformed_handshake > {error,{tls_alert,"handshake failure"}} > > Which lead me to look into erlang:get_stacktrace/0 > > erlang:get_stacktrace(). > [{tls_connection,start_fsm,8, > [{file,"tls_connection.erl"},{line,79}]}, > {ssl_connection,connect,8, > [{file,"ssl_connection.erl"},{line,84}]}, > {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,674}]}, > {shell,exprs,7,[{file,"shell.erl"},{line,686}]}, > {shell,eval_exprs,7,[{file,"shell.erl"},{line,641}]}, > {shell,eval_loop,3,[{file,"shell.erl"},{line,626}]}] > > My problem is that the certificate that the server is using should be > legitimate. I can connect, without error, using other clients and languages. > > I took a look through Wireshark in order to try and figure out what is > going on and this is what I see. > > * The Client says hello and presents a list of cipher suites. > * The Server says hello and says that it wants to use > TLS_RSA_WITH_AES_256_GCM_SHA384 (This cipher is in the list that the > client sends) > * The Server sends the Client the certificates it is using (these both > appear to be signed by DigiCert). > * The Server sends an encrypted handshake message. > * The Client responds with a Fatal Alert stating a Handshake Failure. > * The Server sends another encrypted handshake message. > > I have verified that ssl:connect/4 is working fine as I can connect to a > number of different miscellaneous services (including a number of other > MongoDB instances). > > I was wondering if there is anything else I can do to try and figure out > why erlang does not allow this connection. > > Justin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From evnu@REDACTED Wed Dec 21 22:06:43 2016 From: evnu@REDACTED (Magnus Ottenklinger) Date: Wed, 21 Dec 2016 22:06:43 +0100 Subject: [erlang-questions] Access control framework and method for data versioning In-Reply-To: <24dc9270-1778-c363-8e27-703b51864988@xs4all.nl> References: <24dc9270-1778-c363-8e27-703b51864988@xs4all.nl> Message-ID: <148235440300.2077.15346649390204227656@central-dogma.fritz.box> > For the same application, I need to implement versioning of data, i.e. > data should not be changed after it is recorded but superseded by a new > version offering a full history of the data. Data is stored in some > NoSQL database system. > I would love to hear some suggestions on implementing versioning of data. Some time ago, I did a lot of thinking on how I would implement something like this. A friend and I prototyped something like that over a day by using git as a backing storage. The pull/push model of git does not fit the problem perfectly, but storing documents in git is very straight forward to get started. On top of that, you only need a simple PUT/GET API (no other calls, as nothing committed to storage may ever change) and a call to traverse the tree. For calling git, we called the git executable directly after taking a look into erlgit[1], which offered more than we actually needed for the prototype. Now, to make it completely distributed, a simple KV-store seems very appropriate, although storing the meta data in an efficient manner comes down to maintaining your own secondary index (you want to represent a tree of objects, after all). Additionally, concurrent access to a shared tree representation is not trivial without support for transactions. As I did not find enough time to really implement this system, I also did not take the time to research into other fitting storage mechanisms. Maybe a graph database would be the simplest approach to implement this. Magnus [1] https://github.com/gleber/erlgit From ok@REDACTED Thu Dec 22 01:13:53 2016 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 22 Dec 2016 13:13:53 +1300 Subject: [erlang-questions] Remove behaviour checking from erl_lint (continued) In-Reply-To: <5858ED77.4040009@gmail.com> References: <5858568C.6010306@gmail.com> <19643ace-bc11-4329-aa97-6fa8e0aa0a27@Spark> <5858ED77.4040009@gmail.com> Message-ID: <179d6f7b-86df-7b0e-be8f-00ab9103868b@cs.otago.ac.nz> Let me offer another perspective. Once you add -type and -spec declarations to your Erlang code, it becomes ASTONISHING that your code is not in fact type checked by the compiler. There is no other programming language that I use or have ever used in which source code *with* type information isn't checked by the normal compiler, to the extent that it *can* be checked locally. It is fatally easy to think that because you took care to write -type and -spec and the compiler is silent that your code is type correct, when it isn't. Long term, we should not be shunting off ever more checking from the compiler to the dialyzer; on the contrary, the dialyzer as a separate tool should disappear. There should still be some compiler option to say how *much* checking is to be done, but the default should be "I'll check everything I can within the limits of what you've given me". From mjtruog@REDACTED Thu Dec 22 01:51:20 2016 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 21 Dec 2016 16:51:20 -0800 Subject: [erlang-questions] Remove behaviour checking from erl_lint (continued) In-Reply-To: <179d6f7b-86df-7b0e-be8f-00ab9103868b@cs.otago.ac.nz> References: <5858568C.6010306@gmail.com> <19643ace-bc11-4329-aa97-6fa8e0aa0a27@Spark> <5858ED77.4040009@gmail.com> <179d6f7b-86df-7b0e-be8f-00ab9103868b@cs.otago.ac.nz> Message-ID: <585B2388.6060102@gmail.com> On 12/21/2016 04:13 PM, Richard A. O'Keefe wrote: > Let me offer another perspective. > > Once you add -type and -spec declarations to your Erlang code, > it becomes ASTONISHING that your code is not in fact type checked > by the compiler. > > There is no other programming language that I use or have ever > used in which source code *with* type information isn't checked > by the normal compiler, to the extent that it *can* be checked > locally. > > It is fatally easy to think that because you took care > to write -type and -spec and the compiler is silent that > your code is type correct, when it isn't. > > Long term, we should not be shunting off ever more checking > from the compiler to the dialyzer; on the contrary, the > dialyzer as a separate tool should disappear. There should > still be some compiler option to say how *much* checking is > to be done, but the default should be "I'll check everything > I can within the limits of what you've given me". > It doesn't seem like what you have said is as specific as it could be. I agree it would be nice to have the compiler to type checking on the single module it is processing. However, are you suggesting that the Erlang compiler that is fed a single module should know about all other modules that will be used along with it, at the same time, and effectively run dialyzer on the single module? Since, if so, that should inherit the speed and memory problems dialyzer has, only they would be multiplied by the number of modules being processed due to the overhead involved. That would be a way of avoiding the need for dialyzer though. From kuna.prime@REDACTED Thu Dec 22 01:54:15 2016 From: kuna.prime@REDACTED (Karlo Kuna) Date: Thu, 22 Dec 2016 01:54:15 +0100 Subject: [erlang-questions] Remove behaviour checking from erl_lint (continued) In-Reply-To: <179d6f7b-86df-7b0e-be8f-00ab9103868b@cs.otago.ac.nz> References: <5858568C.6010306@gmail.com> <19643ace-bc11-4329-aa97-6fa8e0aa0a27@Spark> <5858ED77.4040009@gmail.com> <179d6f7b-86df-7b0e-be8f-00ab9103868b@cs.otago.ac.nz> Message-ID: my opinion, if intent is to simplify components in tool chain such as compiler -> xref -> dialyzer ..... or some variation then i agree that compiler should not check for any external deps. But in that case there should be added some standard tool to run that tool chain, because users don't want to spend much time on configuring things like that, and it should be default practice to run that tool instead just compiler. And if you have some special needs to skip some steps or just use single step, than you still have all this discreet tools available to use them as you want. But for that Dialyzer in my opinion should be heavily optimized. in a sense i see need for clear separation of tasks and responsibilities but on other hand i want default use case to be as simple as possible and also as strict as possible. On Thu, Dec 22, 2016 at 1:13 AM, Richard A. O'Keefe wrote: > Let me offer another perspective. > > Once you add -type and -spec declarations to your Erlang code, > it becomes ASTONISHING that your code is not in fact type checked > by the compiler. > > There is no other programming language that I use or have ever > used in which source code *with* type information isn't checked > by the normal compiler, to the extent that it *can* be checked > locally. > > It is fatally easy to think that because you took care > to write -type and -spec and the compiler is silent that > your code is type correct, when it isn't. > > Long term, we should not be shunting off ever more checking > from the compiler to the dialyzer; on the contrary, the > dialyzer as a separate tool should disappear. There should > still be some compiler option to say how *much* checking is > to be done, but the default should be "I'll check everything > I can within the limits of what you've given me". > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Thu Dec 22 02:02:58 2016 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 22 Dec 2016 14:02:58 +1300 Subject: [erlang-questions] Remove behaviour checking from erl_lint (continued) In-Reply-To: <585B2388.6060102@gmail.com> References: <5858568C.6010306@gmail.com> <19643ace-bc11-4329-aa97-6fa8e0aa0a27@Spark> <5858ED77.4040009@gmail.com> <179d6f7b-86df-7b0e-be8f-00ab9103868b@cs.otago.ac.nz> <585B2388.6060102@gmail.com> Message-ID: On 22/12/16 1:51 PM, Michael Truog wrote: >> There is no other programming language that I use or have ever >> used in which source code *with* type information isn't checked >> by the normal compiler, to the extent that it *can* be checked >> locally. > It doesn't seem like what you have said is as specific as it could be. It was a perspective, not a program! > I agree it would be nice to have the compiler to type checking on the > single module it is processing. However, are you suggesting that the > Erlang compiler that is fed a single module should know about all other > modules that will be used along with it, at the same time, and > effectively run dialyzer on the single module? I was in fact specific about that: "to the extent that it *can* be checked locally". So no, I obviously *wasn't* suggesting that. To be a bit more specific: - use the PLT to get information about library modules - assume that all calls to this module from outside are correct - assume that all calls from this module to outside code not described in the PLT are correct - check only that. - on request show a list of external functions called but not checked, mainly to avoid the illusion that everything was checked. More generally, there should be a way to compile N modules together, allowing calls between them to be checked but not calls outside that group. I find that the majority of my type errors are internal. From ok@REDACTED Thu Dec 22 02:07:59 2016 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 22 Dec 2016 14:07:59 +1300 Subject: [erlang-questions] Remove behaviour checking from erl_lint (continued) In-Reply-To: References: <5858568C.6010306@gmail.com> <19643ace-bc11-4329-aa97-6fa8e0aa0a27@Spark> <5858ED77.4040009@gmail.com> <179d6f7b-86df-7b0e-be8f-00ab9103868b@cs.otago.ac.nz> <585B2388.6060102@gmail.com> Message-ID: <53a98b60-d4cb-ce56-8583-40edce4a1c07@cs.otago.ac.nz> On 22/12/16 2:02 PM, Richard A. O'Keefe wrote: > I find that the majority of my type errors are internal. There's a theory about why this is so: + when I call a library function, I look it up. - when I call module-internal functions, there might not be anything to look up yet. In fact I might be writing the code *in order to find out* what the interface should be. + library interfaces are stable. - whenever I refactor module-internal code, there's a risk of an incomplete edit, so that code that *was* correct isn't any more. From ingela.andin@REDACTED Thu Dec 22 12:56:06 2016 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 22 Dec 2016 12:56:06 +0100 Subject: [erlang-questions] SSL handshake failure In-Reply-To: References: Message-ID: Hi! I think this might be a timing bug. Can you try the following patch? Regards Ingela Erlang/OTP Team - Ericsson AB *diff --git a/lib/ssl/src/tls_connection.erl b/lib/ssl/src/tls_connection.erl* *index 32991d3..e1f8e78 100644* *--- a/lib/ssl/src/tls_connection.erl* *+++ b/lib/ssl/src/tls_connection.erl* @@ -420,6 +420,7 @@ handle_common_event(internal, #alert{} = Alert, StateName, handle_common_event(internal, #ssl_tls{type = ?HANDSHAKE, fragment = Data}, StateName, #state{protocol_buffers = #protocol_buffers{tls_handshake_buffer = Buf0} = Buffers, + unprocessed_handshake_events = NumUnprocessed, negotiated_version = Version, ssl_options = Options} = State0) -> try @@ -432,7 +433,7 @@ handle_common_event(internal, #ssl_tls{type = ?HANDSHAKE, fragment = Data}, connection -> ssl_connection:hibernate_after(StateName, State, Events); _ -> - {next_state, StateName, State#state{unprocessed_handshake_events = unprocessed_events(Events)}, Events} + {next_state, StateName, State#state{unprocessed_handshake_events = unprocessed_events(Events, NumUnprocessed)}, Events} end catch throw:#alert{} = Alert -> ssl_connection:handle_own_alert(Alert, Version, StateName, State0) @@ -615,8 +616,6 @@ next_event(StateName, Record, State, Actions) -> {next_state, StateName, State, [{next_event, internal, Alert} | Actions]} end. -tls_handshake_events([]) -> - throw(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, malformed_handshake)); tls_handshake_events(Packets) -> lists:map(fun(Packet) -> {next_event, internal, {handshake, Packet}} @@ -727,8 +726,10 @@ gen_info(Event, StateName, #state{negotiated_version = Version} = State) -> malformed_handshake_data), Version, StateName, State) end. - -unprocessed_events(Events) -> + +unprocessed_events([], Unprocessed) -> + Unprocessed; +unprocessed_events(Events, _) -> %% The first handshake event will be processed immediately %% as it is entered first in the event queue and %% when it is processed there will be length(Events)-1 2016-12-21 14:47 GMT+01:00 Justin Wood : > After upgrading to Erlang OTP-19.2 and, in different requests, specified > the version as one of tlsv1, tlsv1.1 or tlsv1.2. Every time I received the > same error. > > I'm not sure if it is helpful at all, but I can replicate this on Windows, > OSX 10.11 (El Capitan) and Ubuntu 14.04. I have not tried on other > platforms. > > Is there any other information that I would be able to give that would be > helpful? > > On Wed, Dec 21, 2016 at 3:38 AM, Technion wrote: > >> Hi Justin, >> >> >> Is this earlier discussion relevant: >> >> >> http://erlang.org/pipermail/erlang-questions/2016-November/090780.html >> >> >> >> >> >> ------------------------------ >> *From:* erlang-questions-bounces@REDACTED < >> erlang-questions-bounces@REDACTED> on behalf of Justin Wood < >> justin.k.wood@REDACTED> >> *Sent:* Wednesday, 21 December 2016 11:56 AM >> *To:* erlang-questions@REDACTED >> *Subject:* [erlang-questions] SSL handshake failure >> >> Hi there, >> >> I am attempting to use the ssl module in order to get a socket to a >> remote server (MongoDB). I am using the following line of code (OTP 19) >> >> ssl:connect("my-server.net", 27017, [binary, {active, false}, {packet, >> raw}], 5000). >> >> Whenever I use this, I get the following: >> >> =ERROR REPORT==== 20-Dec-2016::19:13:13 === >> SSL: certify: tls_connection.erl:603:Fatal error: handshake failure - >> malformed_handshake >> {error,{tls_alert,"handshake failure"}} >> >> Which lead me to look into erlang:get_stacktrace/0 >> >> erlang:get_stacktrace(). >> [{tls_connection,start_fsm,8, >> [{file,"tls_connection.erl"},{line,79}]}, >> {ssl_connection,connect,8, >> [{file,"ssl_connection.erl"},{line,84}]}, >> {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,674}]}, >> {shell,exprs,7,[{file,"shell.erl"},{line,686}]}, >> {shell,eval_exprs,7,[{file,"shell.erl"},{line,641}]}, >> {shell,eval_loop,3,[{file,"shell.erl"},{line,626}]}] >> >> My problem is that the certificate that the server is using should be >> legitimate. I can connect, without error, using other clients and languages. >> >> I took a look through Wireshark in order to try and figure out what is >> going on and this is what I see. >> >> * The Client says hello and presents a list of cipher suites. >> * The Server says hello and says that it wants to use >> TLS_RSA_WITH_AES_256_GCM_SHA384 (This cipher is in the list that the >> client sends) >> * The Server sends the Client the certificates it is using (these both >> appear to be signed by DigiCert). >> * The Server sends an encrypted handshake message. >> * The Client responds with a Fatal Alert stating a Handshake Failure. >> * The Server sends another encrypted handshake message. >> >> I have verified that ssl:connect/4 is working fine as I can connect to a >> number of different miscellaneous services (including a number of other >> MongoDB instances). >> >> I was wondering if there is anything else I can do to try and figure out >> why erlang does not allow this connection. >> >> Justin >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From schneider@REDACTED Thu Dec 22 14:40:12 2016 From: schneider@REDACTED (Schneider) Date: Thu, 22 Dec 2016 14:40:12 +0100 Subject: [erlang-questions] Access control framework and method for data versioning In-Reply-To: <148235440300.2077.15346649390204227656@central-dogma.fritz.box> References: <24dc9270-1778-c363-8e27-703b51864988@xs4all.nl> <148235440300.2077.15346649390204227656@central-dogma.fritz.box> Message-ID: On 12/21/2016 10:06 PM, Magnus Ottenklinger wrote: >> For the same application, I need to implement versioning of data, i.e. >> data should not be changed after it is recorded but superseded by a new >> version offering a full history of the data. Data is stored in some >> NoSQL database system. >> I would love to hear some suggestions on implementing versioning of data. > Some time ago, I did a lot of thinking on how I would implement something like this. A > friend and I prototyped something like that over a day by using git as a backing storage. > The pull/push model of git does not fit the problem perfectly, but storing documents in > git is very straight forward to get started. On top of that, you only need a simple > PUT/GET API (no other calls, as nothing committed to storage may ever change) and a call > to traverse the tree. For calling git, we called the git executable directly after taking > a look into erlgit[1], which offered more than we actually needed for the prototype. > > Now, to make it completely distributed, a simple KV-store seems very appropriate, although > storing the meta data in an efficient manner comes down to maintaining your own secondary > index (you want to represent a tree of objects, after all). Additionally, concurrent > access to a shared tree representation is not trivial without support for transactions. > > As I did not find enough time to really implement this system, I also did not take the > time to research into other fitting storage mechanisms. Maybe a graph database would be > the simplest approach to implement this. > > Magnus > > [1] https://github.com/gleber/erlgit Thanks Magnus, I will investigate the git-like approach. Might solve the issue of having data forked onto disjoint systems and merged back later as well. Is anybody familiar with using XACML, Argus [1] or PolicyMachine [2] for access control? Frans [1] https://argus-documentation.readthedocs.io/en/latest/index.html [2] http://csrc.nist.gov/pm/ From donpedrothird@REDACTED Thu Dec 22 15:22:06 2016 From: donpedrothird@REDACTED (John Doe) Date: Thu, 22 Dec 2016 17:22:06 +0300 Subject: [erlang-questions] getting application name in parse transform Message-ID: Hi, Is there a way to get in a parse transform the name of the application the module belongs to? The parse_transform function has parameter Options so it is possible to add an argument to erlc with app name, but is it possible to get the name of the app automatically? I tried {attribute, _, module, ModuleName} = lists:keyfind(module, 3, Forms), application:get_application(ModuleName), but it returns undefined -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Thu Dec 22 16:44:10 2016 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 22 Dec 2016 16:44:10 +0100 Subject: [erlang-questions] SSL handshake failure In-Reply-To: References: Message-ID: Hi! Well maybe I did not understand the problem correctly then, I thought I had found a potential problem but as I have nothing to test it against. Can you provide me with a site to test against or maybe use tracing to give me more info. dbg:tracer(). dbg:p(all, [call]). dbg:tpl(tls_handshake, get_tls_handshake, x). Could be a good way to start.rds Regards Ingela 2016-12-22 16:29 GMT+01:00 Justin Wood : > Thank you for the patch Ingela. Unfortunately it did not fix my issue. > > If there is anything else I can provide to help figure this it, just let > me know. > > On Thu, Dec 22, 2016 at 6:56 AM, Ingela Andin > wrote: > >> Hi! >> >> I think this might be a timing bug. Can you try the following patch? >> >> Regards Ingela Erlang/OTP Team - Ericsson AB >> >> *diff --git a/lib/ssl/src/tls_connection.erl >> b/lib/ssl/src/tls_connection.erl* >> >> *index 32991d3..e1f8e78 100644* >> >> *--- a/lib/ssl/src/tls_connection.erl* >> >> *+++ b/lib/ssl/src/tls_connection.erl* >> >> @@ -420,6 +420,7 @@ handle_common_event(internal, #alert{} = Alert, >> StateName, >> >> handle_common_event(internal, #ssl_tls{type = ?HANDSHAKE, fragment = >> Data}, >> >> StateName, #state{protocol_buffers = >> >> #protocol_buffers{tls_handshake_buffer >> = Buf0} = Buffers, >> >> + unprocessed_handshake_events = >> NumUnprocessed, >> >> negotiated_version = Version, >> >> ssl_options = Options} = State0) -> >> >> try >> >> @@ -432,7 +433,7 @@ handle_common_event(internal, #ssl_tls{type = >> ?HANDSHAKE, fragment = Data}, >> >> connection -> >> >> ssl_connection:hibernate_after(StateName, State, Events); >> >> _ -> >> >> - {next_state, StateName, State#state{unprocessed_handshake_events >> = unprocessed_events(Events)}, Events} >> >> + {next_state, StateName, State#state{unprocessed_handshake_events >> = unprocessed_events(Events, NumUnprocessed)}, Events} >> >> end >> >> catch throw:#alert{} = Alert -> >> >> ssl_connection:handle_own_alert(Alert, Version, StateName, >> State0) >> >> @@ -615,8 +616,6 @@ next_event(StateName, Record, State, Actions) -> >> >> {next_state, StateName, State, [{next_event, internal, Alert} >> | Actions]} >> >> end. >> >> >> >> -tls_handshake_events([]) -> >> >> - throw(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, malformed_handshake)); >> >> tls_handshake_events(Packets) -> >> >> lists:map(fun(Packet) -> >> >> {next_event, internal, {handshake, Packet}} >> >> @@ -727,8 +726,10 @@ gen_info(Event, StateName, >> #state{negotiated_version = Version} = State) -> >> >> >> malformed_handshake_data), >> >> Version, StateName, State) >> >> end. >> >> - >> >> -unprocessed_events(Events) -> >> >> + >> >> +unprocessed_events([], Unprocessed) -> >> >> + Unprocessed; >> >> +unprocessed_events(Events, _) -> >> >> %% The first handshake event will be processed immediately >> >> %% as it is entered first in the event queue and >> >> %% when it is processed there will be length(Events)-1 >> >> >> >> >> 2016-12-21 14:47 GMT+01:00 Justin Wood : >> >>> After upgrading to Erlang OTP-19.2 and, in different requests, specified >>> the version as one of tlsv1, tlsv1.1 or tlsv1.2. Every time I received the >>> same error. >>> >>> I'm not sure if it is helpful at all, but I can replicate this on >>> Windows, OSX 10.11 (El Capitan) and Ubuntu 14.04. I have not tried on other >>> platforms. >>> >>> Is there any other information that I would be able to give that would >>> be helpful? >>> >>> On Wed, Dec 21, 2016 at 3:38 AM, Technion wrote: >>> >>>> Hi Justin, >>>> >>>> >>>> Is this earlier discussion relevant: >>>> >>>> >>>> http://erlang.org/pipermail/erlang-questions/2016-November/090780.html >>>> >>>> >>>> >>>> >>>> >>>> ------------------------------ >>>> *From:* erlang-questions-bounces@REDACTED < >>>> erlang-questions-bounces@REDACTED> on behalf of Justin Wood < >>>> justin.k.wood@REDACTED> >>>> *Sent:* Wednesday, 21 December 2016 11:56 AM >>>> *To:* erlang-questions@REDACTED >>>> *Subject:* [erlang-questions] SSL handshake failure >>>> >>>> Hi there, >>>> >>>> I am attempting to use the ssl module in order to get a socket to a >>>> remote server (MongoDB). I am using the following line of code (OTP 19) >>>> >>>> ssl:connect("my-server.net", 27017, [binary, {active, false}, {packet, >>>> raw}], 5000). >>>> >>>> Whenever I use this, I get the following: >>>> >>>> =ERROR REPORT==== 20-Dec-2016::19:13:13 === >>>> SSL: certify: tls_connection.erl:603:Fatal error: handshake failure - >>>> malformed_handshake >>>> {error,{tls_alert,"handshake failure"}} >>>> >>>> Which lead me to look into erlang:get_stacktrace/0 >>>> >>>> erlang:get_stacktrace(). >>>> [{tls_connection,start_fsm,8, >>>> [{file,"tls_connection.erl"},{line,79}]}, >>>> {ssl_connection,connect,8, >>>> [{file,"ssl_connection.erl"},{line,84}]}, >>>> {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,674}]}, >>>> {shell,exprs,7,[{file,"shell.erl"},{line,686}]}, >>>> {shell,eval_exprs,7,[{file,"shell.erl"},{line,641}]}, >>>> {shell,eval_loop,3,[{file,"shell.erl"},{line,626}]}] >>>> >>>> My problem is that the certificate that the server is using should be >>>> legitimate. I can connect, without error, using other clients and languages. >>>> >>>> I took a look through Wireshark in order to try and figure out what is >>>> going on and this is what I see. >>>> >>>> * The Client says hello and presents a list of cipher suites. >>>> * The Server says hello and says that it wants to use >>>> TLS_RSA_WITH_AES_256_GCM_SHA384 (This cipher is in the list that the >>>> client sends) >>>> * The Server sends the Client the certificates it is using (these both >>>> appear to be signed by DigiCert). >>>> * The Server sends an encrypted handshake message. >>>> * The Client responds with a Fatal Alert stating a Handshake Failure. >>>> * The Server sends another encrypted handshake message. >>>> >>>> I have verified that ssl:connect/4 is working fine as I can connect to >>>> a number of different miscellaneous services (including a number of other >>>> MongoDB instances). >>>> >>>> I was wondering if there is anything else I can do to try and figure >>>> out why erlang does not allow this connection. >>>> >>>> Justin >>>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewdpb@REDACTED Thu Dec 22 16:16:50 2016 From: ewdpb@REDACTED (=?UTF-8?Q?Wilmar_P=C3=A9rez?=) Date: Thu, 22 Dec 2016 10:16:50 -0500 Subject: [erlang-questions] OpenFlow Framework Message-ID: Hello I have been tasked with finding an efficient and hopefully mature OpenFlow Controller. I like the idea of an Erlang OpenFlow controller but it seems like all projects are abandoned. Is there any that is still being actively developed? Thanks Wilmar -------------- next part -------------- An HTML attachment was scrubbed... URL: From justin.k.wood@REDACTED Thu Dec 22 16:29:11 2016 From: justin.k.wood@REDACTED (Justin Wood) Date: Thu, 22 Dec 2016 10:29:11 -0500 Subject: [erlang-questions] SSL handshake failure In-Reply-To: References: Message-ID: Thank you for the patch Ingela. Unfortunately it did not fix my issue. If there is anything else I can provide to help figure this it, just let me know. On Thu, Dec 22, 2016 at 6:56 AM, Ingela Andin wrote: > Hi! > > I think this might be a timing bug. Can you try the following patch? > > Regards Ingela Erlang/OTP Team - Ericsson AB > > *diff --git a/lib/ssl/src/tls_connection.erl > b/lib/ssl/src/tls_connection.erl* > > *index 32991d3..e1f8e78 100644* > > *--- a/lib/ssl/src/tls_connection.erl* > > *+++ b/lib/ssl/src/tls_connection.erl* > > @@ -420,6 +420,7 @@ handle_common_event(internal, #alert{} = Alert, > StateName, > > handle_common_event(internal, #ssl_tls{type = ?HANDSHAKE, fragment = > Data}, > > StateName, #state{protocol_buffers = > > #protocol_buffers{tls_handshake_buffer > = Buf0} = Buffers, > > + unprocessed_handshake_events = > NumUnprocessed, > > negotiated_version = Version, > > ssl_options = Options} = State0) -> > > try > > @@ -432,7 +433,7 @@ handle_common_event(internal, #ssl_tls{type = > ?HANDSHAKE, fragment = Data}, > > connection -> > > ssl_connection:hibernate_after(StateName, State, Events); > > _ -> > > - {next_state, StateName, State#state{unprocessed_handshake_events > = unprocessed_events(Events)}, Events} > > + {next_state, StateName, State#state{unprocessed_handshake_events > = unprocessed_events(Events, NumUnprocessed)}, Events} > > end > > catch throw:#alert{} = Alert -> > > ssl_connection:handle_own_alert(Alert, Version, StateName, > State0) > > @@ -615,8 +616,6 @@ next_event(StateName, Record, State, Actions) -> > > {next_state, StateName, State, [{next_event, internal, Alert} > | Actions]} > > end. > > > > -tls_handshake_events([]) -> > > - throw(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, malformed_handshake)); > > tls_handshake_events(Packets) -> > > lists:map(fun(Packet) -> > > {next_event, internal, {handshake, Packet}} > > @@ -727,8 +726,10 @@ gen_info(Event, StateName, #state{negotiated_version > = Version} = State) -> > > > malformed_handshake_data), > > Version, StateName, State) > > end. > > - > > -unprocessed_events(Events) -> > > + > > +unprocessed_events([], Unprocessed) -> > > + Unprocessed; > > +unprocessed_events(Events, _) -> > > %% The first handshake event will be processed immediately > > %% as it is entered first in the event queue and > > %% when it is processed there will be length(Events)-1 > > > > > 2016-12-21 14:47 GMT+01:00 Justin Wood : > >> After upgrading to Erlang OTP-19.2 and, in different requests, specified >> the version as one of tlsv1, tlsv1.1 or tlsv1.2. Every time I received the >> same error. >> >> I'm not sure if it is helpful at all, but I can replicate this on >> Windows, OSX 10.11 (El Capitan) and Ubuntu 14.04. I have not tried on other >> platforms. >> >> Is there any other information that I would be able to give that would be >> helpful? >> >> On Wed, Dec 21, 2016 at 3:38 AM, Technion wrote: >> >>> Hi Justin, >>> >>> >>> Is this earlier discussion relevant: >>> >>> >>> http://erlang.org/pipermail/erlang-questions/2016-November/090780.html >>> >>> >>> >>> >>> >>> ------------------------------ >>> *From:* erlang-questions-bounces@REDACTED < >>> erlang-questions-bounces@REDACTED> on behalf of Justin Wood < >>> justin.k.wood@REDACTED> >>> *Sent:* Wednesday, 21 December 2016 11:56 AM >>> *To:* erlang-questions@REDACTED >>> *Subject:* [erlang-questions] SSL handshake failure >>> >>> Hi there, >>> >>> I am attempting to use the ssl module in order to get a socket to a >>> remote server (MongoDB). I am using the following line of code (OTP 19) >>> >>> ssl:connect("my-server.net", 27017, [binary, {active, false}, {packet, >>> raw}], 5000). >>> >>> Whenever I use this, I get the following: >>> >>> =ERROR REPORT==== 20-Dec-2016::19:13:13 === >>> SSL: certify: tls_connection.erl:603:Fatal error: handshake failure - >>> malformed_handshake >>> {error,{tls_alert,"handshake failure"}} >>> >>> Which lead me to look into erlang:get_stacktrace/0 >>> >>> erlang:get_stacktrace(). >>> [{tls_connection,start_fsm,8, >>> [{file,"tls_connection.erl"},{line,79}]}, >>> {ssl_connection,connect,8, >>> [{file,"ssl_connection.erl"},{line,84}]}, >>> {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,674}]}, >>> {shell,exprs,7,[{file,"shell.erl"},{line,686}]}, >>> {shell,eval_exprs,7,[{file,"shell.erl"},{line,641}]}, >>> {shell,eval_loop,3,[{file,"shell.erl"},{line,626}]}] >>> >>> My problem is that the certificate that the server is using should be >>> legitimate. I can connect, without error, using other clients and languages. >>> >>> I took a look through Wireshark in order to try and figure out what is >>> going on and this is what I see. >>> >>> * The Client says hello and presents a list of cipher suites. >>> * The Server says hello and says that it wants to use >>> TLS_RSA_WITH_AES_256_GCM_SHA384 (This cipher is in the list that the >>> client sends) >>> * The Server sends the Client the certificates it is using (these both >>> appear to be signed by DigiCert). >>> * The Server sends an encrypted handshake message. >>> * The Client responds with a Fatal Alert stating a Handshake Failure. >>> * The Server sends another encrypted handshake message. >>> >>> I have verified that ssl:connect/4 is working fine as I can connect to a >>> number of different miscellaneous services (including a number of other >>> MongoDB instances). >>> >>> I was wondering if there is anything else I can do to try and figure out >>> why erlang does not allow this connection. >>> >>> Justin >>> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex0player@REDACTED Thu Dec 22 17:17:43 2016 From: alex0player@REDACTED (Alex S.) Date: Thu, 22 Dec 2016 19:17:43 +0300 Subject: [erlang-questions] getting application name in parse transform In-Reply-To: References: Message-ID: No wonder it does, too: get_application does its thing for compiled and loaded modules. 22 ???. 2016 ?. 17:22 ???????????? "John Doe" ???????: > Hi, > Is there a way to get in a parse transform the name of the application the > module belongs to? The parse_transform function has parameter Options so it > is possible to add an argument to erlc with app name, but is it possible to > get the name of the app automatically? > > I tried > {attribute, _, module, ModuleName} = lists:keyfind(module, 3, Forms), > application:get_application(ModuleName), > but it returns undefined > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacob01@REDACTED Thu Dec 22 20:49:08 2016 From: jacob01@REDACTED (Jacob) Date: Thu, 22 Dec 2016 20:49:08 +0100 Subject: [erlang-questions] getting application name in parse transform In-Reply-To: References: Message-ID: On 22.12.2016 17:17, Alex S. wrote: > No wonder it does, too: get_application does its thing for compiled and > loaded modules. The _application_ has to be loaded, the information is taken from the application_controller's ETS table (which in general gets its information from the app files). The module itself does not have to be loaded. So it depends on which modules are loaded by the _compiling_ node. HTH Jacob > > 22 ???. 2016 ?. 17:22 ???????????? "John Doe" > ???????: > > Hi, > Is there a way to get in a parse transform the name of the > application the module belongs to? The parse_transform function has > parameter Options so it is possible to add an argument to erlc with > app name, but is it possible to get the name of the app automatically? > > I tried > {attribute, _, module, ModuleName} = lists:keyfind(module, 3, Forms), > application:get_application(ModuleName), > but it returns undefined > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From ingela.andin@REDACTED Thu Dec 22 23:00:30 2016 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 22 Dec 2016 23:00:30 +0100 Subject: [erlang-questions] SSL handshake failure In-Reply-To: References: Message-ID: *Hi!* *Did get the solution slightly wrong ;) * *Try this instead.* *Regards Ingela * *diff --git a/lib/ssl/src/tls_connection.erl b/lib/ssl/src/tls_connection.erl* *index 32991d3..108bf50 100644* *--- a/lib/ssl/src/tls_connection.erl* *+++ b/lib/ssl/src/tls_connection.erl* @@ -424,18 +424,24 @@ handle_common_event(internal, #ssl_tls{type = ?HANDSHAKE, fragment = Data}, ssl_options = Options} = State0) -> try {Packets, Buf} = tls_handshake:get_tls_handshake(Version,Data,Buf0, Options), - State = + State1 = State0#state{protocol_buffers = Buffers#protocol_buffers{tls_handshake_buffer = Buf}}, - Events = tls_handshake_events(Packets), - case StateName of - connection -> - ssl_connection:hibernate_after(StateName, State, Events); - _ -> - {next_state, StateName, State#state{unprocessed_handshake_events = unprocessed_events(Events)}, Events} - end + case Packets of + [] -> + {Record, State} = next_record(State1), + next_event(StateName, Record, State); + _ -> + Events = tls_handshake_events(Packets), + case StateName of + connection -> + ssl_connection:hibernate_after(StateName, State1, Events); + _ -> + {next_state, StateName, State1#state{unprocessed_handshake_events = unprocessed_events(Events)}, Events} + end + end catch throw:#alert{} = Alert -> - ssl_connection:handle_own_alert(Alert, Version, StateName, State0) + ssl_connection:handle_own_alert(Alert, Version, StateName, State0) end; %%% TLS record protocol level application data messages handle_common_event(internal, #ssl_tls{type = ?APPLICATION_DATA, fragment = Data}, StateName, State) -> 2016-12-22 16:44 GMT+01:00 Ingela Andin : > Hi! > > > Well maybe I did not understand the problem correctly then, I thought I > had found a potential problem but as I have nothing to test it against. > Can you provide me with a site to test > against or maybe use tracing to give me more info. dbg:tracer(). > dbg:p(all, [call]). dbg:tpl(tls_handshake, get_tls_handshake, x). Could > be a good way to start.rds > > > Regards Ingela > > > 2016-12-22 16:29 GMT+01:00 Justin Wood : > >> Thank you for the patch Ingela. Unfortunately it did not fix my issue. >> >> If there is anything else I can provide to help figure this it, just let >> me know. >> >> On Thu, Dec 22, 2016 at 6:56 AM, Ingela Andin >> wrote: >> >>> Hi! >>> >>> I think this might be a timing bug. Can you try the following patch? >>> >>> Regards Ingela Erlang/OTP Team - Ericsson AB >>> >>> *diff --git a/lib/ssl/src/tls_connection.erl >>> b/lib/ssl/src/tls_connection.erl* >>> >>> *index 32991d3..e1f8e78 100644* >>> >>> *--- a/lib/ssl/src/tls_connection.erl* >>> >>> *+++ b/lib/ssl/src/tls_connection.erl* >>> >>> @@ -420,6 +420,7 @@ handle_common_event(internal, #alert{} = Alert, >>> StateName, >>> >>> handle_common_event(internal, #ssl_tls{type = ?HANDSHAKE, fragment = >>> Data}, >>> >>> StateName, #state{protocol_buffers = >>> >>> #protocol_buffers{tls_handshake_buffer >>> = Buf0} = Buffers, >>> >>> + unprocessed_handshake_events = >>> NumUnprocessed, >>> >>> negotiated_version = Version, >>> >>> ssl_options = Options} = State0) -> >>> >>> try >>> >>> @@ -432,7 +433,7 @@ handle_common_event(internal, #ssl_tls{type = >>> ?HANDSHAKE, fragment = Data}, >>> >>> connection -> >>> >>> ssl_connection:hibernate_after(StateName, State, >>> Events); >>> >>> _ -> >>> >>> - {next_state, StateName, State#state{unprocessed_handshake_events >>> = unprocessed_events(Events)}, Events} >>> >>> + {next_state, StateName, State#state{unprocessed_handshake_events >>> = unprocessed_events(Events, NumUnprocessed)}, Events} >>> >>> end >>> >>> catch throw:#alert{} = Alert -> >>> >>> ssl_connection:handle_own_alert(Alert, Version, StateName, >>> State0) >>> >>> @@ -615,8 +616,6 @@ next_event(StateName, Record, State, Actions) -> >>> >>> {next_state, StateName, State, [{next_event, internal, >>> Alert} | Actions]} >>> >>> end. >>> >>> >>> >>> -tls_handshake_events([]) -> >>> >>> - throw(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, malformed_handshake)); >>> >>> tls_handshake_events(Packets) -> >>> >>> lists:map(fun(Packet) -> >>> >>> {next_event, internal, {handshake, Packet}} >>> >>> @@ -727,8 +726,10 @@ gen_info(Event, StateName, >>> #state{negotiated_version = Version} = State) -> >>> >>> >>> malformed_handshake_data), >>> >>> Version, StateName, State) >>> >>> end. >>> >>> - >>> >>> -unprocessed_events(Events) -> >>> >>> + >>> >>> +unprocessed_events([], Unprocessed) -> >>> >>> + Unprocessed; >>> >>> +unprocessed_events(Events, _) -> >>> >>> %% The first handshake event will be processed immediately >>> >>> %% as it is entered first in the event queue and >>> >>> %% when it is processed there will be length(Events)-1 >>> >>> >>> >>> >>> 2016-12-21 14:47 GMT+01:00 Justin Wood : >>> >>>> After upgrading to Erlang OTP-19.2 and, in different requests, >>>> specified the version as one of tlsv1, tlsv1.1 or tlsv1.2. Every time I >>>> received the same error. >>>> >>>> I'm not sure if it is helpful at all, but I can replicate this on >>>> Windows, OSX 10.11 (El Capitan) and Ubuntu 14.04. I have not tried on other >>>> platforms. >>>> >>>> Is there any other information that I would be able to give that would >>>> be helpful? >>>> >>>> On Wed, Dec 21, 2016 at 3:38 AM, Technion wrote: >>>> >>>>> Hi Justin, >>>>> >>>>> >>>>> Is this earlier discussion relevant: >>>>> >>>>> >>>>> http://erlang.org/pipermail/erlang-questions/2016-November/090780.html >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------ >>>>> *From:* erlang-questions-bounces@REDACTED < >>>>> erlang-questions-bounces@REDACTED> on behalf of Justin Wood < >>>>> justin.k.wood@REDACTED> >>>>> *Sent:* Wednesday, 21 December 2016 11:56 AM >>>>> *To:* erlang-questions@REDACTED >>>>> *Subject:* [erlang-questions] SSL handshake failure >>>>> >>>>> Hi there, >>>>> >>>>> I am attempting to use the ssl module in order to get a socket to a >>>>> remote server (MongoDB). I am using the following line of code (OTP 19) >>>>> >>>>> ssl:connect("my-server.net", 27017, [binary, {active, false}, >>>>> {packet, raw}], 5000). >>>>> >>>>> Whenever I use this, I get the following: >>>>> >>>>> =ERROR REPORT==== 20-Dec-2016::19:13:13 === >>>>> SSL: certify: tls_connection.erl:603:Fatal error: handshake failure - >>>>> malformed_handshake >>>>> {error,{tls_alert,"handshake failure"}} >>>>> >>>>> Which lead me to look into erlang:get_stacktrace/0 >>>>> >>>>> erlang:get_stacktrace(). >>>>> [{tls_connection,start_fsm,8, >>>>> [{file,"tls_connection.erl"},{line,79}]}, >>>>> {ssl_connection,connect,8, >>>>> [{file,"ssl_connection.erl"},{line,84}]}, >>>>> {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,674}]}, >>>>> {shell,exprs,7,[{file,"shell.erl"},{line,686}]}, >>>>> {shell,eval_exprs,7,[{file,"shell.erl"},{line,641}]}, >>>>> {shell,eval_loop,3,[{file,"shell.erl"},{line,626}]}] >>>>> >>>>> My problem is that the certificate that the server is using should be >>>>> legitimate. I can connect, without error, using other clients and languages. >>>>> >>>>> I took a look through Wireshark in order to try and figure out what is >>>>> going on and this is what I see. >>>>> >>>>> * The Client says hello and presents a list of cipher suites. >>>>> * The Server says hello and says that it wants to use >>>>> TLS_RSA_WITH_AES_256_GCM_SHA384 (This cipher is in the list that the >>>>> client sends) >>>>> * The Server sends the Client the certificates it is using (these both >>>>> appear to be signed by DigiCert). >>>>> * The Server sends an encrypted handshake message. >>>>> * The Client responds with a Fatal Alert stating a Handshake Failure. >>>>> * The Server sends another encrypted handshake message. >>>>> >>>>> I have verified that ssl:connect/4 is working fine as I can connect to >>>>> a number of different miscellaneous services (including a number of other >>>>> MongoDB instances). >>>>> >>>>> I was wondering if there is anything else I can do to try and figure >>>>> out why erlang does not allow this connection. >>>>> >>>>> Justin >>>>> >>>> >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aschultz@REDACTED Fri Dec 23 09:57:29 2016 From: aschultz@REDACTED (Andreas Schultz) Date: Fri, 23 Dec 2016 09:57:29 +0100 (CET) Subject: [erlang-questions] OpenFlow Framework In-Reply-To: References: Message-ID: <47438874.223421.1482483449877.JavaMail.zimbra@tpip.net> Hi Wilmar, I guess you have seen flower [1], it is not really abandoned, but we don't really use it actively at the moment. If you have any questions or would like to contribute changes, let me know. Regards Andreas [1]: https://github.com/travelping/flower > From: "Wilmar P?rez" > To: "erlang-questions" > Sent: Thursday, December 22, 2016 4:16:50 PM > Subject: [erlang-questions] OpenFlow Framework > Hello > I have been tasked with finding an efficient and hopefully mature OpenFlow > Controller. I like the idea of an Erlang OpenFlow controller but it seems like > all projects are abandoned. Is there any that is still being actively > developed? > Thanks > Wilmar > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From justin.k.wood@REDACTED Fri Dec 23 15:50:46 2016 From: justin.k.wood@REDACTED (Justin Wood) Date: Fri, 23 Dec 2016 09:50:46 -0500 Subject: [erlang-questions] SSL handshake failure In-Reply-To: References: Message-ID: Hi Ingela! That patch fixed my issue. I appreciate the quick help. Would you be able to try to explain what the actual bug was here? Also, since I am still new to Erlang, would you be able to say when this patch may land in an actual release? Thank you! On Thu, Dec 22, 2016 at 5:00 PM, Ingela Andin wrote: > *Hi!* > > *Did get the solution slightly wrong ;) * > > *Try this instead.* > > *Regards Ingela * > > *diff --git a/lib/ssl/src/tls_connection.erl > b/lib/ssl/src/tls_connection.erl* > > *index 32991d3..108bf50 100644* > > *--- a/lib/ssl/src/tls_connection.erl* > > *+++ b/lib/ssl/src/tls_connection.erl* > > @@ -424,18 +424,24 @@ handle_common_event(internal, #ssl_tls{type = > ?HANDSHAKE, fragment = Data}, > > ssl_options = Options} = State0) -> > > try > > {Packets, Buf} = tls_handshake:get_tls_handshake(Version,Data,Buf0, > Options), > > - State = > > + State1 = > > State0#state{protocol_buffers = > > Buffers#protocol_buffers{tls_handshake_buffer > = Buf}}, > > - Events = tls_handshake_events(Packets), > > - case StateName of > > - connection -> > > - ssl_connection:hibernate_after(StateName, State, Events); > > - _ -> > > - {next_state, StateName, State#state{unprocessed_handshake_events > = unprocessed_events(Events)}, Events} > > - end > > + case Packets of > > + [] -> > > + {Record, State} = next_record(State1), > > + next_event(StateName, Record, State); > > + _ -> > > + Events = tls_handshake_events(Packets), > > + case StateName of > > + connection -> > > + ssl_connection:hibernate_after(StateName, > State1, Events); > > + _ -> > > + {next_state, StateName, State1#state{unprocessed_handshake_events > = unprocessed_events(Events)}, Events} > > + end > > + end > > catch throw:#alert{} = Alert -> > > - ssl_connection:handle_own_alert(Alert, Version, StateName, > State0) > > + ssl_connection:handle_own_alert(Alert, Version, StateName, > State0) > > end; > > %%% TLS record protocol level application data messages > > handle_common_event(internal, #ssl_tls{type = ?APPLICATION_DATA, fragment > = Data}, StateName, State) -> > > 2016-12-22 16:44 GMT+01:00 Ingela Andin : > >> Hi! >> >> >> Well maybe I did not understand the problem correctly then, I thought I >> had found a potential problem but as I have nothing to test it against. >> Can you provide me with a site to test >> against or maybe use tracing to give me more info. dbg:tracer(). >> dbg:p(all, [call]). dbg:tpl(tls_handshake, get_tls_handshake, x). Could >> be a good way to start.rds >> >> >> Regards Ingela >> >> >> 2016-12-22 16:29 GMT+01:00 Justin Wood : >> >>> Thank you for the patch Ingela. Unfortunately it did not fix my issue. >>> >>> If there is anything else I can provide to help figure this it, just let >>> me know. >>> >>> On Thu, Dec 22, 2016 at 6:56 AM, Ingela Andin >>> wrote: >>> >>>> Hi! >>>> >>>> I think this might be a timing bug. Can you try the following patch? >>>> >>>> Regards Ingela Erlang/OTP Team - Ericsson AB >>>> >>>> *diff --git a/lib/ssl/src/tls_connection.erl >>>> b/lib/ssl/src/tls_connection.erl* >>>> >>>> *index 32991d3..e1f8e78 100644* >>>> >>>> *--- a/lib/ssl/src/tls_connection.erl* >>>> >>>> *+++ b/lib/ssl/src/tls_connection.erl* >>>> >>>> @@ -420,6 +420,7 @@ handle_common_event(internal, #alert{} = Alert, >>>> StateName, >>>> >>>> handle_common_event(internal, #ssl_tls{type = ?HANDSHAKE, fragment = >>>> Data}, >>>> >>>> StateName, #state{protocol_buffers = >>>> >>>> #protocol_buffers{tls_handshake_buffer >>>> = Buf0} = Buffers, >>>> >>>> + unprocessed_handshake_events = >>>> NumUnprocessed, >>>> >>>> negotiated_version = Version, >>>> >>>> ssl_options = Options} = State0) >>>> -> >>>> >>>> try >>>> >>>> @@ -432,7 +433,7 @@ handle_common_event(internal, #ssl_tls{type = >>>> ?HANDSHAKE, fragment = Data}, >>>> >>>> connection -> >>>> >>>> ssl_connection:hibernate_after(StateName, State, >>>> Events); >>>> >>>> _ -> >>>> >>>> - {next_state, StateName, State#state{unprocessed_handshake_events >>>> = unprocessed_events(Events)}, Events} >>>> >>>> + {next_state, StateName, State#state{unprocessed_handshake_events >>>> = unprocessed_events(Events, NumUnprocessed)}, Events} >>>> >>>> end >>>> >>>> catch throw:#alert{} = Alert -> >>>> >>>> ssl_connection:handle_own_alert(Alert, Version, StateName, >>>> State0) >>>> >>>> @@ -615,8 +616,6 @@ next_event(StateName, Record, State, Actions) -> >>>> >>>> {next_state, StateName, State, [{next_event, internal, >>>> Alert} | Actions]} >>>> >>>> end. >>>> >>>> >>>> >>>> -tls_handshake_events([]) -> >>>> >>>> - throw(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, malformed_handshake)); >>>> >>>> tls_handshake_events(Packets) -> >>>> >>>> lists:map(fun(Packet) -> >>>> >>>> {next_event, internal, {handshake, Packet}} >>>> >>>> @@ -727,8 +726,10 @@ gen_info(Event, StateName, >>>> #state{negotiated_version = Version} = State) -> >>>> >>>> >>>> malformed_handshake_data), >>>> >>>> Version, StateName, State) >>>> >>>> end. >>>> >>>> - >>>> >>>> -unprocessed_events(Events) -> >>>> >>>> + >>>> >>>> +unprocessed_events([], Unprocessed) -> >>>> >>>> + Unprocessed; >>>> >>>> +unprocessed_events(Events, _) -> >>>> >>>> %% The first handshake event will be processed immediately >>>> >>>> %% as it is entered first in the event queue and >>>> >>>> %% when it is processed there will be length(Events)-1 >>>> >>>> >>>> >>>> >>>> 2016-12-21 14:47 GMT+01:00 Justin Wood : >>>> >>>>> After upgrading to Erlang OTP-19.2 and, in different requests, >>>>> specified the version as one of tlsv1, tlsv1.1 or tlsv1.2. Every time I >>>>> received the same error. >>>>> >>>>> I'm not sure if it is helpful at all, but I can replicate this on >>>>> Windows, OSX 10.11 (El Capitan) and Ubuntu 14.04. I have not tried on other >>>>> platforms. >>>>> >>>>> Is there any other information that I would be able to give that would >>>>> be helpful? >>>>> >>>>> On Wed, Dec 21, 2016 at 3:38 AM, Technion >>>>> wrote: >>>>> >>>>>> Hi Justin, >>>>>> >>>>>> >>>>>> Is this earlier discussion relevant: >>>>>> >>>>>> >>>>>> http://erlang.org/pipermail/erlang-questions/2016-November/0 >>>>>> 90780.html >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> *From:* erlang-questions-bounces@REDACTED < >>>>>> erlang-questions-bounces@REDACTED> on behalf of Justin Wood < >>>>>> justin.k.wood@REDACTED> >>>>>> *Sent:* Wednesday, 21 December 2016 11:56 AM >>>>>> *To:* erlang-questions@REDACTED >>>>>> *Subject:* [erlang-questions] SSL handshake failure >>>>>> >>>>>> Hi there, >>>>>> >>>>>> I am attempting to use the ssl module in order to get a socket to a >>>>>> remote server (MongoDB). I am using the following line of code (OTP 19) >>>>>> >>>>>> ssl:connect("my-server.net", 27017, [binary, {active, false}, >>>>>> {packet, raw}], 5000). >>>>>> >>>>>> Whenever I use this, I get the following: >>>>>> >>>>>> =ERROR REPORT==== 20-Dec-2016::19:13:13 === >>>>>> SSL: certify: tls_connection.erl:603:Fatal error: handshake failure - >>>>>> malformed_handshake >>>>>> {error,{tls_alert,"handshake failure"}} >>>>>> >>>>>> Which lead me to look into erlang:get_stacktrace/0 >>>>>> >>>>>> erlang:get_stacktrace(). >>>>>> [{tls_connection,start_fsm,8, >>>>>> [{file,"tls_connection.erl"},{line,79}]}, >>>>>> {ssl_connection,connect,8, >>>>>> [{file,"ssl_connection.erl"},{line,84}]}, >>>>>> {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,674}]}, >>>>>> {shell,exprs,7,[{file,"shell.erl"},{line,686}]}, >>>>>> {shell,eval_exprs,7,[{file,"shell.erl"},{line,641}]}, >>>>>> {shell,eval_loop,3,[{file,"shell.erl"},{line,626}]}] >>>>>> >>>>>> My problem is that the certificate that the server is using should be >>>>>> legitimate. I can connect, without error, using other clients and languages. >>>>>> >>>>>> I took a look through Wireshark in order to try and figure out what >>>>>> is going on and this is what I see. >>>>>> >>>>>> * The Client says hello and presents a list of cipher suites. >>>>>> * The Server says hello and says that it wants to use >>>>>> TLS_RSA_WITH_AES_256_GCM_SHA384 (This cipher is in the list that the >>>>>> client sends) >>>>>> * The Server sends the Client the certificates it is using (these >>>>>> both appear to be signed by DigiCert). >>>>>> * The Server sends an encrypted handshake message. >>>>>> * The Client responds with a Fatal Alert stating a Handshake Failure. >>>>>> * The Server sends another encrypted handshake message. >>>>>> >>>>>> I have verified that ssl:connect/4 is working fine as I can connect >>>>>> to a number of different miscellaneous services (including a number of >>>>>> other MongoDB instances). >>>>>> >>>>>> I was wondering if there is anything else I can do to try and figure >>>>>> out why erlang does not allow this connection. >>>>>> >>>>>> Justin >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Fri Dec 23 23:40:13 2016 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 23 Dec 2016 23:40:13 +0100 Subject: [erlang-questions] SSL handshake failure In-Reply-To: References: Message-ID: Hi! 2016-12-23 15:50 GMT+01:00 Justin Wood : > Hi Ingela! > > That patch fixed my issue. I appreciate the quick help. > > Would you be able to try to explain what the actual bug was here? > Well in short the TLS handshake protocol packets are packed into TLS record packets that are received on the tcp socket. Usually TLS handshake packets are relativily small so when reaching the code of decoding TLS handshake packets normally there will be atleast one compleate handshake packet. But your certificate was very big and then when reaching the code there was no compleate handshake and we need to wait for more input from the socket. In the more common case, when a compleate handshake message is assembled, we want to process it before asking for more data from the socket. > Also, since I am still new to Erlang, would you be able to say when this > patch may land in an actual release? > > Well as it is christmas holidays now I would not expect anything until after Januari 11. After that it will be merged to github maint. Then it might be officially patched if deemed importante enough, and if not it will be released in the next planned service release. That release will be ,if I remember correctly, in March. Regards Ingela Erlang/OTP team - Ericsson AB Thank you! > > On Thu, Dec 22, 2016 at 5:00 PM, Ingela Andin > wrote: > >> *Hi!* >> >> *Did get the solution slightly wrong ;) * >> >> *Try this instead.* >> >> *Regards Ingela * >> >> *diff --git a/lib/ssl/src/tls_connection.erl >> b/lib/ssl/src/tls_connection.erl* >> >> *index 32991d3..108bf50 100644* >> >> *--- a/lib/ssl/src/tls_connection.erl* >> >> *+++ b/lib/ssl/src/tls_connection.erl* >> >> @@ -424,18 +424,24 @@ handle_common_event(internal, #ssl_tls{type = >> ?HANDSHAKE, fragment = Data}, >> >> ssl_options = Options} = State0) -> >> >> try >> >> {Packets, Buf} = tls_handshake:get_tls_handshake(Version,Data,Buf0, >> Options), >> >> - State = >> >> + State1 = >> >> State0#state{protocol_buffers = >> >> Buffers#protocol_buffers{tls_handshake_buffer >> = Buf}}, >> >> - Events = tls_handshake_events(Packets), >> >> - case StateName of >> >> - connection -> >> >> - ssl_connection:hibernate_after(StateName, State, Events); >> >> - _ -> >> >> - {next_state, StateName, State#state{unprocessed_handshake_events >> = unprocessed_events(Events)}, Events} >> >> - end >> >> + case Packets of >> >> + [] -> >> >> + {Record, State} = next_record(State1), >> >> + next_event(StateName, Record, State); >> >> + _ -> >> >> + Events = tls_handshake_events(Packets), >> >> + case StateName of >> >> + connection -> >> >> + ssl_connection:hibernate_after(StateName, >> State1, Events); >> >> + _ -> >> >> + {next_state, StateName, >> State1#state{unprocessed_handshake_events = unprocessed_events(Events)}, >> Events} >> >> + end >> >> + end >> >> catch throw:#alert{} = Alert -> >> >> - ssl_connection:handle_own_alert(Alert, Version, StateName, >> State0) >> >> + ssl_connection:handle_own_alert(Alert, Version, StateName, >> State0) >> >> end; >> >> %%% TLS record protocol level application data messages >> >> handle_common_event(internal, #ssl_tls{type = ?APPLICATION_DATA, >> fragment = Data}, StateName, State) -> >> >> 2016-12-22 16:44 GMT+01:00 Ingela Andin : >> >>> Hi! >>> >>> >>> Well maybe I did not understand the problem correctly then, I thought I >>> had found a potential problem but as I have nothing to test it against. >>> Can you provide me with a site to test >>> against or maybe use tracing to give me more info. dbg:tracer(). >>> dbg:p(all, [call]). dbg:tpl(tls_handshake, get_tls_handshake, x). Could >>> be a good way to start.rds >>> >>> >>> Regards Ingela >>> >>> >>> 2016-12-22 16:29 GMT+01:00 Justin Wood : >>> >>>> Thank you for the patch Ingela. Unfortunately it did not fix my issue. >>>> >>>> If there is anything else I can provide to help figure this it, just >>>> let me know. >>>> >>>> On Thu, Dec 22, 2016 at 6:56 AM, Ingela Andin >>>> wrote: >>>> >>>>> Hi! >>>>> >>>>> I think this might be a timing bug. Can you try the following patch? >>>>> >>>>> Regards Ingela Erlang/OTP Team - Ericsson AB >>>>> >>>>> *diff --git a/lib/ssl/src/tls_connection.erl >>>>> b/lib/ssl/src/tls_connection.erl* >>>>> >>>>> *index 32991d3..e1f8e78 100644* >>>>> >>>>> *--- a/lib/ssl/src/tls_connection.erl* >>>>> >>>>> *+++ b/lib/ssl/src/tls_connection.erl* >>>>> >>>>> @@ -420,6 +420,7 @@ handle_common_event(internal, #alert{} = Alert, >>>>> StateName, >>>>> >>>>> handle_common_event(internal, #ssl_tls{type = ?HANDSHAKE, fragment >>>>> = Data}, >>>>> >>>>> StateName, #state{protocol_buffers = >>>>> >>>>> >>>>> #protocol_buffers{tls_handshake_buffer = Buf0} = Buffers, >>>>> >>>>> + unprocessed_handshake_events = >>>>> NumUnprocessed, >>>>> >>>>> negotiated_version = Version, >>>>> >>>>> ssl_options = Options} = State0) >>>>> -> >>>>> >>>>> try >>>>> >>>>> @@ -432,7 +433,7 @@ handle_common_event(internal, #ssl_tls{type = >>>>> ?HANDSHAKE, fragment = Data}, >>>>> >>>>> connection -> >>>>> >>>>> ssl_connection:hibernate_after(StateName, State, >>>>> Events); >>>>> >>>>> _ -> >>>>> >>>>> - {next_state, StateName, State#state{unprocessed_handshake_events >>>>> = unprocessed_events(Events)}, Events} >>>>> >>>>> + {next_state, StateName, State#state{unprocessed_handshake_events >>>>> = unprocessed_events(Events, NumUnprocessed)}, Events} >>>>> >>>>> end >>>>> >>>>> catch throw:#alert{} = Alert -> >>>>> >>>>> ssl_connection:handle_own_alert(Alert, Version, >>>>> StateName, State0) >>>>> >>>>> @@ -615,8 +616,6 @@ next_event(StateName, Record, State, Actions) -> >>>>> >>>>> {next_state, StateName, State, [{next_event, internal, >>>>> Alert} | Actions]} >>>>> >>>>> end. >>>>> >>>>> >>>>> >>>>> -tls_handshake_events([]) -> >>>>> >>>>> - throw(?ALERT_REC(?FATAL, ?HANDSHAKE_FAILURE, >>>>> malformed_handshake)); >>>>> >>>>> tls_handshake_events(Packets) -> >>>>> >>>>> lists:map(fun(Packet) -> >>>>> >>>>> {next_event, internal, {handshake, Packet}} >>>>> >>>>> @@ -727,8 +726,10 @@ gen_info(Event, StateName, >>>>> #state{negotiated_version = Version} = State) -> >>>>> >>>>> >>>>> malformed_handshake_data), >>>>> >>>>> Version, StateName, State) >>>>> >>>>> >>>>> end. >>>>> >>>>> - >>>>> >>>>> -unprocessed_events(Events) -> >>>>> >>>>> + >>>>> >>>>> +unprocessed_events([], Unprocessed) -> >>>>> >>>>> + Unprocessed; >>>>> >>>>> +unprocessed_events(Events, _) -> >>>>> >>>>> %% The first handshake event will be processed immediately >>>>> >>>>> %% as it is entered first in the event queue and >>>>> >>>>> %% when it is processed there will be length(Events)-1 >>>>> >>>>> >>>>> >>>>> >>>>> 2016-12-21 14:47 GMT+01:00 Justin Wood : >>>>> >>>>>> After upgrading to Erlang OTP-19.2 and, in different requests, >>>>>> specified the version as one of tlsv1, tlsv1.1 or tlsv1.2. Every time I >>>>>> received the same error. >>>>>> >>>>>> I'm not sure if it is helpful at all, but I can replicate this on >>>>>> Windows, OSX 10.11 (El Capitan) and Ubuntu 14.04. I have not tried on other >>>>>> platforms. >>>>>> >>>>>> Is there any other information that I would be able to give that >>>>>> would be helpful? >>>>>> >>>>>> On Wed, Dec 21, 2016 at 3:38 AM, Technion >>>>>> wrote: >>>>>> >>>>>>> Hi Justin, >>>>>>> >>>>>>> >>>>>>> Is this earlier discussion relevant: >>>>>>> >>>>>>> >>>>>>> http://erlang.org/pipermail/erlang-questions/2016-November/0 >>>>>>> 90780.html >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ------------------------------ >>>>>>> *From:* erlang-questions-bounces@REDACTED < >>>>>>> erlang-questions-bounces@REDACTED> on behalf of Justin Wood < >>>>>>> justin.k.wood@REDACTED> >>>>>>> *Sent:* Wednesday, 21 December 2016 11:56 AM >>>>>>> *To:* erlang-questions@REDACTED >>>>>>> *Subject:* [erlang-questions] SSL handshake failure >>>>>>> >>>>>>> Hi there, >>>>>>> >>>>>>> I am attempting to use the ssl module in order to get a socket to a >>>>>>> remote server (MongoDB). I am using the following line of code (OTP 19) >>>>>>> >>>>>>> ssl:connect("my-server.net", 27017, [binary, {active, false}, >>>>>>> {packet, raw}], 5000). >>>>>>> >>>>>>> Whenever I use this, I get the following: >>>>>>> >>>>>>> =ERROR REPORT==== 20-Dec-2016::19:13:13 === >>>>>>> SSL: certify: tls_connection.erl:603:Fatal error: handshake failure >>>>>>> - malformed_handshake >>>>>>> {error,{tls_alert,"handshake failure"}} >>>>>>> >>>>>>> Which lead me to look into erlang:get_stacktrace/0 >>>>>>> >>>>>>> erlang:get_stacktrace(). >>>>>>> [{tls_connection,start_fsm,8, >>>>>>> [{file,"tls_connection.erl"},{line,79}]}, >>>>>>> {ssl_connection,connect,8, >>>>>>> [{file,"ssl_connection.erl"},{line,84}]}, >>>>>>> {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,674}]}, >>>>>>> {shell,exprs,7,[{file,"shell.erl"},{line,686}]}, >>>>>>> {shell,eval_exprs,7,[{file,"shell.erl"},{line,641}]}, >>>>>>> {shell,eval_loop,3,[{file,"shell.erl"},{line,626}]}] >>>>>>> >>>>>>> My problem is that the certificate that the server is using should >>>>>>> be legitimate. I can connect, without error, using other clients and >>>>>>> languages. >>>>>>> >>>>>>> I took a look through Wireshark in order to try and figure out what >>>>>>> is going on and this is what I see. >>>>>>> >>>>>>> * The Client says hello and presents a list of cipher suites. >>>>>>> * The Server says hello and says that it wants to use >>>>>>> TLS_RSA_WITH_AES_256_GCM_SHA384 (This cipher is in the list that >>>>>>> the client sends) >>>>>>> * The Server sends the Client the certificates it is using (these >>>>>>> both appear to be signed by DigiCert). >>>>>>> * The Server sends an encrypted handshake message. >>>>>>> * The Client responds with a Fatal Alert stating a Handshake Failure. >>>>>>> * The Server sends another encrypted handshake message. >>>>>>> >>>>>>> I have verified that ssl:connect/4 is working fine as I can connect >>>>>>> to a number of different miscellaneous services (including a number of >>>>>>> other MongoDB instances). >>>>>>> >>>>>>> I was wondering if there is anything else I can do to try and figure >>>>>>> out why erlang does not allow this connection. >>>>>>> >>>>>>> Justin >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>> erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From icfp.publicity@REDACTED Sat Dec 24 03:56:24 2016 From: icfp.publicity@REDACTED (Lindsey Kuper) Date: Fri, 23 Dec 2016 18:56:24 -0800 Subject: [erlang-questions] Call for Papers: ICFP 2017 Message-ID: <585de3d87e739_36f43fd79c865bec6438@landin.local.mail> ICFP 2017 The 22nd ACM SIGPLAN International Conference on Functional Programming Oxford, United Kingdom http://icfp17.sigplan.org/ Call for Papers ### Important dates Submissions due: Monday, February 27, Anywhere on Earth https://icfp17.hotcrp.com Author response: Monday, April 17, 2017, 15:00 (UTC) - Thursday, April 20, 2017, 15:00 (UTC) Notification: Monday, 1 May, 2017 Final copy due: Monday, 5 June 2017 Early registration: TBA Conference: Monday, 4 September - Wednesday, 6 September, 2017 ### New this year Those familiar with previous ICFP conferences should be aware of two significant changes that are being introduced in 2017: 1. Papers selected for ICFP 2017 will be published as the ICFP 2017 issue of a new journal, Proceedings of the ACM on Programming Languages (PACMPL), which replaces the previous ICFP conference proceedings. The move to PACMPL will have two noticeable impacts on authors: * A new, two-phase selection and reviewing process that conforms to ACM?s journal reviewing guidelines. * A new, single-column format for submissions. 2. Authors of papers that are conditionally accepted in the first phase of the reviewing process will have the option to submit materials for Artifact Evaluation. Further details on each of these changes are included in the following text. ### Scope ICFP 2017 seeks original papers on the art and science of functional programming. Submissions are invited on all topics from principles to practice, from foundations to features, and from abstraction to application. The scope includes all languages that encourage functional programming, including both purely applicative and imperative languages, as well as languages with objects, concurrency, or parallelism. Topics of interest include (but are not limited to): * *Language Design*: concurrency, parallelism, and distribution; modules; components and composition; metaprogramming; type systems; interoperability; domain-specific languages; and relations to imperative, object-oriented, or logic programming. * *Implementation*: abstract machines; virtual machines; interpretation; compilation; compile-time and run-time optimization; garbage collection and memory management; multi-threading; exploiting parallel hardware; interfaces to foreign functions, services, components, or low-level machine resources. * *Software-Development Techniques*: algorithms and data structures; design patterns; specification; verification; validation; proof assistants; debugging; testing; tracing; profiling. * *Foundations*: formal semantics; lambda calculus; rewriting; type theory; monads; continuations; control; state; effects; program verification; dependent types. * *Analysis and Transformation*: control-flow; data-flow; abstract interpretation; partial evaluation; program calculation. * *Applications*: symbolic computing; formal-methods tools; artificial intelligence; systems programming; distributed-systems and web programming; hardware design; databases; XML processing; scientific and numerical computing; graphical user interfaces; multimedia and 3D graphics programming; scripting; system administration; security. * *Education*: teaching introductory programming; parallel programming; mathematical proof; algebra. Submissions will be evaluated according to their relevance, correctness, significance, originality, and clarity. Each submission should explain its contributions in both general and technical terms, clearly identifying what has been accomplished, explaining why it is significant, and comparing it with previous work. The technical content should be accessible to a broad audience. ICFP 2017 also welcomes submissions in two separate categories — Functional Pearls and Experience Reports — that must be marked as such at the time of submission and that need not report original research results. Detailed guidelines on both categories are given at the end of this call. Please contact the program chair if you have questions or are concerned about the appropriateness of a topic. ### Preparation of submissions **Deadline**: The deadline for submissions is Monday, February 27, 2017, Anywhere on Earth (). This deadline will be strictly enforced. **Formatting**: (NOTE: NEW FORMAT REQUIREMENTS FOR ICFP 2017) Submissions must be in PDF format, printable in black and white on US Letter sized paper, and interpretable by common PDF tools. All submissions must adhere to the "ACM Large" template that is available (in both LaTeX and Word formats) from . For authors using LaTeX, a lighter-weight package, including only the essential files, is available from . There is a limit of 24 pages for a full paper or 12 pages for an Experience Report; in either case, the bibliography will not be counted against these limits. These page limits have been chosen to allow essentially the same amount of content with the new single-column format as was possible with the two-column format used in past ICFP conferences. Submissions that exceed the page limits or, for other reasons, do not meet the requirements for formatting, will be summarily rejected. **Submission**: Submissions will be accepted at (in preparation at the time of writing). Improved versions of a paper may be submitted at any point before the submission deadline using the same web interface. **Author Response Period**: Authors will have a 72-hour period, starting at 15:00 UTC on Monday, April 17, 2017, to read reviews and respond to them. **Supplementary Materials**: Authors have the option to attach supplementary material to a submission, on the understanding that reviewers may choose not to look at it. The material should be uploaded at submission time, as a single pdf or a tarball, not via a URL. This supplementary material may or may not be anonymized; if not anonymized, it will only be revealed to reviewers after they have submitted their review of the paper and learned the identity of the author(s). **Authorship Policies**: All submissions are expected to comply with the ACM Policies for Authorship that are detailed at . **Republication Policies**: Each submission must adhere to SIGPLAN's republication policy, as explained on the web at . **Resubmitted Papers**: Authors who submit a revised version of a paper that has previously been rejected by another conference have the option to attach an annotated copy of the reviews of their previous submission(s), explaining how they have addressed these previous reviews in the present submission. If a reviewer identifies him/herself as a reviewer of this previous submission and wishes to see how his/her comments have been addressed, the program chair will communicate to this reviewer the annotated copy of his/her previous review. Otherwise, no reviewer will read the annotated copies of the previous reviews. ### Review Process This section outlines the two-stage process with lightweight double-blind reviewing that will be used to select papers for presentation at ICFP 2017. We anticipate that there will be a need to clarify and expand on this process, and we will maintain a list of frequently asked questions and answers on the conference website to address common concerns. **ICFP 2017 will employ a two-stage review process.** The first stage in the review process will assess submitted papers using the criteria stated above and will allow for feedback and input on initial reviews through the author response period mentioned previously. At the PC meeting, a set of papers will be conditionally accepted and all other papers will be rejected. Authors will be notified of these decisions on May 1, 2017. Authors of conditionally accepted papers will be provided with committee reviews (just as in previous conferences) along with a set of mandatory revisions. After five weeks (June 5, 2017), the authors will provide a second submission. The second and final reviewing phase assesses whether the mandatory revisions have been adequately addressed by the authors and thereby determines the final accept/reject status of the paper. The intent and expectation is that the mandatory revisions can be addressed within five weeks and hence that conditionally accepted papers will in general be accepted in the second phase. The second submission should clearly identify how the mandatory revisions were addressed. To that end, the second submission must be accompanied by a cover letter mapping each mandatory revision request to specific parts of the paper. The cover letter will facilitate a quick second review, allowing for confirmation of final acceptance within two weeks. Conversely, the absence of a cover letter will be grounds for the paper?s rejection. This process is intended as a refinement of the review process that has been used in previous ICFP conferences. By incorporating a second stage, the process will conform to ACM?s journal reviewing guidelines for PACMPL. **ICFP 2017 will employ a lightweight double-blind reviewing process.** To facilitate this, submitted papers must adhere to two rules: 1. **author names and institutions must be omitted**, and 2. **references to authors' own related work should be in the third person** (e.g., not "We build on our previous work ..." but rather "We build on the work of ..."). The purpose of this process is to help the PC and external reviewers come to an initial judgement about the paper without bias, not to make it impossible for them to discover the authors if they were to try. Nothing should be done in the name of anonymity that weakens the submission or makes the job of reviewing the paper more difficult (e.g., important background references should not be omitted or anonymized). In addition, authors should feel free to disseminate their ideas or draft versions of their paper as they normally would. For instance, authors may post drafts of their papers on the web or give talks on their research ideas. ### Information for Authors of Accepted Papers * As a condition of acceptance, final versions of all papers must adhere to the new ACM Large format. The page limits for final versions of papers will be increased to ensure that authors have space to respond to reviewer comments and mandatory revisions. * Authors of accepted submissions will be required to agree to one of the three ACM licensing options: copyright transfer to ACM; retaining copyright but granting ACM exclusive publication rights; or open access on payment of a fee. Further information about ACM author rights is available from . * At least one author of each accepted submissions will be expected to attend and present their paper at the conference. The schedule for presentations will be determined and shared with authors after the full program has been selected. Presentations will be videotaped and released online if the presenter consents. * We intend that the proceedings will be freely available for download from the ACM Digital Library in perpetuity via the OpenTOC mechanism. * ACM Author-Izer is a unique service that enables ACM authors to generate and post links on either their home page or institutional repository for visitors to download the definitive version of their articles from the ACM Digital Library at no charge. Downloads through Author-Izer links are captured in official ACM statistics, improving the accuracy of usage and impact measurements. Consistently linking to the definitive version of an ACM article should reduce user confusion over article versioning. After an article has been published and assigned to the appropriate ACM Author Profile pages, authors should visit to learn how to create links for free downloads from the ACM DL. * The official publication date is the date the proceedings are made available in the ACM Digital Library. This date may be up to *two weeks prior* to the first day of the conference. The official publication date affects the deadline for any patent filings related to published work. ### Artifact Evaluation Authors of papers that are conditionally accepted in the first phase of the review process will be encouraged (but not required) to submit supporting materials for Artifact Evaluation. These items will then be reviewed by a committee, separate from the program committee, whose task is to assess how the artifacts support the work described in the associated paper. Papers that go through the Artifact Evaluation process successfully will receive a seal of approval printed on the papers themselves. Authors of accepted papers will be encouraged to make the supporting materials publicly available upon publication of the proceedings, for example, by including them as "source materials" in the ACM Digital Library. An additional seal will mark papers whose artifacts are made available, as outlined in the ACM guidelines for artifact badging. Participation in Artifact Evaluation is voluntary and will not influence the final decision regarding paper acceptance. Further information about the motivations and expectations for Artifact Evaluation can be found at . ### Special categories of papers In addition to research papers, ICFP solicits two kinds of papers that do not require original research contributions: Functional Pearls, which are full papers, and Experience Reports, which are limited to half the length of a full paper. Authors submitting such papers should consider the following guidelines. #### Functional Pearls A Functional Pearl is an elegant essay about something related to functional programming. Examples include, but are not limited to: * a new and thought-provoking way of looking at an old idea * an instructive example of program calculation or proof * a nifty presentation of an old or new data structure * an interesting application of functional programming techniques * a novel use or exposition of functional programming in the classroom While pearls often demonstrate an idea through the development of a short program, there is no requirement or expectation that they do so. Thus, they encompass the notions of theoretical and educational pearls. Functional Pearls are valued as highly and judged as rigorously as ordinary papers, but using somewhat different criteria. In particular, a pearl is not required to report original research, but, it should be concise, instructive, and entertaining. A pearl is likely to be rejected if its readers get bored, if the material gets too complicated, if too much specialized knowledge is needed, or if the writing is inelegant. The key to writing a good pearl is polishing. A submission that is intended to be treated as a pearl must be marked as such on the submission web page, and should contain the words "Functional Pearl" somewhere in its title or subtitle. These steps will alert reviewers to use the appropriate evaluation criteria. Pearls will be combined with ordinary papers, however, for the purpose of computing the conference's acceptance rate. #### Experience Reports The purpose of an Experience Report is to help create a body of published, refereed, citable evidence that functional programming really works — or to describe what obstacles prevent it from working. Possible topics for an Experience Report include, but are not limited to: * insights gained from real-world projects using functional programming * comparison of functional programming with conventional programming in the context of an industrial project or a university curriculum * project-management, business, or legal issues encountered when using functional programming in a real-world project * curricular issues encountered when using functional programming in education * real-world constraints that created special challenges for an implementation of a functional language or for functional programming in general An Experience Report is distinguished from a normal ICFP paper by its title, by its length, and by the criteria used to evaluate it. * Both in the proceedings and in any citations, the title of each accepted Experience Report must begin with the words "Experience Report" followed by a colon. The acceptance rate for Experience Reports will be computed and reported separately from the rate for ordinary papers. * Experience Report submissions can be at most 12 pages long, excluding bibliography. * Each accepted Experience Report will be presented at the conference, but depending on the number of Experience Reports and regular papers accepted, authors of Experience reports may be asked to give shorter talks. * Because the purpose of Experience Reports is to enable our community to accumulate a body of evidence about the efficacy of functional programming, an acceptable Experience Report need not add to the body of knowledge of the functional-programming community by presenting novel results or conclusions. It is sufficient if the Report states a clear thesis and provides supporting evidence. The thesis must be relevant to ICFP, but it need not be novel. The program committee will accept or reject Experience Reports based on whether they judge the evidence to be convincing. Anecdotal evidence will be acceptable provided it is well argued and the author explains what efforts were made to gather as much evidence as possible. Typically, more convincing evidence is obtained from papers which show how functional programming was used than from papers which only say that functional programming was used. The most convincing evidence often includes comparisons of situations before and after the introduction or discontinuation of functional programming. Evidence drawn from a single person's experience may be sufficient, but more weight will be given to evidence drawn from the experience of groups of people. An Experience Report should be short and to the point: it should make a claim about how well functional programming worked on a particular project and why, and produce evidence to substantiate this claim. If functional programming worked in this case in the same ways it has worked for others, the paper need only summarize the results — the main part of the paper should discuss how well it worked and in what context. Most readers will not want to know all the details of the project and its implementation, but the paper should characterize the project and its context well enough so that readers can judge to what degree this experience is relevant to their own projects. The paper should take care to highlight any unusual aspects of the project. Specifics about the project are more valuable than generalities about functional programming; for example, it is more valuable to say that the team delivered its software a month ahead of schedule than it is to say that functional programming made the team more productive. If the paper not only describes experience but also presents new technical results, or if the experience refutes cherished beliefs of the functional-programming community, it may be better off submitted it as a full paper, which will be judged by the usual criteria of novelty, originality, and relevance. The program chair will be happy to advise on any concerns about which category to submit to. ### Organizers General Chair: Jeremy Gibbons (University of Oxford, UK) Program Chair: Mark Jones (Portland State University, USA) Artifact Evaluation Chair: Ryan R. Newton (Indiana University, USA) Industrial Relations Chair: Ryan Trinkle (Obsidian Systems LLC, USA) Programming Contest Organiser: Sam Lindley (University of Edinburgh, UK) Publicity and Web Chair: Lindsey Kuper (Intel Labs, USA) Student Research Competition Chair: Ilya Sergey (University College London, UK) Video Chair: Jose Calderon (Galois, Inc., USA) Workshops Co-Chair: Andres L?h (Well-Typed LLP) Workshops Co-Chair: David Christiansen (Indiana University, USA) Program Committee: Bob Atkey (University of Strathclyde, Scotland) Adam Chlipala (MIT, USA) Dominique Devriese (KU Leuven, Belgium) Martin Erwig (Oregon State, USA) Matthew Flatt (University of Utah, USA) Ronald Garcia (University of British Columbia, Canada) Kathryn Gray (University of Cambridge, England) John Hughes (Chalmers University and Quvik, Sweden) Chung-Kil Hur (Seoul National University, Korea) Graham Hutton (University of Nottingham, England) Alan Jeffrey (Mozilla Research, USA) Ranjit Jhala (University of California, San Diego, USA) Shin-ya Katsumata (Kyoto University, Japan) Lindsey Kuper (Intel Labs, USA) Dan Licata (Wesleyan University, USA) Ben Lippmeier (Digital Asset, Australia) Gabriel Scherer (Northeastern University, USA) Alexandra Silva (University College London, England) Nikhil Swamy (Microsoft Research, USA) Sam Tobin-Hochstadt (Indiana University, USA) Nicolas Wu (University of Bristol, England) Beta Ziliani (CONICET and FAMAF, Universidad Nacional de C?rdoba, Argentina) From max.lapshin@REDACTED Sat Dec 24 12:44:05 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 24 Dec 2016 14:44:05 +0300 Subject: [erlang-questions] getting application name in parse transform In-Reply-To: References: Message-ID: Look at how lager does it. Something like: guess_application(Dirname, Attr) when Dirname /= undefined -> case find_app_file(Dirname) of no_idea -> %% try it based on source file directory (app.src most likely) guess_application(undefined, Attr); _ -> no_idea end; guess_application(undefined, {attribute, _, file, {Filename, _}}) -> Dir = filename:dirname(Filename), find_app_file(Dir); guess_application(_, _) -> no_idea. find_app_file(Dir) -> case filelib:wildcard(Dir++"/*.{app,app.src}") of [] -> -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Sat Dec 24 22:34:22 2016 From: lloyd@REDACTED (lloyd@REDACTED) Date: Sat, 24 Dec 2016 16:34:22 -0500 (EST) Subject: [erlang-questions] =?utf-8?q?How_can_I_break_this_string_into_a_l?= =?utf-8?q?ist_of_strings=3F?= Message-ID: <1482615262.7413280@apps.rackspace.com> Hello, A naive question. But every time I look into the re module my heart starts pounding. I spend uncountable time figuring out what I need to do. But this time I failed. Suppose I have the following string: "

Hello!

\n

How are you?

\n

Some text\n and more text.

" I would like to break it into a list: ["

Hello!

", "

How are you?

", "

Some text\n and more text.

"] string:token(MyString, "$\n") doesn't work because it would break the paragraph. So I try: re:replace(Copy, [$>,$\n], [$>,$",$,,$"], [global, {return, list}]). But I get: "

Hello!

\",\"

How are you?

\",\"

This is.... I don't want those pesky escaped quotes at the end of the heads. I want the real deal. But how can I make it happen? Thanks and holiday cheer to all, LRP From povetkin.k@REDACTED Sat Dec 24 22:43:20 2016 From: povetkin.k@REDACTED (Constantine Povietkin) Date: Sat, 24 Dec 2016 23:43:20 +0200 Subject: [erlang-questions] How can I break this string into a list of strings? In-Reply-To: <1482615262.7413280@apps.rackspace.com> References: <1482615262.7413280@apps.rackspace.com> Message-ID: <1482615800.17457.0.camel@gmail.com> Hi, Just convert it to binary and use binary:split. ? ???, 24/12/2016 ? 16:34 -0500, lloyd@REDACTED ?????: > Hello, > > A naive question.? > > But every time I look into the re module my heart starts pounding. I > spend uncountable time figuring out what I need to do. But this time > I failed. > > Suppose I have the following string: > > ????"

Hello!

\n?????

How are you?

\n????

Some text\n > and more text.

" > > I would like to break it into a list: > > ????["

Hello!

", "

How are you?

", "

Some text\n and > more text.

"] > > string:token(MyString, "$\n") doesn't work because it would break the > paragraph.? > > So I try: > > ????re:replace(Copy, [$>,$\n], [$>,$",$,,$"], [global, {return, > list}]). > > But I get: > > ???"

Hello!

\",\"?????

How are you?

\",\"????

This > is.... > > I don't want those pesky escaped quotes at the end of the heads. I > want the real deal.? > > But how can I make it happen? > > Thanks and holiday cheer to all, > > LRP > > > > > > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From kennethlakin@REDACTED Sat Dec 24 23:42:58 2016 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Sat, 24 Dec 2016 14:42:58 -0800 Subject: [erlang-questions] How can I break this string into a list of strings? In-Reply-To: <1482615262.7413280@apps.rackspace.com> References: <1482615262.7413280@apps.rackspace.com> Message-ID: On 12/24/2016 01:34 PM, lloyd@REDACTED wrote: > I don't want those pesky escaped quotes at the end of the heads. I want the real deal. Does it help you to know that those double-quotes aren't _actually_ escaped and that the backslashes are added by the pretty-print code? Continuing on with your example: 2> B=re:replace(Copy, [$>,$\n], [$>,$",$,,$"], [global, {return, list}]). %re:replace output elided 3> lists:nth(16, B). 34 4> lists:nth(15, B). 62 ASCII 34 is '"'. ASCII 62 is '>'. There's no "\" in the output, which would be ASCII 92: 5> lists:foldl(fun(92, Acc) -> Acc ++ [92]; (_, Acc) -> Acc end, [], B). [] The pretty-printer is a nice feature (AFAICT, anything the pretty-printer prints is valid Erlang), but I suspect that being startled by its escaping is a rite of passage. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From eric.pailleau@REDACTED Sun Dec 25 00:33:13 2016 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Sun, 25 Dec 2016 00:33:13 +0100 Subject: [erlang-questions] How can I break this string into a list of strings? In-Reply-To: <1482615262.7413280@apps.rackspace.com> References: <1482615262.7413280@apps.rackspace.com> Message-ID: Hi, If re module make your heart pounding, you may give a try to https://github.com/crownedgrouse/swab and create powerful macros to apply on your buffers. "Envoy? depuis mon mobile " Eric ---- lloyd@REDACTED a ?crit ---- >Hello, > >A naive question. > >But every time I look into the re module my heart starts pounding. I spend uncountable time figuring out what I need to do. But this time I failed. > >Suppose I have the following string: > > "

Hello!

\n

How are you?

\n

Some text\n and more text.

" > >I would like to break it into a list: > > ["

Hello!

", "

How are you?

", "

Some text\n and more text.

"] > >string:token(MyString, "$\n") doesn't work because it would break the paragraph. > >So I try: > > re:replace(Copy, [$>,$\n], [$>,$",$,,$"], [global, {return, list}]). > >But I get: > > "

Hello!

\",\"

How are you?

\",\"

This is.... > >I don't want those pesky escaped quotes at the end of the heads. I want the real deal. > >But how can I make it happen? > >Thanks and holiday cheer to all, > >LRP > > > > > > > > > > > > >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Sun Dec 25 03:52:17 2016 From: lloyd@REDACTED (lloyd@REDACTED) Date: Sat, 24 Dec 2016 21:52:17 -0500 (EST) Subject: [erlang-questions] =?utf-8?q?How_can_I_break_this_string_into_a_l?= =?utf-8?q?ist_of_strings=3F?= In-Reply-To: References: <1482615262.7413280@apps.rackspace.com> Message-ID: <1482634337.59929712@apps.rackspace.com> Thanks all, but apologies. I misstated my problem. What I'm really need to do is convert: "

Hello!

\n

How are you?

\n

Some text\n> and more text.

" Into: ["

Hello!

", "

How are you?

", "

Some text\n and more text.

"] Such that: length(MyString) returns 3 or [length(String) || String <- MyString] returns [N1, N2,N3] re and string:tokens/2 got a bit muddled in my mind. Thanks again, LRP -----Original Message----- From: "Kenneth Lakin" Sent: Saturday, December 24, 2016 5:42pm To: erlang-questions@REDACTED, "Lloyd R. Prentice" Subject: Re: [erlang-questions] How can I break this string into a list of strings? On 12/24/2016 01:34 PM, lloyd@REDACTED wrote: > I don't want those pesky escaped quotes at the end of the heads. I want the real deal. Does it help you to know that those double-quotes aren't _actually_ escaped and that the backslashes are added by the pretty-print code? Continuing on with your example: 2> B=re:replace(Copy, [$>,$\n], [$>,$",$,,$"], [global, {return, list}]). %re:replace output elided 3> lists:nth(16, B). 34 4> lists:nth(15, B). 62 ASCII 34 is '"'. ASCII 62 is '>'. There's no "\" in the output, which would be ASCII 92: 5> lists:foldl(fun(92, Acc) -> Acc ++ [92]; (_, Acc) -> Acc end, [], B). [] The pretty-printer is a nice feature (AFAICT, anything the pretty-printer prints is valid Erlang), but I suspect that being startled by its escaping is a rite of passage. From lloyd@REDACTED Sun Dec 25 04:10:29 2016 From: lloyd@REDACTED (lloyd@REDACTED) Date: Sat, 24 Dec 2016 22:10:29 -0500 (EST) Subject: [erlang-questions] =?utf-8?q?How_can_I_break_this_string_into_a_l?= =?utf-8?q?ist_of_strings=3F?= In-Reply-To: References: <1482615262.7413280@apps.rackspace.com> Message-ID: <1482635429.913432237@apps.rackspace.com> Hi Eric, Swab looks very cool. But I'll have to wait until I've recovered from my New Year's hangover to dig in sufficiently to understand it. Up for questions if they occur? Thanks, Lloyd -----Original Message----- From: "?ric Pailleau" Sent: Saturday, December 24, 2016 6:33pm To: "Erlang Questions" , "Lloyd R. Prentice" Subject: Re: [erlang-questions] How can I break this string into a list of strings? Hi, If re module make your heart pounding, you may give a try to https://github.com/crownedgrouse/swab and create powerful macros to apply on your buffers. "Envoy? depuis mon mobile " Eric ---- lloyd@REDACTED a ?crit ---- >Hello, > >A naive question. > >But every time I look into the re module my heart starts pounding. I spend uncountable time figuring out what I need to do. But this time I failed. > >Suppose I have the following string: > > "

Hello!

\n

How are you?

\n

Some text\n and more text.

" > >I would like to break it into a list: > > ["

Hello!

", "

How are you?

", "

Some text\n and more text.

"] > >string:token(MyString, "$\n") doesn't work because it would break the paragraph. > >So I try: > > re:replace(Copy, [$>,$\n], [$>,$",$,,$"], [global, {return, list}]). > >But I get: > > "

Hello!

\",\"

How are you?

\",\"

This is.... > >I don't want those pesky escaped quotes at the end of the heads. I want the real deal. > >But how can I make it happen? > >Thanks and holiday cheer to all, > >LRP > > > > > > > > > > > > >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions From zxq9@REDACTED Sun Dec 25 06:21:00 2016 From: zxq9@REDACTED (zxq9) Date: Sun, 25 Dec 2016 14:21 +0900 Subject: [erlang-questions] How can I break this string into a list of strings? In-Reply-To: <1482615262.7413280@apps.rackspace.com> References: <1482615262.7413280@apps.rackspace.com> Message-ID: <6037061.ux5kc5t0N6@changa> Hi Lloyd! On 2016?12?24? ??? 16:34:22 lloyd@REDACTED wrote: > Hello, > > A naive question. > > But every time I look into the re module my heart starts pounding. I spend uncountable time figuring out what I need to do. But this time I failed. > > Suppose I have the following string: > > "

Hello!

\n

How are you?

\n

Some text\n and more text.

" > > I would like to break it into a list: > > ["

Hello!

", "

How are you?

", "

Some text\n and more text.

"] > > string:token(MyString, "$\n") doesn't work because it would break the paragraph. > > So I try: > > re:replace(Copy, [$>,$\n], [$>,$",$,,$"], [global, {return, list}]). > > But I get: > > "

Hello!

\",\"

How are you?

\",\"

This is.... > > I don't want those pesky escaped quotes at the end of the heads. I want the real deal. > > But how can I make it happen? Maybe we can elevate this to a more general problem, like "how can I interpret a list of angle-brackety notation and return a list that contains the parts of it in a way that is meaningful to my Erlang program?" Fortunately the days of "its OK if you have a closing tag, or not, but maybe just in case, sometimes" are over. So these days we know that, at least by HTML5 and XML type rules, we will always have an explicit closing tag, a slash, or a very few tags that are "self-closing" meaning that when we encounter them we can know right away that we won't encounter an ending one. We could get into some listy wizardry with lists of regexes operating over your list of charcters that flip it between binaries and lists and lists of binaries and other nonsense... but I've never seen that lead to a happy place, an understandable place, and it always winds up having dramatically worse performance characteristics (by every criteria, which is remarkable). So let's instead decide that we have an accumulator, which is a deep list (so it mimics the way the elements in the HTML look, like a tree of tags), your original list, and the idea that we need to have a mode for recording label of the tag (the "h1" or "h2" or whatever), recording the data enclosed by the tag ("hello!" and other text we actually care about), and a matching mode to find previous tags that are being closed. 3 modes, basically. To start with I want to move through the list, passing whatever I encounter to an accumulator. Most simple explicit recursion we can do -- so let's write a tinkering module for it: -module(htmler). -export([consume/1]). -spec consume(String) -> Result when String :: unicode:chardata(), Result :: {ok, Structure :: list()} | {error, Reason :: term()}. consume(String) -> consume(String, ""). consume([Char | Rest], Acc) -> consume(Rest, [Char | Acc]); consume([], Acc) -> {ok, lists:reverse(Acc)}. ...and now give it a try... 1> c(htmler). {ok,htmler} 2> htmler:consume("

Hello!

\n

How are you?

\n

Some text\n and more text.

"). {ok, "

Hello!

\n

How are you?

\n

Some text\n and more text.

"} OK, works as expected. If I encounter a $< then I want to start recording the tag label. If I hit whitespace I want to just pass by it until I hit the first meaningful character of the label. I add the label characters to an accumulator for the label. When I encounter whitespace I want to stop recording the tag label, but still move through the next characters until the closing $>. -module(htmler). -export([consume/1]). -spec consume(String) -> Result when String :: unicode:chardata(), Result :: {ok, Structure :: list()} | {error, Reason :: term()}. consume(String) -> consume(String, ""). consume([$< | Rest], Acc) -> case consume_label(Rest, "") of {ok, Label, Remainder} -> consume(Remainder, [Label | Acc]); {error, Label} -> {error, {bad_tag, Label}} end; consume([Char | Rest], Acc) -> consume(Rest, [Char | Acc]); consume([], Acc) -> {ok, lists:reverse(Acc)}. consume_label([$\s | Rest], Acc) -> consume_label(Rest, Acc); consume_label([$\t | Rest], Acc) -> consume_label(Rest, Acc); consume_label(String, Acc) -> consume_label_text(String, Acc). consume_label_text([$> | Rest], Acc) -> Label = lists:reverse(Acc), {ok, Label, Rest}; consume_label_text([$\s | Rest], Acc) -> consume_label_close(Rest, Acc); consume_label_text([$\t | Rest], Acc) -> consume_label_close(Rest, Acc); consume_label_text([Char | Rest], Acc) -> consume_label_text(Rest, [Char | Acc]); consume_label_text([], Acc) -> Label = lists:reverse(Acc), {error, Label}. consume_label_close([$> | Rest], Acc) -> Label = lists:reverse(Acc), {ok, Label, Rest}; consume_label_close([_ | Rest], Acc) -> consume_label_close(Rest, Acc). And now let's give that a try... 3> c(htmler). {ok,htmler} 4> io:format("~tp~n", [htmler:consume("

Hello!

\n

How are you?

\n

Some text\n and more text.

")]). {ok,["h1",72,101,108,108,111,33,"/h1",10,32,32,32,32,32,"h2",72,111,119,32,97, 114,101,32,121,111,117,63,"/h2",10,32,32,32,32,"p",83,111,109,101,32,116, 101,120,116,10,32,97,110,100,32,109,111,114,101,32,116,101,120,116,46, "/p"]} ok Ah, so we're on the right track, but not quite where we want to be. Looking at this I can tell that we probably don't want to just split things up, but extract some meaning. Maybe what we really want is something like a list of tuples with a label that tells us the kind of thing it is, and then has a list containing the data that the label applies to. Like: [{"h1", "Hello!"}, {"h2", "How are you?"}, {"p", "Some text\n and more text."}] I think we can change this a bit to get to that point, but it will require that we remember the tag we are currently in case, for example, there is an tag inside the

component or whatever else -- with these kinds of tags its not about finding a closing tag, it is about finding out which closing tag you've found. Maybe we want some part of this to actually be non-tail recursive -- so we can work on just the tag in which we are parsing at the moment and not remember everything else. -module(htmler). -export([consume/1]). -spec consume(String) -> Result when String :: unicode:chardata(), Result :: {ok, Structure :: list()} | {error, Reason :: term()}. consume(String) -> consume(String, []). consume([$< | Rest], Acc) -> case consume_element(Rest) of {ok, Element, Remainder} -> consume(Remainder, [Element | Acc]); {error, Element} -> {error, {bad_tag, Element}} end; consume([Char | Rest], Acc) -> consume(Rest, [Char | Acc]); consume([], Acc) -> {ok, lists:reverse(Acc)}. consume_element(String) -> case consume_label(String, "") of {ok, Label, Rest} -> {ok, Contents, Remainder} = consume_contents(Label, Rest, ""), {ok, {Label, Contents}, Remainder}; Error -> Error end. consume_label([$\s | Rest], Acc) -> consume_label(Rest, Acc); consume_label([$\t | Rest], Acc) -> consume_label(Rest, Acc); consume_label(String, Acc) -> consume_label_text(String, Acc). consume_label_text([$> | Rest], Acc) -> Label = lists:reverse(Acc), {ok, Label, Rest}; consume_label_text([$\s | Rest], Acc) -> consume_label_end(Rest, Acc); consume_label_text([$\t | Rest], Acc) -> consume_label_end(Rest, Acc); consume_label_text([Char | Rest], Acc) -> consume_label_text(Rest, [Char | Acc]); consume_label_text([], Acc) -> Label = lists:reverse(Acc), {error, Label}. consume_label_end([$/, $> | Rest], Acc) -> Label = lists:reverse(Acc), {ok, Label, Rest}; consume_label_end([$> | Rest], Acc) -> Label = lists:reverse(Acc), {ok, Label, Rest}; consume_label_end([_ | Rest], Acc) -> consume_label_end(Rest, Acc). consume_contents(Label, [$<, $/ | Rest], Acc) -> {ok, Label, Remainder} = consume_label(Rest, ""), Contents = lists:reverse(Acc), {ok, Contents, Remainder}; consume_contents(Label, [$< | Rest], Acc) -> {ok, Element, Remainder} = consume_element(Rest), consume_contents(Label, Remainder, [Element | Acc]); consume_contents(Label, [Char | Rest], Acc) -> consume_contents(Label, Rest, [Char | Acc]); consume_contents(_, "", Acc) -> Contents = lists:reverse(Acc), {ok, Contents}. And how will that work out, I wonder... 13> io:format("~tp~n", [htmler:consume("

Hello!

\n

How are you?

\n

Some text\n and more text.

")]). {ok,[{"h1","Hello!"}, 10,32,32,32,32,32, {"h2","How are you?"}, 10,32,32,32,32, {"p","Some text\n and more text."}]} ok How about with that nested tag case I was worried about earlier? 14> io:format("~tp~n", [htmler:consume("

Hello!

\n

How are you?

\n

Some text\n and more text.

")]). {ok,[{"h1","Hello!"}, 10,32,32,32,32,32, {"h2",[72,111,119,32,97,114,101,32,{"em","you?"}]}, 10,32,32,32,32, {"p","Some text\n and more text."}]} ok Ah! Nice. So it really does nest the "contents" list. So now we have a more semantically correct split happening, and it lets us descend down a tree of HTML tags -- assuming some really huge things that we can't normally assume on the big steaming pile of errors, exceptions and malformed output we call the web. For example: 2> htmler:consume("

Hello

foo"). ** exception error: no match of right hand side value {ok,"body",[]} in function htmler:consume_contents/3 (htmler.erl, line 66) in call from htmler:consume_element/1 (htmler.erl, line 28) in call from htmler:consume_contents/3 (htmler.erl, line 70) in call from htmler:consume_element/1 (htmler.erl, line 28) in call from htmler:consume/2 (htmler.erl, line 13) Whoops! Forgot the closing

tag, so everything catches fire. 3> htmler:consume("

Hello

foo

"). {ok,[{"body",[{"h1","Hello"},{"p","foo"}]}]} Ah, much better. Anyway, handling bad input in HTML is an art, not a science. This could be made dramatically more complex in the interest of accepting as much crap in the tubes as Firefox does -- or left simple and made to return explicit error messages with an indication what part of consuming the input failed (hint: this is a great way to get a headstart on creating a version that accepts bad input). (Also, of course, we would need to put in some cases to catch when the label is one of the implicitly "self-closing" tags, and explicitly self-closed tag, and a few other things to be complete.) Anyway, this doesn't exactly address the problem you have, but hopefully it gives you some ideas about how to parse angle-brackety syntaxes that manage to be both dramatically more complex and profoundly less useful than S-expressions. Regexs just don't cut it in this particular case. Merry Christmas! I hope you are having a good holiday with the family! -Craig From zxq9@REDACTED Sun Dec 25 06:38:47 2016 From: zxq9@REDACTED (zxq9) Date: Sun, 25 Dec 2016 14:38:47 +0900 Subject: [erlang-questions] How can I break this string into a list of strings? In-Reply-To: <6037061.ux5kc5t0N6@changa> References: <1482615262.7413280@apps.rackspace.com> <6037061.ux5kc5t0N6@changa> Message-ID: <3841178.UG7xHRC1Qf@changa> On 2016?12?25? ??? 14:21:00 zxq9 wrote: > On 2016?12?24? ??? 16:34:22 lloyd@REDACTED wrote: > > I would like to break it into a list: > > > > ["

Hello!

", "

How are you?

", "

Some text\n and more text.

"] > > While it is in my mind... we can use a variety of techniques to get output either exactly like, or perhaps more useful but very similar to the above based on the previously presented toy function: 5> {ok, Split} = htmler:consume("

Hello!

\n

How are you?

\n

Some text\n and more text.

"). {ok,[{"h1","Hello!"}, 10,32,32,32,32,32, {"h2","How are you?"}, 10,32,32,32,32, {"p","Some text\n and more text."}]} 6> [Contents || {_, Contents} <- Split]. ["Hello!","How are you?","Some text\n and more text."] Or even: 7> ["<" ++ Label ++ ">" ++ Contents ++ "" || {Label, Contents} <- Split]. ["

Hello!

","

How are you?

","

Some text\n and more text.

"] But seriously, why would I still want those tags in there at all? > 14> io:format("~tp~n", [htmler:consume("

Hello!

\n

How are you?

\n

Some text\n and more text.

")]). > {ok,[{"h1","Hello!"}, > 10,32,32,32,32,32, > {"h2",[72,111,119,32,97,114,101,32,{"em","you?"}]}, > 10,32,32,32,32, > {"p","Some text\n and more text."}]} This case is more interesting and won't work with a simple list comprehension to filter out the elements that are not tuples -- but an explicit function would do the trick. > 3> htmler:consume("

Hello

foo

"). > {ok,[{"body",[{"h1","Hello"},{"p","foo"}]}]} This last case is more like what you are going to actually be encountering in real HTML and XML docs -- and it is very similar to the case above, the real difference is that your "main list" that was returned is not wrapped in a tuple the way everything else is (well, that's not entirely true -- the function actually does return a tuple: {ok, Contents}. Maybe this could be leveraged to write a general pretty printing or interpretation function? Anyway, blahblah. I think you get the idea. Time for pumpkin pie! Wee! -Craig From eric.pailleau@REDACTED Sun Dec 25 08:14:10 2016 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Sun, 25 Dec 2016 08:14:10 +0100 Subject: [erlang-questions] How can I break this string into a list of strings? In-Reply-To: <1482635429.913432237@apps.rackspace.com> Message-ID: <96713c63-6e1f-45fb-9ce8-cebbf47ee192@email.android.com> An HTML attachment was scrubbed... URL: From technion@REDACTED Sun Dec 25 08:14:26 2016 From: technion@REDACTED (Technion) Date: Sun, 25 Dec 2016 07:14:26 +0000 Subject: [erlang-questions] SSL verification Message-ID: Hi, I was wondering if someone could clarify what I'm looking at. From the new_ssl page: http://erlang.org/documentation/doc-5.7.4/lib/ssl-3.10.7/doc/html/new_ssl.html If verify_none is specified x509-certificate path validation errors at the client side will not automatically cause the connection to fail, as it will if the verify type is verify_peer. See also the option verify_fun. Servers only do the path validation if verify_peer is set to true, as it then will send a certificate request to the client (this message is not sent if the verify option is verify_none) and you may then also want to specify the option fail_if_no_peer_cert. Based on the above, I would that if we do not specify, verify_none, the new_ssl library would be expected to verify the path. This doesn't seem to be the case: ssl:connect('self-signed.badssl.com', 443, [binary, {active, false}, {ssl_imp, new}], 5000). {ok,{sslsocket,{gen_tcp,#Port<0.43819>,tls_connection, undefined}, <0.203.0>}} Moreover, ssl errors that aren't related to verifying the peer's path still seem to come up fine by default: ssl:connect('wrong.host.badssl.com',443, [binary, {active, false}, {packet, raw}], 5000). {ok,{sslsocket,{gen_tcp,#Port<0.41371>,tls_connection, undefined}, <0.110.0>}} ssl:connect('expired.badssl.com', 443, [binary, {active, false}, {packet, raw}], 5000). {ok,{sslsocket,{gen_tcp,#Port<0.37428>,tls_connection, undefined}, <0.107.0>}} This sort of thing can lead to a lot of insecure SSL clients, even if users do have the ability to manually turn on a verifier. For example, the popular ibrowse library appears to pass this behaviour directly to the client: ibrowse:send_req("https://wrong.host.badssl.com",[], get). {ok,"200", ... With the only documented statement being "Can talk to secure webservers using SSL". And similarly with the "gun" library: 9> {ok, Pid2} = gun:open("expired.badssl.com", 443), 9> gun:await_up(Pid2). {ok,http} flush() ..etc There are two separate issues here: * Am I simply reading or implementing the new_ssl page incorrectly? Otherwise, I'd suggest the documentation could be considered bugged. I'd be happy to log a bug and propose alternate wording if this is the case. The ssl page provides big red warnings around BEAST mitigation. It would be odd to panic over BEAST yet accept self-signed certs. * Regardless, every major library I've looked at appears to happily accept self signed, expired, or otherwise invalid certs. I'd be happy to reach out to those vendors, but I'm wondering whether the community would feel there's a root cause here that should be addressed first, even if it's just a documentation fix? Any assistance here appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Sun Dec 25 12:44:20 2016 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Sun, 25 Dec 2016 12:44:20 +0100 Subject: [erlang-questions] How can I break this string into a list of strings? In-Reply-To: <96713c63-6e1f-45fb-9ce8-cebbf47ee192@email.android.com> References: <96713c63-6e1f-45fb-9ce8-cebbf47ee192@email.android.com> Message-ID: string:tokens(XMLLikeString, "<>"). > On 25 Dec 2016, at 08:14, ?ric Pailleau wrote: > > Hi, > > Have a look to re:split then. > > Something like "(<[^>]+>[^<]*<[^>]+>)" should do the job. Sorry I am not front of a computer right now, so could not test my proposal but may gives you a start anyway. > Oh oh oh ! > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From kennethlakin@REDACTED Sun Dec 25 15:20:26 2016 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Sun, 25 Dec 2016 06:20:26 -0800 Subject: [erlang-questions] SSL verification In-Reply-To: References: Message-ID: <4097938f-1399-63f6-8684-1ff2d6b71a70@gmail.com> On 12/24/2016 11:14 PM, Technion wrote: > I was wondering if someone could clarify what I'm looking at. From the new_ssl page: > > http://erlang.org/documentation/doc-5.7.4/lib/ssl-3.10.7/doc/html/new_ssl.html Are you using OTP R13* (released ~six years ago)? If you are, then please disregard the rest of this message. If you're not, then why are you consulting documentation for ssl 3.10.7? The current version is 8.1. The current documentation states that path validation is only performed when verify is set to verify_peer. The purpose of new_ssl is described as follows: "The new implementation is Erlang based and all logic is in Erlang and only payload encryption calculations are done in C via the crypto application." Given that the current ssl application seems to do everything but some of the crypto in Erlang, it would seem that new_ssl became the default long ago. Additionally, I can't find any files whose name containins the substring "new_ssl" in the current Erlang source tree. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From eric.pailleau@REDACTED Sun Dec 25 21:41:26 2016 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Sun, 25 Dec 2016 21:41:26 +0100 Subject: [erlang-questions] How can I break this string into a list of strings? In-Reply-To: Message-ID: An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Sun Dec 25 23:43:56 2016 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Sun, 25 Dec 2016 23:43:56 +0100 Subject: [erlang-questions] How can I break this string into a list of strings? In-Reply-To: References: Message-ID: <38A6C400-A369-4BE1-B1D2-E331BEE54227@gmail.com> Hey ?ric Well it's not clear whether tags are nested in the OP. So when my assumption applies, I would just read the list created 3 elements at a time, with strings of whitespace removed. If it doesn't, regexps are known to not be able to parse stack based grammars. > On 25 Dec 2016, at 21:41, ?ric Pailleau wrote: > > Hi Pierre, > > No this way you are loosing html tags. > Lloyd wants to keep html tags, including linefeeds inside html tags. > > I tested with re:split(Html, "(<[^>]+>[^<]*<[^>]+>)",[{return, list}]) . Works. > Lloyd can then loop on result and keep only non empty string after trim on elements. > Regards From zxq9@REDACTED Mon Dec 26 00:43:12 2016 From: zxq9@REDACTED (zxq9) Date: Mon, 26 Dec 2016 08:43:12 +0900 Subject: [erlang-questions] How can I break this string into a list of strings? In-Reply-To: <38A6C400-A369-4BE1-B1D2-E331BEE54227@gmail.com> References: <38A6C400-A369-4BE1-B1D2-E331BEE54227@gmail.com> Message-ID: <1625207.bQMHVKdYep@changa> On 2016?12?25? ??? 23:43:56 Pierre Fenoll wrote: > ...regexps are known to not be able to parse stack based grammars. There is even a famous post on the subject... http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags From technion@REDACTED Mon Dec 26 00:50:38 2016 From: technion@REDACTED (Technion) Date: Sun, 25 Dec 2016 23:50:38 +0000 Subject: [erlang-questions] SSL verification In-Reply-To: <4097938f-1399-63f6-8684-1ff2d6b71a70@gmail.com> References: , <4097938f-1399-63f6-8684-1ff2d6b71a70@gmail.com> Message-ID: Ugh.. I was reading that old document because it was the first hit on Google and I couldn't see a way to get the latest. That said, from the link you provided, here is the current wording under the "client side" heading: {verify, verify_type()} In mode verify_none the default behavior is to allow all x509-path validation errors. See also option verify_fun. I'm still reading that with a strong expectation that if I don't specify "verify_none", there will be verification performed. There is a discussion under "server side" which I believe is what you are quoting, where it refers to verifying client certificates, but if we are talking about ssl:connect we are not talking about server side. ________________________________ From: erlang-questions-bounces@REDACTED on behalf of Kenneth Lakin Sent: Monday, 26 December 2016 1:20 AM To: erlang-questions@REDACTED Subject: Re: [erlang-questions] SSL verification On 12/24/2016 11:14 PM, Technion wrote: > I was wondering if someone could clarify what I'm looking at. From the new_ssl page: > > http://erlang.org/documentation/doc-5.7.4/lib/ssl-3.10.7/doc/html/new_ssl.html new_ssl - Erlang erlang.org This manual page describes functions that are defined in the ssl module and represents the new ssl implementation that coexists with the old one, as the new ... Are you using OTP R13* (released ~six years ago)? If you are, then please disregard the rest of this message. If you're not, then why are you consulting documentation for ssl 3.10.7? The current version is 8.1. The current documentation states Erlang -- ssl erlang.org This module contains interface functions for the SSL/TLS protocol. For detailed information about the supported standards see ssl(6). that path validation is only performed when verify is set to verify_peer. The purpose of new_ssl is described as follows: "The new implementation is Erlang based and all logic is in Erlang and only payload encryption calculations are done in C via the crypto application." Given that the current ssl application seems to do everything but some of the crypto in Erlang, it would seem that new_ssl became the default long ago. Additionally, I can't find any files whose name containins the substring "new_ssl" in the current Erlang source tree. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennethlakin@REDACTED Mon Dec 26 01:27:58 2016 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Sun, 25 Dec 2016 16:27:58 -0800 Subject: [erlang-questions] SSL verification In-Reply-To: References: <4097938f-1399-63f6-8684-1ff2d6b71a70@gmail.com> Message-ID: <74d16587-e581-3b2a-bf72-cc2fe8b2d9e7@gmail.com> On 12/25/2016 03:50 PM, Technion wrote: > Ugh.. I was reading that old document because it was the first hit on > Google and I couldn't see a way to get the latest. The pattern for the module documentation is erlang.org/doc/man/$MODULE.html Some (most?) of the Erlang applications also have documentation: erlang.org/doc/man/$APP_app.html > There is a discussion under "server side" which I believe is what you > are quoting, where it refers to verifying client certificates, but if > we are talking about ssl:connect we are not talking about server side. > ... > {verify, verify_type()} > In mode verify_none the default behavior is to allow all > x509-path validation errors. See also option verify_fun. AFAIK, the default verify_fun is the same for both server and client operation. Notice how it's in the "COMMON for SERVER and CLIENT" section. > I'm still reading that with a strong expectation that if I don't specify > "verify_none", there will be verification performed. Odd. My expectation is that -unless I request it- verification will not be performed. I expect that I expect this because it seems more likely than not that your average TLS-equipped server will be using certs that won't validate (whether they be self-signed, expired, or simply not valid for the domain you're accessing). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From herr.hizel@REDACTED Mon Dec 26 10:23:05 2016 From: herr.hizel@REDACTED (Ildar Khizbulin) Date: Mon, 26 Dec 2016 12:23:05 +0300 Subject: [erlang-questions] Can i share nif resource between two nif module? Message-ID: Hi, can I transfer resource from one nif to another nif? For example in first, i write: capture.c: static ErlNifResourceType *f_resource; capture_f() { frame_t* f = enif_alloc_resource(f_resource, sizeof(frame_t)); f_term = enif_make_resource_binary(env, f, .... ret = enif_make_tuple2(env, ok_atom, f_term); enif_release_resource(f); return ret; } load() { if(!f_resource) f_resource = enif_open_resource_type(env, NULL, "f_res", flags, NULL); } in second hi.c: static ErlNifResourceType *f_resource; process_f() { struct frame_t *f; if (!enif_get_resource(env, argv[0], f_resource, (void **)&f)) { ///<- FAIL } load() { if(!f_resource) f_resource = enif_open_resource_type(env, NULL, "f_res", flags, NULL); } logic.erl {ok, Bin} = capture:capture_f(), hi:process_f(Bin) % <- fail From vinoski@REDACTED Mon Dec 26 18:44:43 2016 From: vinoski@REDACTED (Steve Vinoski) Date: Mon, 26 Dec 2016 12:44:43 -0500 Subject: [erlang-questions] Can i share nif resource between two nif module? In-Reply-To: References: Message-ID: On Mon, Dec 26, 2016 at 4:23 AM, Ildar Khizbulin wrote: > Hi, > > can I transfer resource from one nif to another nif? > > For example in first, i write: > > capture.c: > > static ErlNifResourceType *f_resource; > > capture_f() { > frame_t* f = enif_alloc_resource(f_resource, sizeof(frame_t)); > f_term = enif_make_resource_binary(env, f, .... > ret = enif_make_tuple2(env, ok_atom, f_term); > enif_release_resource(f); > return ret; > } > > load() { > if(!f_resource) > f_resource = enif_open_resource_type(env, NULL, "f_res", flags, NULL); } > > in second > > hi.c: > > static ErlNifResourceType *f_resource; > > process_f() { > struct frame_t *f; > if (!enif_get_resource(env, argv[0], f_resource, (void **)&f)) { ///<- > FAIL > > > } > > load() { > if(!f_resource) > f_resource = enif_open_resource_type(env, NULL, "f_res", flags, NULL); > Are you including ERL_NIF_RT_TAKEOVER in your flags value here? --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From herr.hizel@REDACTED Mon Dec 26 19:41:03 2016 From: herr.hizel@REDACTED (Ildar Khizbulin) Date: Mon, 26 Dec 2016 21:41:03 +0300 Subject: [erlang-questions] Can i share nif resource between two nif module? In-Reply-To: References: Message-ID: Yep, including. I create simple example on https://github.com/hizel/enif_test with full code. From vinoski@REDACTED Mon Dec 26 22:28:24 2016 From: vinoski@REDACTED (Steve Vinoski) Date: Mon, 26 Dec 2016 16:28:24 -0500 Subject: [erlang-questions] Can i share nif resource between two nif module? In-Reply-To: References: Message-ID: On Mon, Dec 26, 2016 at 1:41 PM, Ildar Khizbulin wrote: > Yep, including. I create simple example on > https://github.com/hizel/enif_test with full code. > What you're trying to do doesn't appear to be supported currently. The Erlang nif internals require the module name argument (second argument) of enif_open_resource_type() to be a null pointer (see http://erlang.org/doc/man/erl_nif.html#enif_open_resource_type). It generates an atom identifier for the resource type based on the calling module. You can see that code here: https://github.com/erlang/otp/blob/maint/erts/emulator/beam/erl_nif.c#L2012-L2013 As a result, when your second module tries to take over the resource type, the enif_open_resource_type() implementation fails to find the resource you created in your first module, and so it ends up creating a new resource type instead. When you try to use this new type in your second module to retrieve the resource binary argument created by the first module, you get badarg because the two resource types are different types. --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewdpb@REDACTED Mon Dec 26 23:43:15 2016 From: ewdpb@REDACTED (=?UTF-8?Q?Wilmar_P=C3=A9rez?=) Date: Mon, 26 Dec 2016 17:43:15 -0500 Subject: [erlang-questions] OpenFlow Framework In-Reply-To: <47438874.223421.1482483449877.JavaMail.zimbra@tpip.net> References: <47438874.223421.1482483449877.JavaMail.zimbra@tpip.net> Message-ID: Hello Andreas Thanks for the reply. I saw FlowER but since I didn't see any update to you for the last two or three years I assumed it was abandoned. I am just learning my way through Erlang, I don't think I would contribute with anything meaningful. I was only able to find an older presentation and a slide share deck on FlowER. Is there any other information available? Best regards, Wilmar. 2016-12-23 3:57 GMT-05:00 Andreas Schultz : > Hi Wilmar, > > I guess you have seen flower [1], it is not really abandoned, but we don't > really use it actively at the moment. > > If you have any questions or would like to contribute changes, let me know. > > Regards > Andreas > > [1]: https://github.com/travelping/flower > > ------------------------------ > > *From: *"Wilmar P?rez" > *To: *"erlang-questions" > *Sent: *Thursday, December 22, 2016 4:16:50 PM > *Subject: *[erlang-questions] OpenFlow Framework > > Hello > I have been tasked with finding an efficient and hopefully mature OpenFlow > Controller. I like the idea of an Erlang OpenFlow controller but it seems > like all projects are abandoned. Is there any that is still being actively > developed? > > Thanks > > Wilmar > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sweden.feng@REDACTED Tue Dec 27 10:45:06 2016 From: sweden.feng@REDACTED (Alex Feng) Date: Tue, 27 Dec 2016 10:45:06 +0100 Subject: [erlang-questions] How to avoid long_schedule issue ? Message-ID: Hi, I was reading some documents about scheduler, to my understanding there will be no long_schedule issue since BIF has trap mechanism. Then I found this, I couldn't understand, the reduction is keeping increasing, why the BIF(lists:seq) is not interrupted for more than 10ms ? Could someone please explain this ? 1> erlang:system_monitor(self(),[{long_schedule, 10}]). undefined 2> Pid = spawn(fun()->lists:seq(1,2000000000) end). <0.65.0> 3>flush(). Shell got {monitor,<0.65.0>,long_schedule, [{timeout,21}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,38}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,17}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,21}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,23}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,31}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,30}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,43}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,53}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,56}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,51}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,81}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,17}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,18}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,110}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,74}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,24}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,76}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,112}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,43}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,130}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,205}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} Shell got {monitor,<0.65.0>,long_schedule, [{timeout,77}, {in,{lists,seq_loop,3}}, {out,{lists,seq_loop,3}}]} 20> process_info(Pid, reductions). {reductions,4166000} 21> process_info(Pid, reductions). {reductions,7150000} 22> process_info(Pid, reductions). {reductions,8328000} 23> process_info(Pid, reductions). {reductions,12330000} 24> process_info(Pid, reductions). {reductions,14354000} 25> process_info(Pid, reductions). {reductions,24774000} Br, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Wed Dec 28 02:19:26 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Wed, 28 Dec 2016 01:19:26 +0000 Subject: [erlang-questions] {active, N} to build an echo TCP server? Message-ID: Hi guys While able to build an echo TCP server which uses {active, once} and works as expected, I?m unable to design the same echo server using {active, N}. Can someone help me understand how to use it? How and where to decrement the N? What happens when N reaches 0? Thank you. /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Dec 28 05:19:55 2016 From: zxq9@REDACTED (zxq9) Date: Wed, 28 Dec 2016 13:19:55 +0900 Subject: [erlang-questions] {active, N} to build an echo TCP server? In-Reply-To: References: Message-ID: <4259049.pMQ17oP7et@changa> On 2016?12?28? ??? 01:19:26 Frank Muller wrote: > Hi guys > > While able to build an echo TCP server which uses {active, once} and works > as expected, > I?m unable to design the same echo server using {active, N}. > > Can someone help me understand how to use it? > How and where to decrement the N? > What happens when N reaches 0? Hi, Frank. I don't know any better way to explain this than with an example. Below is an echo server that can talk to a telnet client. There are two versions in this module. The first is an {active, once} version that closes when it receives "bye\r\n" from the client or the connection is closed. No surprises there. The second is an {active, Count} version that gets stuck when it runs through its count and notifies the client of the situation. This version can accept an Erlang message to its process to reset the count. -module(echoserve). -export([start/1, start/2]). %%% The {active, once} version start(PortNum) -> spawn_link(fun() -> listen(PortNum) end). listen(PortNum) -> {ok, Listener} = gen_tcp:listen(PortNum, [{active, false}, {mode, list}]), accept(Listener). accept(Listener) -> {ok, Socket} = gen_tcp:accept(Listener), loop(Socket). loop(Socket) -> ok = inet:setopts(Socket, [{active, once}]), receive {tcp, Socket, "bye\r\n"} -> ok = io:format("~p: Client is retiring.~n", [self()]), ok = gen_tcp:send(Socket, "Bye!\r\n"), ok = gen_tcp:shutdown(Socket, read_write), exit(normal); {tcp, Socket, Data} -> ok = io:format("~p: Received ~tp~n", [self(), Data]), Message = ["You sent ", Data], ok = gen_tcp:send(Socket, Message), loop(Socket); {tcp_closed, Socket} -> ok = io:format("~p: Connection closed, retiring.~n", [self()]), exit(normal); Unexpected -> ok = io:format("~p: Unexpected message ~tp. Retiring.~n", [self(), Unexpected]), exit(normal) end. %%% The {active, Count} version start(PortNum, ActiveCount) -> spawn_link(fun() -> listen(PortNum, ActiveCount) end). listen(PortNum, Count) -> {ok, Listener} = gen_tcp:listen(PortNum, [{active, false}, {mode, list}]), accept(Listener, Count). accept(Listener, Count) -> {ok, Socket} = gen_tcp:accept(Listener), ok = inet:setopts(Socket, [{active, Count}]), StringCount = integer_to_list(Count), Greeting = ["Hi! I am a telnet echo server\r\n" "I will only respond to you ", StringCount, " times unless " "the operator sends me a {set, Count} message.\r\n"], ok = gen_tcp:send(Socket, Greeting), countdown_loop(Socket). countdown_loop(Socket) -> receive {tcp, Socket, "bye\r\n"} -> ok = io:format("~p: Client is retiring.~n", [self()]), ok = gen_tcp:send(Socket, "Bye!\r\n"), ok = gen_tcp:shutdown(Socket, read_write), exit(normal); {tcp, Socket, Data} -> ok = io:format("~p: Received ~tp~n", [self(), Data]), Message = ["You sent ", Data], ok = gen_tcp:send(Socket, Message), countdown_loop(Socket); {tcp_passive, Socket} -> Message = "Oh noes! I've just run out of responses. Whatever you send will be " "buffered until the operator sends me a {set, Count} message!\r\n" "This also means that I won't respond to a \"bye\" message from you " "until I am listening again, and only if the buffer is clear before you " "send that message.\r\n", ok = gen_tcp:send(Socket, Message), countdown_loop(Socket); {tcp_closed, Socket} -> ok = io:format("~p: Connection closed, retiring.~n", [self()]), exit(normal); {set, Count} -> StringCount = integer_to_list(Count), Message = ["The operator set me to respond to ", StringCount, " messages from now!\r\n"], ok = gen_tcp:send(Socket, Message), ok = inet:setopts(Socket, [{active, Count}]), countdown_loop(Socket); Unexpected -> ok = io:format("~p: Unexpected message ~tp. Retiring.~n", [self(), Unexpected]), exit(normal) end. Hopefully your email client didn't choke on my too-long lines -- I sort of scribbled this out in a hurry. Anyway, let's run the first one and see what happens... Server side: 1> c(echoserve). {ok,echoserve} 2> echoserve:start(7777). <0.64.0> <0.64.0>: Received "foo\r\n" <0.64.0>: Received "bar\r\n" <0.64.0>: Received "baz\r\n" <0.64.0>: Client is retiring. Client side: ceverett@REDACTED:~/Code/erlang$ telnet localhost 7777 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. foo You sent foo bar You sent bar baz You sent baz bye Bye! Connection closed by foreign host. Pretty much what you would expect. Now let's try the second version... Server side: 3> echoserve:start(7777, 3). <0.66.0> <0.66.0>: Received "foo\r\n" <0.66.0>: Received "bar\r\n" <0.66.0>: Received "baz\r\n" 4> pid(0,66,0) ! {set, 3}. <0.66.0>: Received "bye\r\nwhat?\r\nWhy won't you close?!?\r\n" {set,3} <0.66.0>: Client is retiring. Client side: ceverett@REDACTED:~/Code/erlang$ telnet localhost 7777 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Hi! I am a telnet echo server I will only respond to you 3 times unless the operator sends me a {set, Count} message. foo You sent foo bar You sent bar baz You sent baz Oh noes! I've just run out of responses. Whatever you send will be buffered until the operator sends me a {set, Count} message! This also means that I won't respond to a "bye" message from you until I am listening again, and only if the buffer is clear before you send that message. bye what? Why won't you close?!? The operator set me to respond to 3 messages from now! You sent bye what? Why won't you close?!? bye Bye! Connection closed by foreign host. So this was a little bit different. Take your time reading over what happened in the second exchange between the client and server. I don't really have a good way to display this histographically in text here (at least not without spending a lot of time doing it). I might make a blog post out of this later. Note that the "bye\r\n" sent by the client did not match the "bye\r\n" clause in the receive of countdown_loop/1. Why not? Because TCP is a stream, not a datagram. What was received by the server was: "bye\r\nwhat?\r\nWhy won't you close?!?\r\n" It was squished together because that's just how much stuff was in the buffer by the time the socket was read from again. The moral of the story? Don't ever fool yourself into thinking you're dealing with datagrams. Also note that the code above represents only a single process as the listener and socket handler. The concurrent version of this is pretty similar, though. I hope I explained more than confused. -Craig From frank.muller.erl@REDACTED Wed Dec 28 06:14:10 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Wed, 28 Dec 2016 05:14:10 +0000 Subject: [erlang-questions] {active, N} to build an echo TCP server? In-Reply-To: <4259049.pMQ17oP7et@changa> References: <4259049.pMQ17oP7et@changa> Message-ID: Hi Craig Exactly what I wanted to know. Thank you for the clear explanations. One last question: how one can determine the best value of N? (i.e not too big to avoid overwhelming the server, not too small to avoid N close to 1. /Frank Le mer. 28 d?c. 2016 ? 05:20, zxq9 a ?crit : > On 2016?12?28? ??? 01:19:26 Frank Muller wrote: > > > Hi guys > > > > > > While able to build an echo TCP server which uses {active, once} and > works > > > as expected, > > > I?m unable to design the same echo server using {active, N}. > > > > > > Can someone help me understand how to use it? > > > How and where to decrement the N? > > > What happens when N reaches 0? > > > > Hi, Frank. > > > > I don't know any better way to explain this than with an example. > > Below is an echo server that can talk to a telnet client. > > There are two versions in this module. > > > > The first is an {active, once} version that closes when it receives > > "bye\r\n" from the client or the connection is closed. No surprises > > there. > > > > The second is an {active, Count} version that gets stuck when it > > runs through its count and notifies the client of the situation. > > This version can accept an Erlang message to its process to reset > > the count. > > > > > > > > -module(echoserve). > > -export([start/1, start/2]). > > > > > > %%% The {active, once} version > > > > start(PortNum) -> > > spawn_link(fun() -> listen(PortNum) end). > > > > > > listen(PortNum) -> > > {ok, Listener} = gen_tcp:listen(PortNum, [{active, false}, {mode, > list}]), > > accept(Listener). > > > > > > accept(Listener) -> > > {ok, Socket} = gen_tcp:accept(Listener), > > loop(Socket). > > > > > > loop(Socket) -> > > ok = inet:setopts(Socket, [{active, once}]), > > receive > > {tcp, Socket, "bye\r\n"} -> > > ok = io:format("~p: Client is retiring.~n", [self()]), > > ok = gen_tcp:send(Socket, "Bye!\r\n"), > > ok = gen_tcp:shutdown(Socket, read_write), > > exit(normal); > > {tcp, Socket, Data} -> > > ok = io:format("~p: Received ~tp~n", [self(), Data]), > > Message = ["You sent ", Data], > > ok = gen_tcp:send(Socket, Message), > > loop(Socket); > > {tcp_closed, Socket} -> > > ok = io:format("~p: Connection closed, retiring.~n", [self()]), > > exit(normal); > > Unexpected -> > > ok = io:format("~p: Unexpected message ~tp. Retiring.~n", > [self(), Unexpected]), > > exit(normal) > > end. > > > > > > > > %%% The {active, Count} version > > > > start(PortNum, ActiveCount) -> > > spawn_link(fun() -> listen(PortNum, ActiveCount) end). > > > > > > listen(PortNum, Count) -> > > {ok, Listener} = gen_tcp:listen(PortNum, [{active, false}, {mode, > list}]), > > accept(Listener, Count). > > > > > > accept(Listener, Count) -> > > {ok, Socket} = gen_tcp:accept(Listener), > > ok = inet:setopts(Socket, [{active, Count}]), > > StringCount = integer_to_list(Count), > > Greeting = ["Hi! I am a telnet echo server\r\n" > > "I will only respond to you ", StringCount, " times unless > " > > "the operator sends me a {set, Count} message.\r\n"], > > ok = gen_tcp:send(Socket, Greeting), > > countdown_loop(Socket). > > > > > > countdown_loop(Socket) -> > > receive > > {tcp, Socket, "bye\r\n"} -> > > ok = io:format("~p: Client is retiring.~n", [self()]), > > ok = gen_tcp:send(Socket, "Bye!\r\n"), > > ok = gen_tcp:shutdown(Socket, read_write), > > exit(normal); > > {tcp, Socket, Data} -> > > ok = io:format("~p: Received ~tp~n", [self(), Data]), > > Message = ["You sent ", Data], > > ok = gen_tcp:send(Socket, Message), > > countdown_loop(Socket); > > {tcp_passive, Socket} -> > > Message = "Oh noes! I've just run out of responses. Whatever > you send will be " > > "buffered until the operator sends me a {set, Count} > message!\r\n" > > "This also means that I won't respond to a \"bye\" > message from you " > > "until I am listening again, and only if the buffer > is clear before you " > > "send that message.\r\n", > > ok = gen_tcp:send(Socket, Message), > > countdown_loop(Socket); > > {tcp_closed, Socket} -> > > ok = io:format("~p: Connection closed, retiring.~n", [self()]), > > exit(normal); > > {set, Count} -> > > StringCount = integer_to_list(Count), > > Message = ["The operator set me to respond to ", StringCount, > " messages from now!\r\n"], > > ok = gen_tcp:send(Socket, Message), > > ok = inet:setopts(Socket, [{active, Count}]), > > countdown_loop(Socket); > > Unexpected -> > > ok = io:format("~p: Unexpected message ~tp. Retiring.~n", > [self(), Unexpected]), > > exit(normal) > > end. > > > > > > > > > > Hopefully your email client didn't choke on my too-long lines -- I sort > > of scribbled this out in a hurry. > > > > Anyway, let's run the first one and see what happens... > > > > Server side: > > > > 1> c(echoserve). > > {ok,echoserve} > > 2> echoserve:start(7777). > > <0.64.0> > > <0.64.0>: Received "foo\r\n" > > <0.64.0>: Received "bar\r\n" > > <0.64.0>: Received "baz\r\n" > > <0.64.0>: Client is retiring. > > > > > > Client side: > > > > ceverett@REDACTED:~/Code/erlang$ telnet localhost 7777 > > Trying 127.0.0.1... > > Connected to localhost. > > Escape character is '^]'. > > foo > > You sent foo > > bar > > You sent bar > > baz > > You sent baz > > bye > > Bye! > > Connection closed by foreign host. > > > > > > Pretty much what you would expect. > > > > Now let's try the second version... > > > > Server side: > > > > 3> echoserve:start(7777, 3). > > <0.66.0> > > <0.66.0>: Received "foo\r\n" > > <0.66.0>: Received "bar\r\n" > > <0.66.0>: Received "baz\r\n" > > 4> pid(0,66,0) ! {set, 3}. > > <0.66.0>: Received "bye\r\nwhat?\r\nWhy won't you close?!?\r\n" > > {set,3} > > <0.66.0>: Client is retiring. > > > > > > Client side: > > > > ceverett@REDACTED:~/Code/erlang$ telnet localhost 7777 > > Trying 127.0.0.1... > > Connected to localhost. > > Escape character is '^]'. > > Hi! I am a telnet echo server > > I will only respond to you 3 times unless the operator sends me a {set, > Count} message. > > foo > > You sent foo > > bar > > You sent bar > > baz > > You sent baz > > Oh noes! I've just run out of responses. Whatever you send will be > buffered until the operator sends me a {set, Count} message! > > This also means that I won't respond to a "bye" message from you until I > am listening again, and only if the buffer is clear before you send that > message. > > bye > > what? > > Why won't you close?!? > > The operator set me to respond to 3 messages from now! > > You sent bye > > what? > > Why won't you close?!? > > bye > > Bye! > > Connection closed by foreign host. > > > > > > So this was a little bit different. Take your time reading over what > happened > > in the second exchange between the client and server. I don't really have a > > good way to display this histographically in text here (at least not > without > > spending a lot of time doing it). I might make a blog post out of this > later. > > > > Note that the "bye\r\n" sent by the client did not match the "bye\r\n" > clause > > in the receive of countdown_loop/1. Why not? Because TCP is a stream, not a > > datagram. What was received by the server was: > > "bye\r\nwhat?\r\nWhy won't you close?!?\r\n" > > > > It was squished together because that's just how much stuff was in the > buffer > > by the time the socket was read from again. The moral of the story? Don't > ever > > fool yourself into thinking you're dealing with datagrams. > > > > Also note that the code above represents only a single process as the > listener > > and socket handler. The concurrent version of this is pretty similar, > though. > > > > I hope I explained more than confused. > > > > -Craig > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennethlakin@REDACTED Wed Dec 28 06:27:30 2016 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Tue, 27 Dec 2016 21:27:30 -0800 Subject: [erlang-questions] {active, N} to build an echo TCP server? In-Reply-To: References: Message-ID: On 12/27/2016 05:19 PM, Frank Muller wrote: > I?m unable to design the same echo server using {active, N}. > > Can someone help me understand how to use it? > How and where to decrement the N? > What happens when N reaches 0? This might be helpful: http://erlang.org/doc/man/inet.html#setopts-2 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From zxq9@REDACTED Wed Dec 28 07:08:56 2016 From: zxq9@REDACTED (zxq9) Date: Wed, 28 Dec 2016 15:08:56 +0900 Subject: [erlang-questions] {active, N} to build an echo TCP server? In-Reply-To: References: <4259049.pMQ17oP7et@changa> Message-ID: <7814820.6stdUKnqEK@changa> On 2016?12?28? ??? 05:14:10 you wrote: > Hi Craig > > Exactly what I wanted to know. Thank you for the clear explanations. > > One last question: how one can determine the best value of N? (i.e not too > big to avoid overwhelming the server, not too small to avoid N close to 1. tl;dr: {active, once} is sufficient for every case you are likely to ever encounter; departure from The True Path will lead you on a descent into madness and undiagnosable weirdness. Discussion: This is a bit tricky. Actually, I think I have only ever used {active, N} as single time -- and its use was a cheap hack based on some overly intimate knowledge of the sender, which is *not* a good way to build networked things (and computers are so insanely fast and getting ever faster as we go... this sort of thing is just not called for). So don't do this based on your knowledge of the sender, it will just make your hair catch fire some day -- "reap what you sow" and all that. Because TCP is a stream you can never know how much data will be in the buffer so the consumption loop will have to be built in such a way that it can correctly interpret partial, complete, and overrun inputs (by "overrun" in this case I mean that you receive part of the next message in a receive). If the message is "I am a message\r\n" you may receive that whole thing in one message, or receive it as a series like "I am ", then "a mess" and finally "age\r\nSurprise! I am the next one\r\n". This is also true of whatever protocol you are using. Almost any standard protocol will have a way to either disambiguate the size of the message or have a trailing delimiter (like telnet, for example, delimits messages per line, so "\r\n" is the end of a single message -- you read through the input until you see "\r\n"). Some protocols will pack the beginning of a message with data you might want to interpret before you read the rest of the stream. For example, HTTP/1.1 does this with a header that tells the message size (but indicates the boundary between the end of the headers, which are of unknown size, and the body that you are told the size of with "\r\n\r\n"). Nearly every binary protocol goes one better by packing the total size of the expected message with a fixed-size field at the front that tells you how many bytes to expect -- so most binary protocols are very easy to interpret in Erlang with something like <> and a few additional function clauses that match on variations of that + "remaining length" checking + concatenating to the buffer as partials are received. [Digression: I have a sort of canonical generic behavior for dealing with this we used at Tsuriai (and now elsewhere) which made the protocol definition itself a callback module (or in cases where the callback overhead was an issue, simply making the protocol handling functions in-module definitions, so that the binary interpreter and loop is basically boilerplate). I'm actually sort of surprised something like this isn't a generic OTP behavior, like gen_protocol or something.] In none of these cases have I ever found it useful to do anything other than recv from a passive socket or (usually better) receive erlang messages from an {active, once} socket, resetting the socket active state at the top of the receiving loop. Setting the socket state is very fast, and there is almost never a case where I want "N receives of unknown size" because that just gives me an aggregate of unknown size. This naturally provides pushback on the sender based on whatever may be slowing the interpretation or handling of the received messages without requiring any more consideration. As a side note, I find socket handling code inside of gen_* modules to usually be horribly ugly and often confusing to read if the sole purpose of the process is to handle a socket. When combined with other activities it might make sense, but that is usually not the case -- normally socket handlers handle sockets, other stuff does other stuff (they usually provide a 1::1 abstraction for whatever is on the other end of the connection). Anyway, it is fairly common to see socket handlers written using proc_lib, probably for the same reason I tend to do it. Or maybe that's just a trend that exists only here in Japan and we're techno outcasts living on the fringe of opinion. From frank.muller.erl@REDACTED Wed Dec 28 08:27:19 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Wed, 28 Dec 2016 07:27:19 +0000 Subject: [erlang-questions] {active, N} to build an echo TCP server? In-Reply-To: <7814820.6stdUKnqEK@changa> References: <4259049.pMQ17oP7et@changa> <7814820.6stdUKnqEK@changa> Message-ID: Thanks again Craig. I'll stick with {active, once} because I've no knowledge on how fast the client can be. /Frank Le mer. 28 d?c. 2016 ? 07:09, zxq9 a ?crit : > On 2016?12?28? ??? 05:14:10 you wrote: > > > Hi Craig > > > > > > Exactly what I wanted to know. Thank you for the clear explanations. > > > > > > One last question: how one can determine the best value of N? (i.e not > too > > > big to avoid overwhelming the server, not too small to avoid N close to > 1. > > > > tl;dr: > > > > {active, once} is sufficient for every case you are likely to ever > encounter; departure from The True Path will lead you on a descent into > madness and undiagnosable weirdness. > > > > > > Discussion: > > > > This is a bit tricky. Actually, I think I have only ever used {active, N} > as single time -- and its use was a cheap hack based on some overly > intimate knowledge of the sender, which is *not* a good way to build > networked things (and computers are so insanely fast and getting ever > faster as we go... this sort of thing is just not called for). So don't do > this based on your knowledge of the sender, it will just make your hair > catch fire some day -- "reap what you sow" and all that. > > > > Because TCP is a stream you can never know how much data will be in the > buffer so the consumption loop will have to be built in such a way that it > can correctly interpret partial, complete, and overrun inputs (by "overrun" > in this case I mean that you receive part of the next message in a > receive). If the message is "I am a message\r\n" you may receive that whole > thing in one message, or receive it as a series like "I am ", then "a mess" > and finally "age\r\nSurprise! I am the next one\r\n". This is also true of > whatever protocol you are using. Almost any standard protocol will have a > way to either disambiguate the size of the message or have a trailing > delimiter (like telnet, for example, delimits messages per line, so "\r\n" > is the end of a single message -- you read through the input until you see > "\r\n"). > > > > Some protocols will pack the beginning of a message with data you might > want to interpret before you read the rest of the stream. For example, > HTTP/1.1 does this with a header that tells the message size (but indicates > the boundary between the end of the headers, which are of unknown size, and > the body that you are told the size of with "\r\n\r\n"). Nearly every > binary protocol goes one better by packing the total size of the expected > message with a fixed-size field at the front that tells you how many bytes > to expect -- so most binary protocols are very easy to interpret in Erlang > with something like <> and a > few additional function clauses that match on variations of that + > "remaining length" checking + concatenating to the buffer as partials are > received. > > > > [Digression: I have a sort of canonical generic behavior for dealing with > this we used at Tsuriai (and now elsewhere) which made the protocol > definition itself a callback module (or in cases where the callback > overhead was an issue, simply making the protocol handling functions > in-module definitions, so that the binary interpreter and loop is basically > boilerplate). I'm actually sort of surprised something like this isn't a > generic OTP behavior, like gen_protocol or something.] > > > > In none of these cases have I ever found it useful to do anything other > than recv from a passive socket or (usually better) receive erlang messages > from an {active, once} socket, resetting the socket active state at the top > of the receiving loop. Setting the socket state is very fast, and there is > almost never a case where I want "N receives of unknown size" because that > just gives me an aggregate of unknown size. This naturally provides > pushback on the sender based on whatever may be slowing the interpretation > or handling of the received messages without requiring any more > consideration. > > > > As a side note, I find socket handling code inside of gen_* modules to > usually be horribly ugly and often confusing to read if the sole purpose of > the process is to handle a socket. When combined with other activities it > might make sense, but that is usually not the case -- normally socket > handlers handle sockets, other stuff does other stuff (they usually provide > a 1::1 abstraction for whatever is on the other end of the connection). > Anyway, it is fairly common to see socket handlers written using proc_lib, > probably for the same reason I tend to do it. Or maybe that's just a trend > that exists only here in Japan and we're techno outcasts living on the > fringe of opinion. > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Oliver.Korpilla@REDACTED Wed Dec 28 10:37:45 2016 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Wed, 28 Dec 2016 10:37:45 +0100 Subject: [erlang-questions] {active, N} to build an echo TCP server? In-Reply-To: <7814820.6stdUKnqEK@changa> References: <4259049.pMQ17oP7et@changa> , <7814820.6stdUKnqEK@changa> Message-ID: Hello. ? Can you give an example of what this gen_protocol could look like in use? ? Thanks, Oliver ? Gesendet:?Mittwoch, 28. Dezember 2016 um 07:08 Uhr Von:?zxq9 An:?erlang-questions@REDACTED Betreff:?Re: [erlang-questions] {active, N} to build an echo TCP server? On 2016?12?28? ??? 05:14:10 you wrote: > Hi Craig > > Exactly what I wanted to know. Thank you for the clear explanations. > > One last question: how one can determine the best value of N? (i.e not too > big to avoid overwhelming the server, not too small to avoid N close to 1. tl;dr: {active, once} is sufficient for every case you are likely to ever encounter; departure from The True Path will lead you on a descent into madness and undiagnosable weirdness. Discussion: This is a bit tricky. Actually, I think I have only ever used {active, N} as single time -- and its use was a cheap hack based on some overly intimate knowledge of the sender, which is *not* a good way to build networked things (and computers are so insanely fast and getting ever faster as we go... this sort of thing is just not called for). So don't do this based on your knowledge of the sender, it will just make your hair catch fire some day -- "reap what you sow" and all that. Because TCP is a stream you can never know how much data will be in the buffer so the consumption loop will have to be built in such a way that it can correctly interpret partial, complete, and overrun inputs (by "overrun" in this case I mean that you receive part of the next message in a receive). If the message is "I am a message\r\n" you may receive that whole thing in one message, or receive it as a series like "I am ", then "a mess" and finally "age\r\nSurprise! I am the next one\r\n". This is also true of whatever protocol you are using. Almost any standard protocol will have a way to either disambiguate the size of the message or have a trailing delimiter (like telnet, for example, delimits messages per line, so "\r\n" is the end of a single message -- you read through the input until you see "\r\n"). Some protocols will pack the beginning of a message with data you might want to interpret before you read the rest of the stream. For example, HTTP/1.1 does this with a header that tells the message size (but indicates the boundary between the end of the headers, which are of unknown size, and the body that you are told the size of with "\r\n\r\n"). Nearly every binary protocol goes one better by packing the total size of the expected message with a fixed-size field at the front that tells you how many bytes to expect -- so most binary protocols are very easy to interpret in Erlang with something like <> and a few additional function clauses that match on variations of that + "remaining length" checking + concatenating to the buffer as partials are received. [Digression: I have a sort of canonical generic behavior for dealing with this we used at Tsuriai (and now elsewhere) which made the protocol definition itself a callback module (or in cases where the callback overhead was an issue, simply making the protocol handling functions in-module definitions, so that the binary interpreter and loop is basically boilerplate). I'm actually sort of surprised something like this isn't a generic OTP behavior, like gen_protocol or something.] In none of these cases have I ever found it useful to do anything other than recv from a passive socket or (usually better) receive erlang messages from an {active, once} socket, resetting the socket active state at the top of the receiving loop. Setting the socket state is very fast, and there is almost never a case where I want "N receives of unknown size" because that just gives me an aggregate of unknown size. This naturally provides pushback on the sender based on whatever may be slowing the interpretation or handling of the received messages without requiring any more consideration. As a side note, I find socket handling code inside of gen_* modules to usually be horribly ugly and often confusing to read if the sole purpose of the process is to handle a socket. When combined with other activities it might make sense, but that is usually not the case -- normally socket handlers handle sockets, other stuff does other stuff (they usually provide a 1::1 abstraction for whatever is on the other end of the connection). Anyway, it is fairly common to see socket handlers written using proc_lib, probably for the same reason I tend to do it. Or maybe that's just a trend that exists only here in Japan and we're techno outcasts living on the fringe of opinion. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From max.lapshin@REDACTED Wed Dec 28 16:23:14 2016 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 28 Dec 2016 18:23:14 +0300 Subject: [erlang-questions] How to avoid long_schedule issue ? In-Reply-To: References: Message-ID: I'm also very interested in how to properly interpret these warnings =( -------------- next part -------------- An HTML attachment was scrubbed... URL: From Oliver.Korpilla@REDACTED Wed Dec 28 16:52:45 2016 From: Oliver.Korpilla@REDACTED (Oliver Korpilla) Date: Wed, 28 Dec 2016 16:52:45 +0100 Subject: [erlang-questions] appup: How to handle processes that need no immediate upgrade? Message-ID: Hello. ? This may seem a very specific and weird question, but it would help me understand OTP code upgrades so much better! ? I have two kinds of processes in my system: * Permanent workers and supervisors that live on and on. * Dynamic/transient workers that are created per initial request, handle a limited messaging scenario covering a few exchanges, and "die." ? I would be perfectly happy if the dynamic workers did not participate in the hot code upgrade since they could simply run to completion and die. I would however want to have new ones be started with the new version of their respective function or callback modules. ? Is that possible or even desirable? ? I can see how this would not work if I have to transform my DB. In that case I would need the new version of the module to call only into the new version of the DB. (But as long as I only add fields, neither rename nor delete any, even the old code could work to satisfaction, I guess?) ? What happens if I supply a new version of a module for transient workers without a "load_module" in the appup file? ? Thank you and regards, Oliver From khitai.pang@REDACTED Wed Dec 28 21:52:53 2016 From: khitai.pang@REDACTED (Khitai Pang) Date: Wed, 28 Dec 2016 20:52:53 +0000 Subject: [erlang-questions] Iterate over fragmented table Message-ID: Hi, Exception is thrown when using mnesia:foldl/3 to iterate over a fragmented table: mnesia:foldl(Iterator, [], report_timestamp) exception throw: {badarg, [{ets,match_object, [5849148, {{locks,report_timestamp_frag37, '______WHOLETABLE_____'}, '_'}], []}, {mnesia_locker,need_lock,4, [{file,"mnesia_locker.erl"},{line,882}]}, {mnesia_locker,rlock,3, [{file,"mnesia_locker.erl"},{line,1036}]}, {mnesia,init_iteration,4, [{file,"mnesia.erl"},{line,1042}]}, {mnesia,foldl,6,[{file,"mnesia.erl"},{line,969}]}, {lists,foldl,3,[{file,"lists.erl"},{line,1263}]}, {mnesia_tm,apply_fun,3, [{file,"mnesia_tm.erl"},{line,836}]}, {mnesia_tm,execute_transaction,5, [{file,"mnesia_tm.erl"},{line,811}]}]} Why is this happening? Is this the correct way to iterate over a fragmented table? Erlang version is 19.2. Thanks Khitai From khitai.pang@REDACTED Wed Dec 28 21:59:00 2016 From: khitai.pang@REDACTED (Khitai Pang) Date: Wed, 28 Dec 2016 20:59:00 +0000 Subject: [erlang-questions] Iterate over fragmented table In-Reply-To: References: Message-ID: Code used for iteration: F = fun() -> mnesia:foldl(IterFun, [], report_timestamp) end, mnesia:activity(sync_transaction, F, [], mnesia_frag). Thanks Khitai On 2016/12/29 04:52, Khitai Pang wrote: > Hi, > > Exception is thrown when using mnesia:foldl/3 to iterate over a > fragmented table: > > mnesia:foldl(Iterator, [], report_timestamp) > > exception throw: {badarg, > [{ets,match_object, > [5849148, > {{locks,report_timestamp_frag37, > '______WHOLETABLE_____'}, > '_'}], > []}, > {mnesia_locker,need_lock,4, > [{file,"mnesia_locker.erl"},{line,882}]}, > {mnesia_locker,rlock,3, > [{file,"mnesia_locker.erl"},{line,1036}]}, > {mnesia,init_iteration,4, > [{file,"mnesia.erl"},{line,1042}]}, > {mnesia,foldl,6,[{file,"mnesia.erl"},{line,969}]}, > {lists,foldl,3,[{file,"lists.erl"},{line,1263}]}, > {mnesia_tm,apply_fun,3, > [{file,"mnesia_tm.erl"},{line,836}]}, > {mnesia_tm,execute_transaction,5, > [{file,"mnesia_tm.erl"},{line,811}]}]} > > > Why is this happening? Is this the correct way to iterate over a > fragmented table? > > Erlang version is 19.2. > > > Thanks > Khitai > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From dszoboszlay@REDACTED Thu Dec 29 00:19:45 2016 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Wed, 28 Dec 2016 23:19:45 +0000 Subject: [erlang-questions] How to avoid long_schedule issue ? In-Reply-To: References: Message-ID: Hi, First of all, lists:seq/2 is not a BIF, it is a pure Erlang function. And the 20-40 ms "long schedule" events are absolutely normal, you should use a much larger threshold (I would recommend at least 500 ms) to filter for the real outliers only. Now let's see what more can I tell you about long schedules! When you monitor long schedules, the schedulers will simply note the wall clock time when they schedule in a process and compare it with the wall clock time when they schedule it out. If more time has passed than the threshold you set, you get a message. This unfortunately can be quite a bit misleading, as it may include times you wouldn't expect. Just like: - Other OS processes running on the CPU. The OS can preempt the scheduler thread and give the CPU to some other process for 10-20 ms or more. And the scheduler will not know about this interruption. I'm pretty sure this is the reason for your long schedules, Alex. And this is fine as long as the host does not actually become CPU limited. If there are many OS processes fighting for the CPU, you will see horrible long schedules all over the place. - The OS performing some time consuming task for you. Like a page fault that requires reading from the disk swap. Whoops, your scheduler is suspended for tens of milliseconds without noticing it! - The OS performing some time consuming interrupt handling while your scheduler has the CPU. This is my personal favourite, because this means some totally unrelated OS code (like crappy NIC drivers) runs in your process' context, and can log scary looking messages that all seem to come from the pure beam process. Not to mention long schedules, of course. - The time it takes to grab some internal locks in the VM. If the lock is held by a long scheduling process, every other process waiting for the same lock will also long schedule. In our production system for example ~50% of long schedules come from a single monitoring process that periodically collects process info from other processes. Of course if a process long schedules with e.g. 1200 ms, the monitoring process will have to wait up to 1200 ms as well to grab the lock on it required for fetching process info. It is also good to know that not all BIF-s are preemptable, and those that are, will calculate their reduction cost in very ad-hoc looking ways. For example, it looks like that the lists:reverse/2 BIF can process 40 list elements per reduction, while lists:keyfind/3 can search 10 list elements per reduction. Do you think that reverting 40 list elements and looking through 10 list elements would take exactly the same wall clock time? Probably not. And they probably won't take exactly as much time either as an average reduction when executing your application's Erlang code. These reduction cost estimates work fine in most cases, but can be inaccurate when you give huge inputs to these functions. If they happen to be too low estimates on your system, you may still see long schedules when all your BIF-s and NIF-s are nice and preemptable. Now if you still have interesting long schedules that you want to debug, you need to keep in mind that the schedule in and schedule out functions are not necessarily the point where the time is wasted. For example if you have a gen_server that - when handling one particular message - calls an unfriendly BIF/NIF which doesn't update the reduction count, you will typically see that both the schedule in and schedule out points are gen_server:loop/6. Nothing will point to the BIF/NIF in the event, so good luck finding the offender! You have to consider all execution pathes that may lead from the schedule in point to the schedule out point. The offender can be any of the functions used on any of these pathes. Finally, a bit about finding the ideal long schedule threshold. 10 ms is typically too low: it basically means every time the OS schedules out your VM thread you will get a long schedule. But you need to consider the latency requirements of your application: if you're doing high frequency trading or ad bidding or whatever, maybe a 10 ms pause would be too much for you. In this case you can use such a low threshold, but be sure to turn off swapping and pin your schedulers to cores that are exclusively used by the VM, and where you have disabled interrupts, tick handling etc. In general, for a system where you need to keep latency under T, it makes sense to monitor long schedules with a threshold of ~0.5 T - 0.8 T or so. Both heart and the net ticktime of the distribution protocol give you such latency requirements: heart needs a heart beat message every 60 seconds and the distribution protocol sends one hear beat message every 15 seconds. So long schedules in the 15,000 ms range start to interfere with the distribution protocol, and above 60,000 ms can kill your node. (These limits may sound crazy, but I regularly see ~20,000 ms long schedules in our systems. Unfortunately.) Hope this helps! Daniel On Wed, 28 Dec 2016 at 16:23 Max Lapshin wrote: > I'm also very interested in how to properly interpret these warnings =( > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Thu Dec 29 06:27:26 2016 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Thu, 29 Dec 2016 00:27:26 -0500 Subject: [erlang-questions] How can I break this string into a list of strings? In-Reply-To: <38A6C400-A369-4BE1-B1D2-E331BEE54227@gmail.com> References: <38A6C400-A369-4BE1-B1D2-E331BEE54227@gmail.com> Message-ID: Many thanks all for your excellent fodder for thought. I would have thought that my question was trivial and the solution a well-worn path. But the issues are deeper than I imagined. All the best in this new year, Lloyd Sent from my iPad > On Dec 25, 2016, at 5:43 PM, Pierre Fenoll wrote: > > Hey ?ric > > Well it's not clear whether tags are nested in the OP. > So when my assumption applies, I would just read the list created 3 elements at a time, with strings of whitespace removed. > > If it doesn't, regexps are known to not be able to parse stack based grammars. > >> On 25 Dec 2016, at 21:41, ?ric Pailleau wrote: >> >> Hi Pierre, >> >> No this way you are loosing html tags. >> Lloyd wants to keep html tags, including linefeeds inside html tags. >> >> I tested with re:split(Html, "(<[^>]+>[^<]*<[^>]+>)",[{return, list}]) . Works. >> Lloyd can then loop on result and keep only non empty string after trim on elements. >> Regards From tuanbeo0502@REDACTED Thu Dec 29 04:37:23 2016 From: tuanbeo0502@REDACTED (Tuan Cao) Date: Thu, 29 Dec 2016 10:37:23 +0700 Subject: [erlang-questions] Erlang ODBC App returns wrong results for multi-statement request Message-ID: I am sorry if this is a duplicated post. When running multi-statement request in which the first statement returns zero rows, the Erlang ODBC App returns wrong result. To reproduce: 1> odbc:start(). ok 2> {ok, Ref}=odbc:connect("Driver=ODBC Driver 13 for SQL Server;Server=XXX;Database=YYY;Uid=ZZZ;Pwd=PPP;Encrypt= yes;TrustServerCertificate=no;Connection Timeout=30;", []). {ok,<0.40.0>} 3> odbc:sql_query(Ref, "create table z1(i integer);"). {updated,undefined} 4> odbc:sql_query(Ref, "insert into z1 select * from z1 where 1<0; select 1;"). {updated,0} As you can see, odbc_sql_query only returns the first result, and ignore all subsequence results. Looking at the source code, I found that this code block swallow subsequence result: /* OTP-5759, fails when 0 rows deleted */ if (result == SQL_NO_DATA_FOUND) { msg = encode_result(state); } else { /* Handle multiple result sets */ do { ei_x_encode_list_header(&dynamic_buffer(state), 1); msg = encode_result(state); /* We don't want to continue if an error occured */ if (msg.length != 0) { break; } msg = more_result_sets(state); /* We don't want to continue if an error occured */ if (msg.length != 0) { break; } } while (exists_more_result_sets(state)); ei_x_encode_empty_list(&dynamic_buffer(state)); } I can't find any detailed information about the issue OTP-5759. IMHO, the else branch is good enough for this code block. In particular, we can replace that code block by: /* Handle multiple result sets */ do { ei_x_encode_list_header(&dynamic_buffer(state), 1); msg = encode_result(state); /* We don't want to continue if an error occured */ if (msg.length != 0) { break; } msg = more_result_sets(state); /* We don't want to continue if an error occured */ if (msg.length != 0) { break; } } while (exists_more_result_sets(state)); ei_x_encode_empty_list(&dynamic_buffer(state)); I am afraid that doing so will re-surface the issue reported by OTP-5759. If you have detailed information(for example: bug report, how it was fixed), please share with me. Thanks, Tuan -------------- next part -------------- An HTML attachment was scrubbed... URL: From sweden.feng@REDACTED Thu Dec 29 10:22:49 2016 From: sweden.feng@REDACTED (Alex Feng) Date: Thu, 29 Dec 2016 10:22:49 +0100 Subject: [erlang-questions] How to avoid long_schedule issue ? In-Reply-To: References: Message-ID: Hi Daniel, The answer is very helpful, I have learnt a lot from your answer. Thank you very much and Happy New Year. Br, Alex 2016-12-29 0:19 GMT+01:00 D?niel Szoboszlay : > Hi, > > First of all, lists:seq/2 is not a BIF, it is a pure Erlang function. And > the 20-40 ms "long schedule" events are absolutely normal, you should use a > much larger threshold (I would recommend at least 500 ms) to filter for the > real outliers only. > > Now let's see what more can I tell you about long schedules! > > When you monitor long schedules, the schedulers will simply note the wall > clock time when they schedule in a process and compare it with the wall > clock time when they schedule it out. If more time has passed than the > threshold you set, you get a message. This unfortunately can be quite a bit > misleading, as it may include times you wouldn't expect. Just like: > > - Other OS processes running on the CPU. The OS can preempt the > scheduler thread and give the CPU to some other process for 10-20 ms or > more. And the scheduler will not know about this interruption. I'm pretty > sure this is the reason for your long schedules, Alex. > And this is fine as long as the host does not actually become CPU > limited. If there are many OS processes fighting for the CPU, you will see > horrible long schedules all over the place. > - The OS performing some time consuming task for you. Like a page > fault that requires reading from the disk swap. Whoops, your scheduler is > suspended for tens of milliseconds without noticing it! > - The OS performing some time consuming interrupt handling while your > scheduler has the CPU. > This is my personal favourite, because this means some totally > unrelated OS code (like crappy NIC drivers) runs in your process' context, > and can log scary looking messages that all seem to come from the pure beam > process. Not to mention long schedules, of course. > - The time it takes to grab some internal locks in the VM. If the lock > is held by a long scheduling process, every other process waiting for the > same lock will also long schedule. > In our production system for example ~50% of long schedules come from > a single monitoring process that periodically collects process info from > other processes. Of course if a process long schedules with e.g. 1200 ms, > the monitoring process will have to wait up to 1200 ms as well to grab the > lock on it required for fetching process info. > > It is also good to know that not all BIF-s are preemptable, and those that > are, will calculate their reduction cost in very ad-hoc looking ways. For > example, it looks like that the lists:reverse/2 BIF can process 40 list > elements per reduction, while lists:keyfind/3 can search 10 list elements > per reduction. Do you think that reverting 40 list elements and looking > through 10 list elements would take exactly the same wall clock time? > Probably not. And they probably won't take exactly as much time either as > an average reduction when executing your application's Erlang code. These > reduction cost estimates work fine in most cases, but can be inaccurate > when you give huge inputs to these functions. If they happen to be too low > estimates on your system, you may still see long schedules when all your > BIF-s and NIF-s are nice and preemptable. > > Now if you still have interesting long schedules that you want to debug, > you need to keep in mind that the schedule in and schedule out functions > are not necessarily the point where the time is wasted. For example if you > have a gen_server that - when handling one particular message - calls an > unfriendly BIF/NIF which doesn't update the reduction count, you will > typically see that both the schedule in and schedule out points are > gen_server:loop/6. Nothing will point to the BIF/NIF in the event, so > good luck finding the offender! You have to consider all execution pathes > that may lead from the schedule in point to the schedule out point. The > offender can be any of the functions used on any of these pathes. > > Finally, a bit about finding the ideal long schedule threshold. 10 ms is > typically too low: it basically means every time the OS schedules out your > VM thread you will get a long schedule. But you need to consider the > latency requirements of your application: if you're doing high frequency > trading or ad bidding or whatever, maybe a 10 ms pause would be too much > for you. In this case you can use such a low threshold, but be sure to turn > off swapping and pin your schedulers to cores that are exclusively used by > the VM, and where you have disabled interrupts, tick handling etc. In > general, for a system where you need to keep latency under T, it makes > sense to monitor long schedules with a threshold of ~0.5 T - 0.8 T or so. > > Both heart and the net ticktime of the distribution protocol give you such > latency requirements: heart needs a heart beat message every 60 seconds and > the distribution protocol sends one hear beat message every 15 seconds. So > long schedules in the 15,000 ms range start to interfere with the > distribution protocol, and above 60,000 ms can kill your node. (These > limits may sound crazy, but I regularly see ~20,000 ms long schedules in > our systems. Unfortunately.) > > Hope this helps! > > Daniel > > On Wed, 28 Dec 2016 at 16:23 Max Lapshin wrote: > >> I'm also very interested in how to properly interpret these warnings =( >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Fri Dec 30 11:24:10 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 30 Dec 2016 10:24:10 +0000 Subject: [erlang-questions] Comodo PositiveSSL certificates with Cowboy 1.x Message-ID: Hi guys, I would like to configure my "Comodo PositiveSSL" certificates with Cowboy. So far the self-signed OpenSSL certificates I've generated worked as expected. But I've no idea how to configure the "Comodo" ones. Can someone point me to a tutorial please? Or help on the setup? Thanks in advance. N.B: Comodo provides explanations for Nginx, Apache, etc. But not Cowboy unfortunately :-( Happy new year !!! /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Fri Dec 30 12:13:01 2016 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Fri, 30 Dec 2016 12:13:01 +0100 Subject: [erlang-questions] [ANN] geas 2.0.10 Message-ID: <9bfc0dac-68a2-a257-5265-7930450ca1bc@wanadoo.fr> Hi, Geas 2.0.10 has been released. This is only an update for Erlang 19.2 database. Geas is a tool that will detect the runnable official Erlang release window for your project, including dependancies, either from source code or beam files. Geas will tell you what are the offending functions in the beam/source files that reduce the available window. Geas will tell you if some beam files are compiled native. Geas is available as a module, erlang.mk and rebar 2/3 plugins. https://github.com/crownedgrouse/geas Cheers, and Happy New Year ! Eric From ali.sabil@REDACTED Fri Dec 30 13:03:02 2016 From: ali.sabil@REDACTED (Ali Sabil) Date: Fri, 30 Dec 2016 12:03:02 +0000 Subject: [erlang-questions] Comodo PositiveSSL certificates with Cowboy 1.x In-Reply-To: References: Message-ID: Hi Frank, I don't remember the exact details, but you should have received a zip file with a set of certificates. This zip file should contain 1. your certificate (foo_com.crt) 2. a set of intermediary certificates (intermediate1.crt, intermediate2.crt) 3. the root certificate (root.crt) You will need to concatenate all the intermediaries starting from the last one into what's called a intermediary certificate chain: cat intermediate2.crt intermediate1.crt > chain.crt The configuration of cowboy is then done using the `certfile` and ` cacertfile` options, for example: [ {certfile, "foo_com.crt"}, {cacertfile, "chain.crt"} ] These options are specified in the documentation of the Erlang SSL app ( http://erlang.org/doc/man/ssl.html) Hope this helps, Ali On Fri, Dec 30, 2016 at 11:24 AM Frank Muller wrote: > Hi guys, > > I would like to configure my "Comodo PositiveSSL" certificates with > Cowboy. > > So far the self-signed OpenSSL certificates I've generated worked as > expected. But I've no idea how to configure the "Comodo" ones. > > > Can someone point me to a tutorial please? Or help on the setup? > > > > > Thanks in advance. > > N.B: Comodo provides explanations for Nginx, Apache, etc. But not Cowboy > unfortunately :-( > > Happy new year !!! > /Frank > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Fri Dec 30 15:45:49 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 30 Dec 2016 14:45:49 +0000 Subject: [erlang-questions] Comodo PositiveSSL certificates with Cowboy 1.x In-Reply-To: References: Message-ID: Hi Ali, This what?s included in the Zip: AddTrustExternalCARoot.crt COMODORSAAddTrustCA.crt COMODORSADomainValidationSecureServerCA.crt STAR_company_com.crt company.key > 1. your certificate (foo_com.crt) So STAR_company_com.crt is my certificate. > 2. a set of intermediary certificates (intermediate1.crt, intermediate2.crt) How do i know which one is the latest ... to build the intermediary certificate chain in this case? They're not numbered. > 3. the root certificate (root.crt) What this one is useful for? Thank you. /Frank Le ven. 30 d?c. 2016 ? 13:03, Ali Sabil a ?crit : > Hi Frank, > > I don't remember the exact details, but you should have received a zip > file with a set of certificates. This zip file should contain > 1. your certificate (foo_com.crt) > 2. a set of intermediary certificates (intermediate1.crt, > intermediate2.crt) > 3. the root certificate (root.crt) > > You will need to concatenate all the intermediaries starting from the last > one into what's called a intermediary certificate chain: > cat intermediate2.crt intermediate1.crt > chain.crt > > The configuration of cowboy is then done using the `certfile` and ` > cacertfile` options, for example: > [ > {certfile, "foo_com.crt"}, > {cacertfile, "chain.crt"} > ] > > These options are specified in the documentation of the Erlang SSL app ( > http://erlang.org/doc/man/ssl.html) > > Hope this helps, > Ali > > > On Fri, Dec 30, 2016 at 11:24 AM Frank Muller > wrote: > > Hi guys, > > I would like to configure my "Comodo PositiveSSL" certificates with > Cowboy. > > So far the self-signed OpenSSL certificates I've generated worked as > expected. But I've no idea how to configure the "Comodo" ones. > > > Can someone point me to a tutorial please? Or help on the setup? > > > > > Thanks in advance. > > N.B: Comodo provides explanations for Nginx, Apache, etc. But not Cowboy > unfortunately :-( > > Happy new year !!! > /Frank > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ali.sabil@REDACTED Fri Dec 30 15:58:31 2016 From: ali.sabil@REDACTED (Ali Sabil) Date: Fri, 30 Dec 2016 14:58:31 +0000 Subject: [erlang-questions] Comodo PositiveSSL certificates with Cowboy 1.x In-Reply-To: References: Message-ID: On Fri, Dec 30, 2016 at 3:46 PM Frank Muller wrote: > Hi Ali, > > This what?s included in the Zip: > > AddTrustExternalCARoot.crt > COMODORSAAddTrustCA.crt > COMODORSADomainValidationSecureServerCA.crt > STAR_company_com.crt > company.key > > > 1. your certificate (foo_com.crt) > > So STAR_company_com.crt is my certificate. > > Yes, exactly > > 2. a set of intermediary certificates (intermediate1.crt, > intermediate2.crt) > > How do i know which one is the latest ... to build the intermediary > certificate chain in this case? > They're not numbered. > > For Comodo, `AddTrustExternalCARoor.crt` is the root certificate, followed by `COMODORSAAddTrustCA.crt` and then ` COMODORSADomainValidationSecureServerCA.crt` so your chain will be: cat COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt > chain.crt > > > 3. the root certificate (root.crt) > > What this one is useful for? > > The root certificate (AddTrustExternalCARoor.crt in your case) is useful for things like OCSP stapling as far as I know, which I don't think is implemented in Erlang SSL. If I am not mistaken, the Erlang SSL configuration is very similar to Apache. > Thank you. > > /Frank > > Le ven. 30 d?c. 2016 ? 13:03, Ali Sabil a ?crit : > > Hi Frank, > > I don't remember the exact details, but you should have received a zip > file with a set of certificates. This zip file should contain > 1. your certificate (foo_com.crt) > 2. a set of intermediary certificates (intermediate1.crt, > intermediate2.crt) > 3. the root certificate (root.crt) > > You will need to concatenate all the intermediaries starting from the last > one into what's called a intermediary certificate chain: > cat intermediate2.crt intermediate1.crt > chain.crt > > The configuration of cowboy is then done using the `certfile` and ` > cacertfile` options, for example: > [ > {certfile, "foo_com.crt"}, > {cacertfile, "chain.crt"} > ] > > These options are specified in the documentation of the Erlang SSL app ( > http://erlang.org/doc/man/ssl.html) > > Hope this helps, > Ali > > > On Fri, Dec 30, 2016 at 11:24 AM Frank Muller > wrote: > > Hi guys, > > I would like to configure my "Comodo PositiveSSL" certificates with > Cowboy. > > So far the self-signed OpenSSL certificates I've generated worked as > expected. But I've no idea how to configure the "Comodo" ones. > > > Can someone point me to a tutorial please? Or help on the setup? > > > > > Thanks in advance. > > N.B: Comodo provides explanations for Nginx, Apache, etc. But not Cowboy > unfortunately :-( > > Happy new year !!! > /Frank > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Fri Dec 30 16:05:17 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 30 Dec 2016 15:05:17 +0000 Subject: [erlang-questions] Comodo PositiveSSL certificates with Cowboy 1.x In-Reply-To: References: Message-ID: Hi again Ali It worked ;-) Here?s what ?I?ve done: 1. Concatenate them by reversing the lexicographical order: $ cat COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt AddTrustExternalCARoot.crt > cacert.pem 2. cp STAR_company_com.crt cert.pem 3. cp company.key key.pem Then, cowboy was happy with these settings: [ {cacertfile, "cacert.pem ?}, {certfile, "cert.pem ?}, {keyfile, "key.pem"} ] And now, ? curl ? isn?t complaining anymore ;-) Thank you. You made my day. /Frank Le ven. 30 d?c. 2016 ? 15:58, Ali Sabil a ?crit : > On Fri, Dec 30, 2016 at 3:46 PM Frank Muller > wrote: > > Hi Ali, > > This what?s included in the Zip: > > AddTrustExternalCARoot.crt > COMODORSAAddTrustCA.crt > COMODORSADomainValidationSecureServerCA.crt > STAR_company_com.crt > company.key > > > 1. your certificate (foo_com.crt) > > So STAR_company_com.crt is my certificate. > > > Yes, exactly > > > > > 2. a set of intermediary certificates (intermediate1.crt, > intermediate2.crt) > > How do i know which one is the latest ... to build the intermediary > certificate chain in this case? > They're not numbered. > > > For Comodo, `AddTrustExternalCARoor.crt` is the root certificate, > followed by `COMODORSAAddTrustCA.crt` and then ` > COMODORSADomainValidationSecureServerCA.crt` > > so your chain will be: > cat COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt > > chain.crt > > > > > > 3. the root certificate (root.crt) > > What this one is useful for? > > > The root certificate (AddTrustExternalCARoor.crt in your case) is useful > for things like OCSP stapling as far as I know, which I don't think is > implemented in Erlang SSL. > > If I am not mistaken, the Erlang SSL configuration is very similar to > Apache. > > > > Thank you. > > /Frank > > Le ven. 30 d?c. 2016 ? 13:03, Ali Sabil a ?crit : > > Hi Frank, > > I don't remember the exact details, but you should have received a zip > file with a set of certificates. This zip file should contain > 1. your certificate (foo_com.crt) > 2. a set of intermediary certificates (intermediate1.crt, > intermediate2.crt) > 3. the root certificate (root.crt) > > You will need to concatenate all the intermediaries starting from the last > one into what's called a intermediary certificate chain: > cat intermediate2.crt intermediate1.crt > chain.crt > > The configuration of cowboy is then done using the `certfile` and ` > cacertfile` options, for example: > [ > {certfile, "foo_com.crt"}, > {cacertfile, "chain.crt"} > ] > > These options are specified in the documentation of the Erlang SSL app ( > http://erlang.org/doc/man/ssl.html) > > Hope this helps, > Ali > > > On Fri, Dec 30, 2016 at 11:24 AM Frank Muller > wrote: > > Hi guys, > > I would like to configure my "Comodo PositiveSSL" certificates with > Cowboy. > > So far the self-signed OpenSSL certificates I've generated worked as > expected. But I've no idea how to configure the "Comodo" ones. > > > Can someone point me to a tutorial please? Or help on the setup? > > > > > Thanks in advance. > > N.B: Comodo provides explanations for Nginx, Apache, etc. But not Cowboy > unfortunately :-( > > Happy new year !!! > /Frank > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ali.sabil@REDACTED Fri Dec 30 16:08:28 2016 From: ali.sabil@REDACTED (Ali Sabil) Date: Fri, 30 Dec 2016 15:08:28 +0000 Subject: [erlang-questions] Comodo PositiveSSL certificates with Cowboy 1.x In-Reply-To: References: Message-ID: Great! But normally you wouldn't need to include the Root in your chain, that will just bloat up the TLS handshake for no good reason. On Fri, Dec 30, 2016 at 4:05 PM Frank Muller wrote: > Hi again Ali > > It worked ;-) > > Here?s what ?I?ve done: > > 1. Concatenate them by reversing the lexicographical order: > $ cat COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt > AddTrustExternalCARoot.crt > cacert.pem > > 2. cp STAR_company_com.crt cert.pem > > 3. cp company.key key.pem > > Then, cowboy was happy with these settings: > [ {cacertfile, "cacert.pem ?}, > {certfile, "cert.pem ?}, > {keyfile, "key.pem"} ] > > And now, ? curl ? isn?t complaining anymore ;-) > > Thank you. You made my day. > > /Frank > > Le ven. 30 d?c. 2016 ? 15:58, Ali Sabil a ?crit : > > On Fri, Dec 30, 2016 at 3:46 PM Frank Muller > wrote: > > Hi Ali, > > This what?s included in the Zip: > > AddTrustExternalCARoot.crt > COMODORSAAddTrustCA.crt > COMODORSADomainValidationSecureServerCA.crt > STAR_company_com.crt > company.key > > > 1. your certificate (foo_com.crt) > > So STAR_company_com.crt is my certificate. > > > Yes, exactly > > > > > 2. a set of intermediary certificates (intermediate1.crt, > intermediate2.crt) > > How do i know which one is the latest ... to build the intermediary > certificate chain in this case? > They're not numbered. > > > For Comodo, `AddTrustExternalCARoor.crt` is the root certificate, > followed by `COMODORSAAddTrustCA.crt` and then ` > COMODORSADomainValidationSecureServerCA.crt` > > so your chain will be: > cat COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt > > chain.crt > > > > > > 3. the root certificate (root.crt) > > What this one is useful for? > > > The root certificate (AddTrustExternalCARoor.crt in your case) is useful > for things like OCSP stapling as far as I know, which I don't think is > implemented in Erlang SSL. > > If I am not mistaken, the Erlang SSL configuration is very similar to > Apache. > > > > Thank you. > > /Frank > > Le ven. 30 d?c. 2016 ? 13:03, Ali Sabil a ?crit : > > Hi Frank, > > I don't remember the exact details, but you should have received a zip > file with a set of certificates. This zip file should contain > 1. your certificate (foo_com.crt) > 2. a set of intermediary certificates (intermediate1.crt, > intermediate2.crt) > 3. the root certificate (root.crt) > > You will need to concatenate all the intermediaries starting from the last > one into what's called a intermediary certificate chain: > cat intermediate2.crt intermediate1.crt > chain.crt > > The configuration of cowboy is then done using the `certfile` and ` > cacertfile` options, for example: > [ > {certfile, "foo_com.crt"}, > {cacertfile, "chain.crt"} > ] > > These options are specified in the documentation of the Erlang SSL app ( > http://erlang.org/doc/man/ssl.html) > > Hope this helps, > Ali > > > On Fri, Dec 30, 2016 at 11:24 AM Frank Muller > wrote: > > Hi guys, > > I would like to configure my "Comodo PositiveSSL" certificates with > Cowboy. > > So far the self-signed OpenSSL certificates I've generated worked as > expected. But I've no idea how to configure the "Comodo" ones. > > > Can someone point me to a tutorial please? Or help on the setup? > > > > > Thanks in advance. > > N.B: Comodo provides explanations for Nginx, Apache, etc. But not Cowboy > unfortunately :-( > > Happy new year !!! > /Frank > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Fri Dec 30 16:15:19 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 30 Dec 2016 15:15:19 +0000 Subject: [erlang-questions] Comodo PositiveSSL certificates with Cowboy 1.x In-Reply-To: References: Message-ID: Just deleted the root certificate from the chain. Thank you again Ali !!! /Frank Le ven. 30 d?c. 2016 ? 16:08, Ali Sabil a ?crit : > Great! But normally you wouldn't need to include the Root in your chain, > that will just bloat up the TLS handshake for no good reason. > > On Fri, Dec 30, 2016 at 4:05 PM Frank Muller > wrote: > > Hi again Ali > > It worked ;-) > > Here?s what ?I?ve done: > > 1. Concatenate them by reversing the lexicographical order: > $ cat COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt > AddTrustExternalCARoot.crt > cacert.pem > > 2. cp STAR_company_com.crt cert.pem > > 3. cp company.key key.pem > > Then, cowboy was happy with these settings: > [ {cacertfile, "cacert.pem ?}, > {certfile, "cert.pem ?}, > {keyfile, "key.pem"} ] > > And now, ? curl ? isn?t complaining anymore ;-) > > Thank you. You made my day. > > /Frank > > Le ven. 30 d?c. 2016 ? 15:58, Ali Sabil a ?crit : > > On Fri, Dec 30, 2016 at 3:46 PM Frank Muller > wrote: > > Hi Ali, > > This what?s included in the Zip: > > AddTrustExternalCARoot.crt > COMODORSAAddTrustCA.crt > COMODORSADomainValidationSecureServerCA.crt > STAR_company_com.crt > company.key > > > 1. your certificate (foo_com.crt) > > So STAR_company_com.crt is my certificate. > > > Yes, exactly > > > > > 2. a set of intermediary certificates (intermediate1.crt, > intermediate2.crt) > > How do i know which one is the latest ... to build the intermediary > certificate chain in this case? > They're not numbered. > > > For Comodo, `AddTrustExternalCARoor.crt` is the root certificate, > followed by `COMODORSAAddTrustCA.crt` and then ` > COMODORSADomainValidationSecureServerCA.crt` > > so your chain will be: > cat COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt > > chain.crt > > > > > > 3. the root certificate (root.crt) > > What this one is useful for? > > > The root certificate (AddTrustExternalCARoor.crt in your case) is useful > for things like OCSP stapling as far as I know, which I don't think is > implemented in Erlang SSL. > > If I am not mistaken, the Erlang SSL configuration is very similar to > Apache. > > > > Thank you. > > /Frank > > Le ven. 30 d?c. 2016 ? 13:03, Ali Sabil a ?crit : > > Hi Frank, > > I don't remember the exact details, but you should have received a zip > file with a set of certificates. This zip file should contain > 1. your certificate (foo_com.crt) > 2. a set of intermediary certificates (intermediate1.crt, > intermediate2.crt) > 3. the root certificate (root.crt) > > You will need to concatenate all the intermediaries starting from the last > one into what's called a intermediary certificate chain: > cat intermediate2.crt intermediate1.crt > chain.crt > > The configuration of cowboy is then done using the `certfile` and ` > cacertfile` options, for example: > [ > {certfile, "foo_com.crt"}, > {cacertfile, "chain.crt"} > ] > > These options are specified in the documentation of the Erlang SSL app ( > http://erlang.org/doc/man/ssl.html) > > Hope this helps, > Ali > > > On Fri, Dec 30, 2016 at 11:24 AM Frank Muller > wrote: > > Hi guys, > > I would like to configure my "Comodo PositiveSSL" certificates with > Cowboy. > > So far the self-signed OpenSSL certificates I've generated worked as > expected. But I've no idea how to configure the "Comodo" ones. > > > Can someone point me to a tutorial please? Or help on the setup? > > > > > Thanks in advance. > > N.B: Comodo provides explanations for Nginx, Apache, etc. But not Cowboy > unfortunately :-( > > Happy new year !!! > /Frank > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Sat Dec 31 09:56:52 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sat, 31 Dec 2016 08:56:52 +0000 Subject: [erlang-questions] Limit the Network Bandwidth per application Message-ID: Hi guys, I stumbled across trickle (https://github.com/mariusae/trickle), a user-land application bandwidth shaper. A very useful small piece of software as explained here: http://www.tecmint.com/manage-and-limit-downloadupload-bandwidth-with-trickle-in-linux/ It relies on libevent (http://libevent.org/), and can be installed vi apt-get or yum. I?m wondering if one can create a trickle equivalent in Erlang (+NIF or whatever)? Feedbacks very appreciated (links to equivalent apps in Erlang, implementation ideas, etc). Thank you and happy new year. /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Sat Dec 31 11:15:49 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sat, 31 Dec 2016 10:15:49 +0000 Subject: [erlang-questions] [ANN] geas 2.0.10 In-Reply-To: <9bfc0dac-68a2-a257-5265-7930450ca1bc@wanadoo.fr> References: <9bfc0dac-68a2-a257-5265-7930450ca1bc@wanadoo.fr> Message-ID: Hi Eric Geas from master on a toy project: $ erl -pa ebin -pa deps/*/ebin Erlang/OTP 17 [erts-6.4] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] Eshell V6.4 (abort with ^G) 1> geas:compat(".", global). ** exception error: no case clause matching {'EXIT', {function_clause, [{string,substr,["dir",0],[{file,"string.erl"},{line,195}]}, {geas,get_src_from_beam,1, [{file,"src/geas.erl"},{line,1350}]}, {geas,get_abstract,2,[{file,"src/geas.erl"},{line,1303}]}, {geas,get_erlang_compat_beam,1, [{file,"src/geas.erl"},{line,875}]}, {geas,'-get_erlang_compat/1-fun-0-',2, [{file,"src/geas.erl"},{line,840}]}, {lists,flatmap,2,[{file,"lists.erl"},{line,1248}]}, {lists,flatmap,2,[{file,"lists.erl"},{line,1248}]}, {geas,get_erlang_compat,1, [{file,"src/geas.erl"},{line,840}]}]}} in function geas:'-compat/2-fun-1-'/1 (src/geas.erl, line 1124) in call from lists:flatmap/2 (lists.erl, line 1248) in call from lists:flatmap/2 (lists.erl, line 1248) in call from geas:compat/2 (src/geas.erl, line 1123) in call from geas:compat/2 (src/geas.erl, line 1088) Am I using it correctly? /Frank Le ven. 30 d?c. 2016 ? 12:13, PAILLEAU Eric a ?crit : > Hi, > > > > Geas 2.0.10 has been released. > > > > This is only an update for Erlang 19.2 database. > > > > Geas is a tool that will detect the runnable official Erlang release > > window for your project, including dependancies, either from source code > > or beam files. > > > > Geas will tell you what are the offending functions in the beam/source > > files that reduce the available window. > > > > Geas will tell you if some beam files are compiled native. > > > > Geas is available as a module, erlang.mk and rebar 2/3 plugins. > > > > https://github.com/crownedgrouse/geas > > > > Cheers, and Happy New Year ! > > > > Eric > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Sat Dec 31 14:12:34 2016 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Sat, 31 Dec 2016 14:12:34 +0100 Subject: [erlang-questions] [ANN] geas 2.0.10 In-Reply-To: Message-ID: An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Sat Dec 31 15:51:45 2016 From: frank.muller.erl@REDACTED (Frank Muller) Date: Sat, 31 Dec 2016 14:51:45 +0000 Subject: [erlang-questions] [ANN] geas 2.0.10 In-Reply-To: References: Message-ID: Yes I did compile my project first. All the .beam are in ./ebin and deps/*/ebin. Ok for the ticket on GitHub. /Frank Le sam. 31 d?c. 2016 ? 14:12, ?ric Pailleau a ?crit : > Hi Frank, > Did you compiled your project first ? Otherwise you have to export a env > variable telling geas to use source files. > If this doesn't solve your problem, please open an issue on github with > the structure of your project. > Regards > > Le 31 d?c. 2016 11:15 AM, Frank Muller a > ?crit : > > Hi Eric > > Geas from master on a toy project: > > $ erl -pa ebin -pa deps/*/ebin > Erlang/OTP 17 [erts-6.4] [source] [64-bit] [smp:8:8] [async-threads:10] > [hipe] [kernel-poll:false] > > Eshell V6.4 (abort with ^G) > 1> geas:compat(".", global). > ** exception error: no case clause matching > {'EXIT', > {function_clause, > > [{string,substr,["dir",0],[{file,"string.erl"},{line,195}]}, > {geas,get_src_from_beam,1, > [{file,"src/geas.erl"},{line,1350}]}, > > {geas,get_abstract,2,[{file,"src/geas.erl"},{line,1303}]}, > {geas,get_erlang_compat_beam,1, > [{file,"src/geas.erl"},{line,875}]}, > {geas,'-get_erlang_compat/1-fun-0-',2, > [{file,"src/geas.erl"},{line,840}]}, > > {lists,flatmap,2,[{file,"lists.erl"},{line,1248}]}, > > {lists,flatmap,2,[{file,"lists.erl"},{line,1248}]}, > {geas,get_erlang_compat,1, > [{file,"src/geas.erl"},{line,840}]}]}} > in function geas:'-compat/2-fun-1-'/1 (src/geas.erl, line 1124) > in call from lists:flatmap/2 (lists.erl, line 1248) > in call from lists:flatmap/2 (lists.erl, line 1248) > in call from geas:compat/2 (src/geas.erl, line 1123) > in call from geas:compat/2 (src/geas.erl, line 1088) > > Am I using it correctly? > > /Frank > > Le ven. 30 d?c. 2016 ? 12:13, PAILLEAU Eric a > ?crit : > > Hi, > > > > Geas 2.0.10 has been released. > > > > This is only an update for Erlang 19.2 database. > > > > Geas is a tool that will detect the runnable official Erlang release > > window for your project, including dependancies, either from source code > > or beam files. > > > > Geas will tell you what are the offending functions in the beam/source > > files that reduce the available window. > > > > Geas will tell you if some beam files are compiled native. > > > > Geas is available as a module, erlang.mk and rebar 2/3 plugins. > > > > https://github.com/crownedgrouse/geas > > > > Cheers, and Happy New Year ! > > > > Eric > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: