From rickard@REDACTED Tue Oct 1 22:06:39 2019 From: rickard@REDACTED (Rickard Green) Date: Tue, 1 Oct 2019 22:06:39 +0200 Subject: [erlang-questions] Suspending Erlang Processes In-Reply-To: <5390547C-8ADF-4370-B702-0CF75BD1A91F@um.edu.mt> References: <5390547C-8ADF-4370-B702-0CF75BD1A91F@um.edu.mt> Message-ID: On Mon, Sep 30, 2019 at 1:57 PM Duncan Paul Attard < duncan.attard.01@REDACTED> wrote: > > I am tracing an Erlang process, say, `P` by invoking the BIF `erlang:trace(Pid_P, true, [set_on_spawn, procs, send, 'receive'])` from some process. As per the Erlang docs, the latter process becomes the tracer for `P`, which I shall call `Trc_Q`. > > Suppose now, that process `P` spawns a new process `Q`. Since the flag `set_on_spawn` was specified in the call to `erlang:trace/3` above, `Q` will automatically be traced by `Trc_P` as well. > > --- > > I want to spawn a **new** tracer, `Trc_Q`, and transfer the ownership of tracing `Q` to it, so that the resulting configuration will be that of process `P` being traced by tracer `Trc_P`, `Q` by `Trc_Q`. > Unfortunately I do not have any ideas on how to accomplish this. > However, Erlang permits **at most** one tracer per process, so I cannot achieve said configuration by invoking `erlang:trace(Pid_Q, true, ..)` from `Trc_Q`. The only way possible is to do it in two steps: > > 1. Tracer `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop `Trc_P` from tracing `Q`; > 2. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing `Q`. > > In the time span between steps **1.** and **2.** above, it might be possible that trace events by process `Q` are **lost** because at that moment, there is no tracer attached. One way of mitigating this is to perform the following: > > 1. Suspend process `Q` by calling `erlang:suspend_process(Pid_Q)` from `Trc_Q` (note that as per Erlang docs, `Trc_Q` remains blocked until `Q` is eventually suspended by the VM); > 2. `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop `Trc_P` from tracing `Q`; > 3. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing `Q`; > 4. Finally, `Trc_Q` calls `erlang:resume_process(Pid_Q)` so that `Q` can continue executing. > > From what I was able to find out, while `Q` is suspended, messages sent to it are queued, and when resumed, `Trc_Q` receives the `{trace, Pid_Q, receive, Msg}` trace events accordingly without any loss. > This is not a feature, it is a bug (introduced in erts 10.0, OTP 21.0) that will be fixed. The trace message should have been delivered even though the receiver was suspended. You cannot even rely on this behavior while this bug is present. If you (or any process in the system) send the suspended process a non-message signal (monitor, demonitor, link, unlink, exit, process_info, ...), the bug will be bypassed and the trace message will be delivered. > However, I am hesitant to use suspend/resume, since the Erlang docs explicitly say that these are to be used for *debugging purposes only*. Mission accomplished! :-) > Any idea as to why this is the case? > The language was designed with other communication primitives intended for use. Suspend/Resume was explicitly introduced for debugging purposes only, and not for usage by ordinary Erlang programs. They will most likely not disappear, but debug functionality in general are not treated as carefully by us at OTP as other ordinary functionality with regards to compatibility, etc. We for example removed the automatic deadlock prevention in suspend_process() that existed prior to erts 10.0 due to performance reasons. Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth@REDACTED Wed Oct 2 09:11:04 2019 From: kenneth@REDACTED (Kenneth Lundin) Date: Wed, 2 Oct 2019 09:11:04 +0200 Subject: [erlang-questions] Suspending Erlang Processes In-Reply-To: References: <5390547C-8ADF-4370-B702-0CF75BD1A91F@um.edu.mt> Message-ID: As a follow up on Rickards answer I think it would be interesting if you can explain why you want different tracers per process? If we know what problem you want to solve we can most probably come with better suggestions. I also recommend that you use tracing via the dbg module which is intended to be a more user friendly API towards tracing. The trace BIFs might give some more detailed control but dbg has support for most use cases and makes it easier to do the right thing, at least that is the intention. Also worth mentioning is that the tracing mechanisms are really not intended to use to achieve a certain functionality which is part of the application, they are intended to be used temporarily for debugging/profiling purposes. Since there is only one tracer at the time the use of tracing as part of the "ordinary" implementation of an application there will be conflicts as soon as any tracing or profiling is needed and probably the intended functionality of the application will then be broken. /Kenneth, Erlang/OTP Ericsson On Tue, Oct 1, 2019 at 10:07 PM Rickard Green wrote: > > > On Mon, Sep 30, 2019 at 1:57 PM Duncan Paul Attard < > duncan.attard.01@REDACTED> wrote: > > > > I am tracing an Erlang process, say, `P` by invoking the BIF > `erlang:trace(Pid_P, true, [set_on_spawn, procs, send, 'receive'])` from > some process. As per the Erlang docs, the latter process becomes the tracer > for `P`, which I shall call `Trc_Q`. > > > > Suppose now, that process `P` spawns a new process `Q`. Since the flag > `set_on_spawn` was specified in the call to `erlang:trace/3` above, `Q` > will automatically be traced by `Trc_P` as well. > > > > --- > > > > I want to spawn a **new** tracer, `Trc_Q`, and transfer the ownership of > tracing `Q` to it, so that the resulting configuration will be that of > process `P` being traced by tracer `Trc_P`, `Q` by `Trc_Q`. > > > > Unfortunately I do not have any ideas on how to accomplish this. > > > However, Erlang permits **at most** one tracer per process, so I cannot > achieve said configuration by invoking `erlang:trace(Pid_Q, true, ..)` from > `Trc_Q`. The only way possible is to do it in two steps: > > > > 1. Tracer `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop `Trc_P` > from tracing `Q`; > > 2. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing > `Q`. > > > > In the time span between steps **1.** and **2.** above, it might be > possible that trace events by process `Q` are **lost** because at that > moment, there is no tracer attached. One way of mitigating this is to > perform the following: > > > > 1. Suspend process `Q` by calling `erlang:suspend_process(Pid_Q)` from > `Trc_Q` (note that as per Erlang docs, `Trc_Q` remains blocked until `Q` is > eventually suspended by the VM); > > 2. `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop `Trc_P` from > tracing `Q`; > > 3. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing > `Q`; > > 4. Finally, `Trc_Q` calls `erlang:resume_process(Pid_Q)` so that `Q` can > continue executing. > > > > From what I was able to find out, while `Q` is suspended, messages sent > to it are queued, and when resumed, `Trc_Q` receives the `{trace, Pid_Q, > receive, Msg}` trace events accordingly without any loss. > > > > This is not a feature, it is a bug (introduced in erts 10.0, OTP 21.0) > that will be fixed. The trace message should have been delivered even > though the receiver was suspended. > > You cannot even rely on this behavior while this bug is present. If you > (or any process in the system) send the suspended process a non-message > signal (monitor, demonitor, link, unlink, exit, process_info, ...), the bug > will be bypassed and the trace message will be delivered. > > > However, I am hesitant to use suspend/resume, since the Erlang docs > explicitly say that these are to be used for *debugging purposes only*. > > Mission accomplished! :-) > > > Any idea as to why this is the case? > > > > The language was designed with other communication primitives intended for > use. Suspend/Resume was explicitly introduced for debugging purposes only, > and not for usage by ordinary Erlang programs. They will most likely not > disappear, but debug functionality in general are not treated as carefully > by us at OTP as other ordinary functionality with regards to compatibility, > etc. We for example removed the automatic deadlock prevention in > suspend_process() that existed prior to erts 10.0 due to performance > reasons. > > Regards, > Rickard > -- > Rickard Green, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From duncan.attard.01@REDACTED Wed Oct 2 08:41:39 2019 From: duncan.attard.01@REDACTED (Duncan Paul Attard) Date: Wed, 2 Oct 2019 08:41:39 +0200 Subject: [erlang-questions] Suspending Erlang Processes In-Reply-To: References: <5390547C-8ADF-4370-B702-0CF75BD1A91F@um.edu.mt> Message-ID: <07062E0D-BCAE-4D1E-8A2A-B34CC92AD2CD@um.edu.mt> Thanks for the explanation and for pointing the bug out. So it seems to me that there is no way to stop ?receive? trace events from being generated, despite the use of suspend. I guess this stems out from the asynchronous nature of the actor model. > The language was designed with other communication primitives intended for use. Suspend/Resume was explicitly introduced for debugging purposes only, and not for usage by ordinary Erlang programs. They will most likely not disappear, but debug functionality in general are not treated as carefully by us at OTP as other ordinary functionality with regards to compatibility, etc. We for example removed the automatic deadlock prevention in suspend_process() that existed prior to erts 10.0 due to performance reasons. I understand and do agree that synchronisation in Erlang, and in general, the actor model is modelled via message exchanges only, and that utilising other primitives such as suspend and resume does not adhere to this model. Yet, in my particular use case (runtime verification), I am restricting myself to systems/processes that *cannot* be instrumented with additional instructions (in this case, receive clauses) so as to block their execution at specific points. Thus, the only way left to me would be to suspend a process whilst it is executing, *without* my having the knowledge of what instruction the process in question is executing. To give you a bit of context, I am creating a monitoring system ?M' that is layered on top of a given system that one wishes to monitor, 'S'. ?M' observes the execution of ?S? via EVM tracing to try and detect infringements of certain logical properties specified over ?S?. The docs mention that suspend and resume are reserved for debugging purposes, and like you said in your reply, draws attention to the fact that careless use of these two functions can lead to inadvertent deadlocks. You mentioned also that automatic deadlock detection has been removed, hinting that that the implementation of suspend and resume might change in future releases of Erlang. I understand that. Besides this however, is there any other reason that suspend and resume should *not* be used? For instance, would executing suspend at any point, say, mess up the internal state of the suspended process? This question is in light of what I said above, namely that I would suspend a process whilst it is executing without having knowledge of what instruction the suspendee is executing. ?suspend_process/1? blocks the suspender until suspendee is eventually suspended: does "eventually suspended" mean that it is safe to assume that the VM brings suspendee to a state where it is ok to suspend it? And out of sheer curiosity, is a suspendee suspended as soon as possible, or does the scheduler execute its remaining number of reductions before suspending it and returns control back to the suspender? Once again, thanks a lot for your kind help Rickard. Best regards, Duncan. > On 01 Oct 2019, at 22:06, Rickard Green wrote: > > > > On Mon, Sep 30, 2019 at 1:57 PM Duncan Paul Attard > wrote: > > > > I am tracing an Erlang process, say, `P` by invoking the BIF `erlang:trace(Pid_P, true, [set_on_spawn, procs, send, 'receive'])` from some process. As per the Erlang docs, the latter process becomes the tracer for `P`, which I shall call `Trc_Q`. > > > > Suppose now, that process `P` spawns a new process `Q`. Since the flag `set_on_spawn` was specified in the call to `erlang:trace/3` above, `Q` will automatically be traced by `Trc_P` as well. > > > > --- > > > > I want to spawn a **new** tracer, `Trc_Q`, and transfer the ownership of tracing `Q` to it, so that the resulting configuration will be that of process `P` being traced by tracer `Trc_P`, `Q` by `Trc_Q`. > > > > Unfortunately I do not have any ideas on how to accomplish this. > > > However, Erlang permits **at most** one tracer per process, so I cannot achieve said configuration by invoking `erlang:trace(Pid_Q, true, ..)` from `Trc_Q`. The only way possible is to do it in two steps: > > > > 1. Tracer `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop `Trc_P` from tracing `Q`; > > 2. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing `Q`. > > > > In the time span between steps **1.** and **2.** above, it might be possible that trace events by process `Q` are **lost** because at that moment, there is no tracer attached. One way of mitigating this is to perform the following: > > > > 1. Suspend process `Q` by calling `erlang:suspend_process(Pid_Q)` from `Trc_Q` (note that as per Erlang docs, `Trc_Q` remains blocked until `Q` is eventually suspended by the VM); > > 2. `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop `Trc_P` from tracing `Q`; > > 3. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing `Q`; > > 4. Finally, `Trc_Q` calls `erlang:resume_process(Pid_Q)` so that `Q` can continue executing. > > > > From what I was able to find out, while `Q` is suspended, messages sent to it are queued, and when resumed, `Trc_Q` receives the `{trace, Pid_Q, receive, Msg}` trace events accordingly without any loss. > > > > This is not a feature, it is a bug (introduced in erts 10.0, OTP 21.0) that will be fixed. The trace message should have been delivered even though the receiver was suspended. > > You cannot even rely on this behavior while this bug is present. If you (or any process in the system) send the suspended process a non-message signal (monitor, demonitor, link, unlink, exit, process_info, ...), the bug will be bypassed and the trace message will be delivered. > > > However, I am hesitant to use suspend/resume, since the Erlang docs explicitly say that these are to be used for *debugging purposes only*. > > Mission accomplished! :-) > > > Any idea as to why this is the case? > > > > The language was designed with other communication primitives intended for use. Suspend/Resume was explicitly introduced for debugging purposes only, and not for usage by ordinary Erlang programs. They will most likely not disappear, but debug functionality in general are not treated as carefully by us at OTP as other ordinary functionality with regards to compatibility, etc. We for example removed the automatic deadlock prevention in suspend_process() that existed prior to erts 10.0 due to performance reasons. > > Regards, > Rickard > -- > Rickard Green, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From rickard@REDACTED Wed Oct 2 12:41:23 2019 From: rickard@REDACTED (Rickard Green) Date: Wed, 2 Oct 2019 12:41:23 +0200 Subject: [erlang-questions] Suspending Erlang Processes In-Reply-To: <07062E0D-BCAE-4D1E-8A2A-B34CC92AD2CD@um.edu.mt> References: <5390547C-8ADF-4370-B702-0CF75BD1A91F@um.edu.mt> <07062E0D-BCAE-4D1E-8A2A-B34CC92AD2CD@um.edu.mt> Message-ID: On Wed, Oct 2, 2019 at 8:41 AM Duncan Paul Attard < duncan.attard.01@REDACTED> wrote: > Thanks for the explanation and for pointing the bug out. So it seems to me > that there is no way to stop ?receive? trace events from being generated, > despite the use of suspend. I guess this stems out from the asynchronous > nature of the actor model. > > The language was designed with other communication primitives intended for > use. Suspend/Resume was explicitly introduced for debugging purposes only, > and not for usage by ordinary Erlang programs. They will most likely not > disappear, but debug functionality in general are not treated as carefully > by us at OTP as other ordinary functionality with regards to compatibility, > etc. We for example removed the automatic deadlock prevention in > suspend_process() that existed prior to erts 10.0 due to performance > reasons. > > > I understand and do agree that synchronisation in Erlang, and in general, > the actor model is modelled via message exchanges only, and that utilising > other primitives such as suspend and resume does not adhere to this model. > > Yet, in my particular use case (runtime verification), I am restricting > myself to systems/processes that *cannot* be instrumented with additional > instructions (in this case, receive clauses) so as to block their execution > at specific points. Thus, the only way left to me would be to suspend a > process whilst it is executing, *without* my having the knowledge of what > instruction the process in question is executing. To give you a bit of > context, I am creating a monitoring system ?M' that is layered on top of a > given system that one wishes to monitor, 'S'. ?M' observes the execution of > ?S? via EVM tracing to try and detect infringements of certain logical > properties specified over ?S?. > > The docs mention that suspend and resume are reserved for debugging > purposes, and like you said in your reply, draws attention to the fact that > careless use of these two functions can lead to inadvertent deadlocks. You > mentioned also that automatic deadlock detection has been removed, hinting > that that the implementation of suspend and resume might change in future > releases of Erlang. I understand that. Besides this however, is there any > other reason that suspend and resume should *not* be used? For instance, > would executing suspend at any point, say, mess up the internal state of > the suspended process? > Not unless there is a bug in the suspend functionality. > This question is in light of what I said above, namely that I would > suspend a process whilst it is executing without having knowledge of what > instruction the suspendee is executing. ?suspend_process/1? blocks the > suspender until suspendee is eventually suspended: does "eventually > suspended" mean that it is safe to assume that the VM brings suspendee to a > state where it is ok to suspend it? > > Yes. The suspender sends the suspendee an asynchronous signal. The suspendee wont suspend until it receives and handle the suspend signal. Currently handling of such signals only occur when a process is scheduled in, when executing a receive (depending on the state of the message queue), and when scheduled out from a dirty scheduler. Signals are handled in received order. If there are a lot of signals to handle, not all of them are necessarily handled before continuing execution. That is, the suspendee may pass one of these points where it handles incoming signals and then continue execution even if there should be a suspend signal waiting for it. > And out of sheer curiosity, is a suspendee suspended as soon as possible, > or does the scheduler execute its remaining number of reductions before > suspending it and returns control back to the suspender? > > As described above, it suspends when it sees the suspend signal which might or might not happen before the remaining number of reductions has been exhausted. > Once again, thanks a lot for your kind help Rickard. > > Best regards, > Duncan. > > > > Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From torrellpremier@REDACTED Wed Oct 2 12:53:33 2019 From: torrellpremier@REDACTED (torrellpremier) Date: Wed, 2 Oct 2019 11:53:33 +0100 Subject: [erlang-questions] Erl_interface deprication & C Node communication Message-ID: Hi all I recently found out that in the new OTP 22 release, the legacy 'erl_interface' library is now deprecated and will be removed in OTP 23. I relied on that library to communicate with my C program, which I'm using in my current and upcoming projects. I was wondering what the alternative is to communicate with C Nodes? It said in the release notes that the 'ei' library will not be removed, which is great, but there aren't tutorials in the Erlang manual that describe how to utilize this library, like how there are tutorials for the erl_interface 'erl_' prefixed functions. ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Wed Oct 2 13:30:14 2019 From: serge@REDACTED (Serge Aleynikov) Date: Wed, 2 Oct 2019 07:30:14 -0400 Subject: [erlang-questions] Erl_interface deprication & C Node communication In-Reply-To: References: Message-ID: Take a look at this one: https://erlangcentral.org/wiki/How_to_use_ei_to_marshal_binary_terms_in_port_programs On Wed, Oct 2, 2019 at 7:05 AM torrellpremier wrote: > Hi all > > I recently found out that in the new OTP 22 release, the legacy > 'erl_interface' library is now deprecated and will be removed in OTP 23. > > I relied on that library to communicate with my C program, which I'm using > in my current and upcoming projects. > > I was wondering what the alternative is to communicate with C Nodes? It > said in the release notes that the 'ei' library will not be removed, which > is great, but there aren't tutorials in the Erlang manual that describe how > to utilize this library, like how there are tutorials for the erl_interface > 'erl_' prefixed functions. > ? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From torrellpremier@REDACTED Wed Oct 2 13:56:48 2019 From: torrellpremier@REDACTED (torrellpremier) Date: Wed, 2 Oct 2019 12:56:48 +0100 Subject: [erlang-questions] Erl_interface deprication & C Node communication In-Reply-To: References: Message-ID: Thanks for that! ? On Wed, Oct 2, 2019 at 12:30 PM Serge Aleynikov wrote: > Take a look at this one: > https://erlangcentral.org/wiki/How_to_use_ei_to_marshal_binary_terms_in_port_programs > > On Wed, Oct 2, 2019 at 7:05 AM torrellpremier > wrote: > >> Hi all >> >> I recently found out that in the new OTP 22 release, the legacy >> 'erl_interface' library is now deprecated and will be removed in OTP 23. >> >> I relied on that library to communicate with my C program, which I'm >> using in my current and upcoming projects. >> >> I was wondering what the alternative is to communicate with C Nodes? It >> said in the release notes that the 'ei' library will not be removed, which >> is great, but there aren't tutorials in the Erlang manual that describe how >> to utilize this library, like how there are tutorials for the erl_interface >> 'erl_' prefixed functions. >> ? >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dszoboszlay@REDACTED Thu Oct 3 00:22:15 2019 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Thu, 3 Oct 2019 00:22:15 +0200 Subject: [erlang-questions] Wildcard certificate rejected with hostname_check_failed Message-ID: Hi, I'm trying to connect to the server foobar.example.com with ss from an Erlang node. The server presents a wildcard certificate with an ?'id-ce-subjectAltName' extension that looks like this: {'Extension', {2,5,29,17}, false, [{dNSName, "*.example.com"}, {dNSName, "example.com"} ]} The peer verification (default one) fails with hostname_check_failed. I traced the problem back to public_key:verify_hostname_match_default0/2 which gets called like this: public_key:verify_hostname_match_default0( {dns_id, "foobar.example.com"}, {dNSName, "*.example.com"}) This check fails. The function has one clause for matching a dns_id against a dNSName, and it requires an exact match. It also has a clause that allows wildcards, but that's only for matching a (non-wrapped) fqdn against a cn tuple: https://github.com/erlang/otp/blob/5bf0a8a2a18a8e883792692bdc060e37552b0d08/lib/public_key/src/public_key.erl#L1672-L1675 Why isn't the wildcard matching enabled for the dns_id against dNSName case too? The fqdn against cn scenario is only used as a fallback when the certificate doesn't contain an ?'id-ce-subjectAltName' extension. But as far as I can tell wildcard certificates typically have the extension, instead of (or besides) using a wildcard in the CN (see for example the certificate presented by https://blog.hu/). The public_key User's Guide says: In case where the Presented IDs are fetched from the Subject certificate > field, the names may contain wildcard characters. The function handles this > as defined in chapter 6.4.3 in RFC 6125 > . But as I understand, chapter 6.4.3 speaks about presented ids *in general*. Where does this restriction to the subject field come from? Thanks, Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From duncan.attard.01@REDACTED Thu Oct 3 11:27:01 2019 From: duncan.attard.01@REDACTED (Duncan Paul Attard) Date: Thu, 3 Oct 2019 11:27:01 +0200 Subject: [erlang-questions] Suspending Erlang Processes In-Reply-To: References: <5390547C-8ADF-4370-B702-0CF75BD1A91F@um.edu.mt> Message-ID: <8D9D5576-E5D8-42C4-90FD-FBAFDA7301B3@um.edu.mt> Kenneth, Rickard, Let me give you a bit of context. I?m working on a runtime verification (RV) tool that focusses on components systems in an asynchronous setting. I?ve chosen Erlang because it nicely models this setting and also facilitates certain aspects in the development of said tool. Very briefly, in RV, the concept is that of instrumenting the system with other processes (called monitors in the RV community, but have nothing to do with Erlang monitors) that analyse the parts of the system (e.g., one process or a group of them, which I will refer to as a "system component") to detect and flag the infringement of some property specified over the component. These properties (which are written using a high-level logic such as Linear Temporal Logic, Hennessy-Milner Logic, etc.), define things like ?Process P cannot send message M to Q when such and such condition arises? or ?Process P must exit when a particular message M is sent to it?, etc. A monitor, or rather, the monitor source code, is synthesised from a property and ?attached? to the component to be monitored. The following is more or less the general workflow: 1. A property is written in a text file using one of the logics mentioned; 2. The property is parsed and compiled to generate the monitor (in Erlang source code, in my particular case); 3. The monitor is spawned as a process that analyses a system component of interest as this executes. The monitor needs to somehow acquire the runtime events emitted by processes, and this it does via the built-in Erlang tracing (i.e., the monitor is itself a tracer process). The important thing to note is that the monitors, despite being processes themselves, may be considered as a meta-layer over the system, and therefore, do not technically form part of the ?ordinary? implementation of the system. This means that monitors can be introduced or removed from the system as needed, and merely function as a second layer that strives to observe the system with *minimal* interference. This brings me to Kenneth?s point, that tracing is a tool intended for debugging/profiling purposes. I agree, and in fact, RV might be considered as a flavour of debugging or profiling that is done at runtime. It differs (amongst other things) from debugging and profiling, in that monitors are the product of autogenerated code resulting from *formal* logical properties. From what I gather, debugging or profiling obtains trace events in a similar way to the one I?m using for monitoring. I also understand and agree with you Kenneth that, if a system process is being monitored by one of my monitors, then it cannot be profiled or debugged due to the one-tracer limit imposed by the EVM. Also, the reason I?m not using ?dbg' but 'erlang:trace/3? directly is that I want the full flexibility of tracing (I might require it in later stages of my research). Way back when I started the project I was not aware of the extent of the functionality ?dbg? offers, and so to play it safe (and after reading Francesco and Simon?s book), I decided to go for the tracing BIFs. Finally, the reason I require different tracers (in my case, monitors) for different system processes (or groups) is that it makes the specification of correctness properties much more manageable. The gist of the idea is that it is far easier to specify a property over a restricted set of processes (e.g., just one process which exhibits *sequential* execution) than it is for a large number of processes, as then the property needs to account for all the possible interleavings of trace events exhibited by different processes. So in a sense, different monitors over different system components allow me to partition and view the otherwise whole trace of the system as a collection of separate traces for different components. Naturally, the monitors generated from smaller properties tend to be small and lightweight themselves, and are easier to work with. Moreover, this allows me to switch off certain monitors dynamically at runtime for system components that might not require monitoring anymore, while leaving others on. Since a system can be viewed as always starting from one root process, I attach (i.e., start tracing) a special root monitor to this system root process. The root monitor creates new monitors on the fly for certain child processes that are spawned by the root system process. Now, to collect trace events without loss, the root monitor is configured with ?set_on_spawn?, meaning that new children of the root system process are automatically traced by the root monitor at first. To spawn a dedicated monitor ?Mon_C' for some child process ?C?, the following is executed: 1. Root monitor ?Mon_R? is currently tracing the new child process ?C? ('set_on_spawn' flag was set on 'Mon_R'); 2. The new monitor ?Mon_C? created for child process ?C? switches tracing *off* for ?C? (i.e., erlang:trace(Pid_C, false, ..)), so the (previous) monitor ?Mon_R' stops being the tracer of ?C?; 3. New monitor ?Mon_C? switches tracing back *on* for child process ?C? and becomes its new tracer. To minimise trace event loss between steps 2 and 3, I was thinking of suspending child process ?C' before step 2, and resuming it after step 3 This way, ?C? is at least blocked, and cannot spawn new processes itself or send messages. I cannot however prevent other processes from sending ?C' messages, meaning that there might be a chance of ?receive? events being lost in the space of time between steps 2 and 3. Therefore, my suggestion still does not banish the problem but merely mitigates it, as steps 2 and 3 do not happen atomically. I wonder whether such a BIF could be realisable, such that the ownership of tracing can be transferred atomically between tracers without incurring any loss of trace events (between monitors ?Mon_R? and ?Mon_C? in my case). FYI, much of the work I?ve discussed has already been published in a previous paper we?ve written in the past. The paper can be found here: http://staff.um.edu.mt/afra1/papers/sefm17.pdf . If you?re interested please let me know. Many thanks for your help! Duncan > On 02 Oct 2019, at 09:11, Kenneth Lundin wrote: > > As a follow up on Rickards answer I think it would be interesting if you can explain why you want different tracers per process? > If we know what problem you want to solve we can most probably come with better suggestions. > > I also recommend that you use tracing via the dbg module which is intended to be a more user friendly API towards tracing. The trace BIFs might give some more detailed control but dbg has support for most use cases and makes it easier to do the right thing, at least that is the intention. > > Also worth mentioning is that the tracing mechanisms are really not intended to use to achieve a certain functionality which is part of the application, they are intended to be used temporarily for debugging/profiling purposes. Since there is only one tracer at the time the use of tracing as part of the "ordinary" implementation of an application there will be conflicts as soon as any tracing or profiling is needed and probably the intended functionality of the application will then be broken. > > /Kenneth, Erlang/OTP Ericsson > > On Tue, Oct 1, 2019 at 10:07 PM Rickard Green > wrote: > > > On Mon, Sep 30, 2019 at 1:57 PM Duncan Paul Attard > wrote: > > > > I am tracing an Erlang process, say, `P` by invoking the BIF `erlang:trace(Pid_P, true, [set_on_spawn, procs, send, 'receive'])` from some process. As per the Erlang docs, the latter process becomes the tracer for `P`, which I shall call `Trc_Q`. > > > > Suppose now, that process `P` spawns a new process `Q`. Since the flag `set_on_spawn` was specified in the call to `erlang:trace/3` above, `Q` will automatically be traced by `Trc_P` as well. > > > > --- > > > > I want to spawn a **new** tracer, `Trc_Q`, and transfer the ownership of tracing `Q` to it, so that the resulting configuration will be that of process `P` being traced by tracer `Trc_P`, `Q` by `Trc_Q`. > > > > Unfortunately I do not have any ideas on how to accomplish this. > > > However, Erlang permits **at most** one tracer per process, so I cannot achieve said configuration by invoking `erlang:trace(Pid_Q, true, ..)` from `Trc_Q`. The only way possible is to do it in two steps: > > > > 1. Tracer `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop `Trc_P` from tracing `Q`; > > 2. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing `Q`. > > > > In the time span between steps **1.** and **2.** above, it might be possible that trace events by process `Q` are **lost** because at that moment, there is no tracer attached. One way of mitigating this is to perform the following: > > > > 1. Suspend process `Q` by calling `erlang:suspend_process(Pid_Q)` from `Trc_Q` (note that as per Erlang docs, `Trc_Q` remains blocked until `Q` is eventually suspended by the VM); > > 2. `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop `Trc_P` from tracing `Q`; > > 3. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing `Q`; > > 4. Finally, `Trc_Q` calls `erlang:resume_process(Pid_Q)` so that `Q` can continue executing. > > > > From what I was able to find out, while `Q` is suspended, messages sent to it are queued, and when resumed, `Trc_Q` receives the `{trace, Pid_Q, receive, Msg}` trace events accordingly without any loss. > > > > This is not a feature, it is a bug (introduced in erts 10.0, OTP 21.0) that will be fixed. The trace message should have been delivered even though the receiver was suspended. > > You cannot even rely on this behavior while this bug is present. If you (or any process in the system) send the suspended process a non-message signal (monitor, demonitor, link, unlink, exit, process_info, ...), the bug will be bypassed and the trace message will be delivered. > > > However, I am hesitant to use suspend/resume, since the Erlang docs explicitly say that these are to be used for *debugging purposes only*. > > Mission accomplished! :-) > > > Any idea as to why this is the case? > > > > The language was designed with other communication primitives intended for use. Suspend/Resume was explicitly introduced for debugging purposes only, and not for usage by ordinary Erlang programs. They will most likely not disappear, but debug functionality in general are not treated as carefully by us at OTP as other ordinary functionality with regards to compatibility, etc. We for example removed the automatic deadlock prevention in suspend_process() that existed prior to erts 10.0 due to performance reasons. > > Regards, > Rickard > -- > Rickard Green, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From Maxim.Minin@REDACTED Fri Oct 4 14:16:44 2019 From: Maxim.Minin@REDACTED (Minin Maxim) Date: Fri, 4 Oct 2019 12:16:44 +0000 Subject: [erlang-questions] problem with eldap module Message-ID: <5139013185794173bf747c0b25f95741@SV2767.baadergroup.net> Hello, I have a problem with eldap module. The call eldap:simple_bind(Handle, Dn, Password) don't work correct if the password contains sign '?'. Is it a known issue? Have I to report this as a bug? I think , the cause of the problem is maybe the encoding function in ELDAPv3 module (asn1 module from eldap lib). I have generated ELDAPv3.erl from ELDAPv3.asn1 (call asn1ct:compile("ELDAPv3.asn1") and found this: ... encode(Type, Data) -> try iolist_to_binary(element(1, encode_disp(Type, Data))) of Bytes -> {ok,Bytes} ...... this call to iolist_to_binary makes the problem by ? sign and should by replace by unicode:characters_to_binary/3 1> erlang:iolist_to_binary([167]). <<"?">> 2> unicode:characters_to_binary([167], utf8, utf8). <<"?"/utf8>> 3> As a workaround I just convert the password string to binary bevor eldap:simple_bind/4 call, like that: ... PaswordAsBin = unicode:characters_to_binary(Password,utf8,utf8), BindAnswer = eldap:simple_bind(UserHandle,DN,PaswordAsBin), ... It works but according to documentation of eldap module the password have to be string: "simple_bind(Handle, Dn, Password) -> return_value() OTP R15B01 Types Handle = handle() Dn = string() Password = string() Authenticate the connection using simple authentication." Thanks Maxim -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5514 bytes Desc: not available URL: From g@REDACTED Fri Oct 4 21:32:13 2019 From: g@REDACTED (Guilherme Andrade) Date: Fri, 4 Oct 2019 20:32:13 +0100 Subject: [erlang-questions] problem with eldap module In-Reply-To: <5139013185794173bf747c0b25f95741@SV2767.baadergroup.net> References: <5139013185794173bf747c0b25f95741@SV2767.baadergroup.net> Message-ID: Hello Maxim, On Fri, 4 Oct 2019 at 13:16, Minin Maxim wrote: > Hello, > > > > I have a problem with eldap module. The call eldap:simple_bind(Handle, > Dn, Password) don't work correct if the password contains sign '?'. > > Is it a known issue? Have I to report this as a bug? > > > > I think , the cause of the problem is maybe the encoding function in > ELDAPv3 module (asn1 module from eldap lib). I have generated ELDAPv3.erl > from ELDAPv3.asn1 (call asn1ct:compile("ELDAPv3.asn1") and found this: > > ... > > encode(Type, Data) -> > > try *iolist_to_binary(element(1, encode_disp(Type, Data)))* of > > Bytes -> > > {ok,Bytes} > > ...... > > > > this call to iolist_to_binary makes the problem by ? sign and should by > replace by unicode:characters_to_binary/3 > > 1> erlang:iolist_to_binary([167]). > > <<"?">> > > 2> unicode:characters_to_binary([167], utf8, utf8). > > <<"?"/utf8>> > > 3> > > > > As a workaround I just convert the password string to binary bevor > eldap:simple_bind/4 call, like that: > > ... > > PaswordAsBin = unicode:characters_to_binary(Password,utf8,utf8), > > BindAnswer = eldap:simple_bind(UserHandle,DN,PaswordAsBin), > I've encountered this issue before; I worked around it the same way you did, while also applying an extra conversion for Dialyzer's sake: Normalized = unicode:characters_to_nfkc_binary(Password), ByteList = binary_to_list(Normalized) Which in effect produces a list of bytes that's UTF-8 encoded rather than in IEC 8859-1. I should point out though, that normalizing it to a single Unicode form is important in this sort of thing, lest you get failed authorizations because some password contains a symbol with more than a single representation. At the time I concluded that the KC norm was the one to be used with LDAP, but I don't recall the source. > ... > > > > It works but according to documentation of eldap module the password have > to be string: > > *"simple_bind(Handle, Dn, Password) -> return_value()* > > * OTP R15B01* > > *Types* > > *Handle = handle()* > > *Dn = string()* > > *Password = string()* > > *Authenticate the connection using simple authentication."* > > > > Thanks > > Maxim > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Guilherme -------------- next part -------------- An HTML attachment was scrubbed... URL: From otp@REDACTED Mon Oct 7 12:52:48 2019 From: otp@REDACTED (Erlang/OTP) Date: Mon, 7 Oct 2019 12:52:48 +0200 Subject: [erlang-questions] Patch Package OTP 20.3.8.23 Released Message-ID: <20191007105247.GA2308@erix.ericsson.se> Patch Package: OTP 20.3.8.23 Git Tag: OTP-20.3.8.23 Date: 2019-10-07 Trouble Report Id: OTP-15867, OTP-15947, OTP-16012, OTP-16058, OTP-16107, OTP-16133 Seq num: ERIERL-373, ERIERL-378, ERL-1049 System: OTP Release: 20 Application: crypto-4.2.2.3, erts-9.3.3.12, snmp-5.2.11.2, syntax_tools-2.1.4.2 Predecessor: OTP 20.3.8.22 Check out the git tag OTP-20.3.8.23, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- crypto-4.2.2.3 -------------------------------------------------- --------------------------------------------------------------------- The crypto-4.2.2.3 application can be applied independently of other applications on a full OTP 20 installation. --- Improvements and New Features --- OTP-16133 Application(s): crypto Related Id(s): PR-2407 The chipers aes_cfb8 and aes_cfb128 are now using the EVP interface. The supported key lengths are 128, 192 and 256 bits. Full runtime dependencies of crypto-4.2.2.3: erts-9.0, kernel-5.3, stdlib-3.4 --------------------------------------------------------------------- --- erts-9.3.3.12 --------------------------------------------------- --------------------------------------------------------------------- The erts-9.3.3.12 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-15867 Application(s): erts Related Id(s): ERIERL-373 The runtime system disconnected a connection if it received an exit/2 signal where the recipient was a process on an old incarnation of the current node. That is, the receiving node had the same node name, but another "creation" number. The signal will now just be dropped since the receiving process no longer exists. OTP-16058 Application(s): erts Related Id(s): PR-2382 Fix a bug in binary_to_term that would crash the emulator if a term larger than 16GB was to be decoded. OTP-16107 Application(s): erts Related Id(s): ERL-1049 When communicating with a simultaneously exiting port via the erlang:port_*() BIFs one could sometimes get stray {Ref, What} messages. Where Ref was a reference and What usually were the atom badarg. Full runtime dependencies of erts-9.3.3.12: kernel-5.0, sasl-3.0.1, stdlib-3.0 --------------------------------------------------------------------- --- snmp-5.2.11.2 --------------------------------------------------- --------------------------------------------------------------------- The snmp-5.2.11.2 application can be applied independently of other applications on a full OTP 20 installation. --- Improvements and New Features --- OTP-15947 Application(s): snmp Related Id(s): ERIERL-378 [manager] The callbacks where executed in a (new) 'temporary' process, that executed the callback call and then exited. This has now been made configurable so that is also possible to specify a 'permanent' callback proxy process. All callback calls will then be executed in this (permanent) process (in sequence). Full runtime dependencies of snmp-5.2.11.2: crypto-3.3, erts-6.0, kernel-3.0, mnesia-4.12, runtime_tools-1.8.14, stdlib-2.5 --------------------------------------------------------------------- --- syntax_tools-2.1.4.2 -------------------------------------------- --------------------------------------------------------------------- The syntax_tools-2.1.4.2 application can be applied independently of other applications on a full OTP 20 installation. --- Fixed Bugs and Malfunctions --- OTP-16012 Application(s): syntax_tools Related Id(s): PR-2348 Add missing calls to erl_syntax:unwrap/1. The nodes concerned represent names and values of maps and map types. Full runtime dependencies of syntax_tools-2.1.4.2: compiler-7.0, erts-9.0, kernel-5.0, stdlib-3.4 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From jozsef.berces@REDACTED Tue Oct 8 09:23:05 2019 From: jozsef.berces@REDACTED (=?iso-8859-1?Q?J=F3zsef_B=E9rces?=) Date: Tue, 8 Oct 2019 07:23:05 +0000 Subject: [erlang-questions] ssh:connect "Key exchange failed" Message-ID: Hi, I need to connect to an ssh server but ssh:connect always returns "Key exchange failed". I can connect with KiTTY, and checking the KiTTY logs I suspect that the problem is the 1024-bit RSA key. Could you please check the KiTTY log below what is not supported by the Erlang SSH client? Is there any way to add the missing support to the Erlang SSH client so that I could connect to this server? Thanks, Jozsef 2019-10-07 14:37:58 Starting 42264 from XXXXX@REDACTED 2019-10-07 14:38:07 Connecting to 10.XXX.XXX.XXX port 22 2019-10-07 14:38:07 We claim version: SSH-2.0-PuTTY_KiTTY 2019-10-07 14:38:07 Server version: SSH-2.0-Sun_SSH_1.1.5 2019-10-07 14:38:07 Using SSH protocol version 2 2019-10-07 14:38:08 Doing Diffie-Hellman group exchange 2019-10-07 14:38:08 Doing Diffie-Hellman key exchange with hash SHA-1 2019-10-07 14:38:08 Server also has ssh-dss host key, but we don't know it 2019-10-07 14:38:08 Host key fingerprint is: 2019-10-07 14:38:08 ssh-rsa 1024 XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX 2019-10-07 14:38:16 Initialised AES-256 CBC client->server encryption 2019-10-07 14:38:16 Initialised HMAC-SHA1-96 client->server MAC algorithm 2019-10-07 14:38:16 Initialised AES-256 CBC server->client encryption 2019-10-07 14:38:16 Initialised HMAC-SHA1-96 server->client MAC algorithm 2019-10-07 14:38:22 Using SSPI from SECUR32.DLL 2019-10-07 14:38:22 Attempting GSSAPI authentication 2019-10-07 14:38:22 GSSAPI authentication initialisation failed 2019-10-07 14:38:22 The target was not recognized. 2019-10-07 14:38:22 Attempting keyboard-interactive authentication 2019-10-07 14:38:30 Access granted 2019-10-07 14:38:30 Opening session as main channel 2019-10-07 14:38:30 Opened main channel 2019-10-07 14:38:30 Allocated pty (ospeed 38400bps, ispeed 38400bps) 2019-10-07 14:38:31 Started a shell/command -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.r.nilsson@REDACTED Tue Oct 8 10:47:11 2019 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Tue, 8 Oct 2019 08:47:11 +0000 Subject: [erlang-questions] ssh:connect "Key exchange failed" In-Reply-To: References: Message-ID: The "Key exchange failed" error indicates that there is no agreement in the first phase where different algorithms are negotiated. A common cause is that there are no common cipher algorithm available. For example the AES-256 CBC is not available in Erlang SSH. Which otp version do you use? What does the function ssh:default_algorithms() return? What algorithms does the server provide? (Check by telnet the server, type "SSH-2.0-xxx" and look at the funny characters that is returned. Or post them here) /Hans ________________________________ Fr?n: erlang-questions-bounces@REDACTED f?r J?zsef B?rces Skickat: den 8 oktober 2019 09:23 Till: Erlang Questions ?mne: [erlang-questions] ssh:connect "Key exchange failed" Hi, I need to connect to an ssh server but ssh:connect always returns "Key exchange failed". I can connect with KiTTY, and checking the KiTTY logs I suspect that the problem is the 1024-bit RSA key. Could you please check the KiTTY log below what is not supported by the Erlang SSH client? Is there any way to add the missing support to the Erlang SSH client so that I could connect to this server? Thanks, Jozsef 2019-10-07 14:37:58 Starting 42264 from XXXXX@REDACTED 2019-10-07 14:38:07 Connecting to 10.XXX.XXX.XXX port 22 2019-10-07 14:38:07 We claim version: SSH-2.0-PuTTY_KiTTY 2019-10-07 14:38:07 Server version: SSH-2.0-Sun_SSH_1.1.5 2019-10-07 14:38:07 Using SSH protocol version 2 2019-10-07 14:38:08 Doing Diffie-Hellman group exchange 2019-10-07 14:38:08 Doing Diffie-Hellman key exchange with hash SHA-1 2019-10-07 14:38:08 Server also has ssh-dss host key, but we don't know it 2019-10-07 14:38:08 Host key fingerprint is: 2019-10-07 14:38:08 ssh-rsa 1024 XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX 2019-10-07 14:38:16 Initialised AES-256 CBC client->server encryption 2019-10-07 14:38:16 Initialised HMAC-SHA1-96 client->server MAC algorithm 2019-10-07 14:38:16 Initialised AES-256 CBC server->client encryption 2019-10-07 14:38:16 Initialised HMAC-SHA1-96 server->client MAC algorithm 2019-10-07 14:38:22 Using SSPI from SECUR32.DLL 2019-10-07 14:38:22 Attempting GSSAPI authentication 2019-10-07 14:38:22 GSSAPI authentication initialisation failed 2019-10-07 14:38:22 The target was not recognized. 2019-10-07 14:38:22 Attempting keyboard-interactive authentication 2019-10-07 14:38:30 Access granted 2019-10-07 14:38:30 Opening session as main channel 2019-10-07 14:38:30 Opened main channel 2019-10-07 14:38:30 Allocated pty (ospeed 38400bps, ispeed 38400bps) 2019-10-07 14:38:31 Started a shell/command -------------- next part -------------- An HTML attachment was scrubbed... URL: From otp@REDACTED Tue Oct 8 11:35:30 2019 From: otp@REDACTED (Erlang/OTP) Date: Tue, 8 Oct 2019 11:35:30 +0200 Subject: [erlang-questions] Patch Package OTP 22.1.2 Released Message-ID: <20191008093530.GA1483@erix.ericsson.se> Patch Package: OTP 22.1.2 Git Tag: OTP-22.1.2 Date: 2019-10-08 Trouble Report Id: OTP-16125 Seq num: ERIERL-412 System: OTP Release: 22 Application: erts-10.5.2 Predecessor: OTP 22.1.1 Check out the git tag OTP-22.1.2, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- erts-10.5.2 ----------------------------------------------------- --------------------------------------------------------------------- The erts-10.5.2 application can be applied independently of other applications on a full OTP 22 installation. --- Improvements and New Features --- OTP-16125 Application(s): erts Related Id(s): ERIERL-412 Added the environment variable ERLC_SERVER_ID, which allows multiple compile servers to run separately under the same user. Full runtime dependencies of erts-10.5.2: kernel-6.1, sasl-3.3, stdlib-3.5 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From otp@REDACTED Tue Oct 8 12:22:08 2019 From: otp@REDACTED (Erlang/OTP) Date: Tue, 8 Oct 2019 12:22:08 +0200 Subject: [erlang-questions] Patch Package OTP 21.3.8.8 Released Message-ID: <20191008102208.GA30012@erix.ericsson.se> Patch Package: OTP 21.3.8.8 Git Tag: OTP-21.3.8.8 Date: 2019-10-08 Trouble Report Id: OTP-16058, OTP-16107, OTP-16133, OTP-16134 Seq num: ERL-1049 System: OTP Release: 21 Application: crypto-4.4.2.1, erts-10.3.5.6 Predecessor: OTP 21.3.8.7 Check out the git tag OTP-21.3.8.8, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- crypto-4.4.2.1 -------------------------------------------------- --------------------------------------------------------------------- The crypto-4.4.2.1 application can be applied independently of other applications on a full OTP 21 installation. --- Improvements and New Features --- OTP-16133 Application(s): crypto Related Id(s): PR-2407 The chipers aes_cfb8 and aes_cfb128 are now using the EVP interface. The supported key lengths are 128, 192 and 256 bits. OTP-16134 Application(s): crypto Related Id(s): PR-2407 The chipers aes_cfb8 and aes_cfb128 are now available in FIPS enabled mode. Full runtime dependencies of crypto-4.4.2.1: erts-9.0, kernel-5.3, stdlib-3.4 --------------------------------------------------------------------- --- erts-10.3.5.6 --------------------------------------------------- --------------------------------------------------------------------- Note! The erts-10.3.5.6 application *cannot* be applied independently of other applications on an arbitrary OTP 21 installation. On a full OTP 21 installation, also the following runtime dependencies have to be satisfied: -- kernel-6.1 (first satisfied in OTP 21.1) -- sasl-3.3 (first satisfied in OTP 21.2) --- Fixed Bugs and Malfunctions --- OTP-16058 Application(s): erts Related Id(s): PR-2382 Fix a bug in binary_to_term that would crash the emulator if a term larger than 16GB was to be decoded. OTP-16107 Application(s): erts Related Id(s): ERL-1049 When communicating with a simultaneously exiting port via the erlang:port_*() BIFs one could sometimes get stray {Ref, What} messages. Where Ref was a reference and What usually were the atom badarg. Full runtime dependencies of erts-10.3.5.6: kernel-6.1, sasl-3.3, stdlib-3.5 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From jozsef.berces@REDACTED Tue Oct 8 14:36:26 2019 From: jozsef.berces@REDACTED (=?iso-8859-1?Q?J=F3zsef_B=E9rces?=) Date: Tue, 8 Oct 2019 12:36:26 +0000 Subject: [erlang-questions] ssh:connect "Key exchange failed" In-Reply-To: References: Message-ID: Hi Hans, I tried it with several OTP releases up to 21. I have not tried OTP 22. Here is what I received with telnet: Q?Cb8???~x|q?ob'gss-group1-sha1-toWM5Slw5Ew8Mqkay+al2g==,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1ssh-rsa,ssh-dss aes256-cbc,aes192-cbc,aes128-cbc aes256-cbc,aes192-cbc,aes128-cbchmac-sha1-96,hmac-md5-96hmac-sha1-96,hmac-md5-96 none,zlib none,zlib es,i-default es,i-defaultConnection closed by foreign host. And these are the default algorithms: [{kex,['ecdh-sha2-nistp384','ecdh-sha2-nistp521', 'ecdh-sha2-nistp256','diffie-hellman-group-exchange-sha256', 'diffie-hellman-group16-sha512', 'diffie-hellman-group18-sha512', 'diffie-hellman-group14-sha256', 'diffie-hellman-group14-sha1', 'diffie-hellman-group-exchange-sha1']}, {public_key,['ecdsa-sha2-nistp384','ecdsa-sha2-nistp521', 'ecdsa-sha2-nistp256','ssh-rsa','rsa-sha2-256', 'rsa-sha2-512','ssh-dss']}, {cipher,[{client2server,['aes256-gcm@REDACTED', 'aes256-ctr','aes192-ctr','aes128-gcm@REDACTED', 'aes128-ctr','aes128-cbc','3des-cbc']}, {server2client,['aes256-gcm@REDACTED','aes256-ctr', 'aes192-ctr','aes128-gcm@REDACTED','aes128-ctr', 'aes128-cbc','3des-cbc']}]}, {mac,[{client2server,['hmac-sha2-256','hmac-sha2-512', 'hmac-sha1']}, {server2client,['hmac-sha2-256','hmac-sha2-512', 'hmac-sha1']}]}, {compression,[{client2server,[none,'zlib@REDACTED',zlib]}, {server2client,[none,'zlib@REDACTED',zlib]}]}] 5> ssh_transport:supported_algorithms(mac). [{client2server,['hmac-sha2-256','hmac-sha2-512', 'hmac-sha1','AEAD_AES_128_GCM','AEAD_AES_256_GCM']}, {server2client,['hmac-sha2-256','hmac-sha2-512','hmac-sha1', 'AEAD_AES_128_GCM','AEAD_AES_256_GCM']}] Thanks, Jozsef From: Hans Nilsson R Sent: Tuesday, 8 October, 2019 10:47 To: J?zsef B?rces ; Erlang Questions Subject: Sv: ssh:connect "Key exchange failed" The "Key exchange failed" error indicates that there is no agreement in the first phase where different algorithms are negotiated. A common cause is that there are no common cipher algorithm available. For example the AES-256 CBC is not available in Erlang SSH. Which otp version do you use? What does the function ssh:default_algorithms() return? What algorithms does the server provide? (Check by telnet the server, type "SSH-2.0-xxx" and look at the funny characters that is returned. Or post them here) /Hans ________________________________ Fr?n: erlang-questions-bounces@REDACTED f?r J?zsef B?rces Skickat: den 8 oktober 2019 09:23 Till: Erlang Questions ?mne: [erlang-questions] ssh:connect "Key exchange failed" Hi, I need to connect to an ssh server but ssh:connect always returns "Key exchange failed". I can connect with KiTTY, and checking the KiTTY logs I suspect that the problem is the 1024-bit RSA key. Could you please check the KiTTY log below what is not supported by the Erlang SSH client? Is there any way to add the missing support to the Erlang SSH client so that I could connect to this server? Thanks, Jozsef 2019-10-07 14:37:58 Starting 42264 from XXXXX@REDACTED 2019-10-07 14:38:07 Connecting to 10.XXX.XXX.XXX port 22 2019-10-07 14:38:07 We claim version: SSH-2.0-PuTTY_KiTTY 2019-10-07 14:38:07 Server version: SSH-2.0-Sun_SSH_1.1.5 2019-10-07 14:38:07 Using SSH protocol version 2 2019-10-07 14:38:08 Doing Diffie-Hellman group exchange 2019-10-07 14:38:08 Doing Diffie-Hellman key exchange with hash SHA-1 2019-10-07 14:38:08 Server also has ssh-dss host key, but we don't know it 2019-10-07 14:38:08 Host key fingerprint is: 2019-10-07 14:38:08 ssh-rsa 1024 XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX 2019-10-07 14:38:16 Initialised AES-256 CBC client->server encryption 2019-10-07 14:38:16 Initialised HMAC-SHA1-96 client->server MAC algorithm 2019-10-07 14:38:16 Initialised AES-256 CBC server->client encryption 2019-10-07 14:38:16 Initialised HMAC-SHA1-96 server->client MAC algorithm 2019-10-07 14:38:22 Using SSPI from SECUR32.DLL 2019-10-07 14:38:22 Attempting GSSAPI authentication 2019-10-07 14:38:22 GSSAPI authentication initialisation failed 2019-10-07 14:38:22 The target was not recognized. 2019-10-07 14:38:22 Attempting keyboard-interactive authentication 2019-10-07 14:38:30 Access granted 2019-10-07 14:38:30 Opening session as main channel 2019-10-07 14:38:30 Opened main channel 2019-10-07 14:38:30 Allocated pty (ospeed 38400bps, ispeed 38400bps) 2019-10-07 14:38:31 Started a shell/command -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.r.nilsson@REDACTED Tue Oct 8 16:15:27 2019 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Tue, 8 Oct 2019 14:15:27 +0000 Subject: [erlang-questions] ssh:connect "Key exchange failed" In-Reply-To: References: , Message-ID: I've found it. The server and client has no common mac algorithm. I tried to add 'hmac-sha1-96' to Erlang SSH: $ git diff diff --git a/lib/ssh/src/ssh_transport.erl b/lib/ssh/src/ssh_transport.erl index a85926354e..66d06c3360 100644 --- a/lib/ssh/src/ssh_transport.erl +++ b/lib/ssh/src/ssh_transport.erl @@ -171,6 +171,7 @@ supported_algorithms(mac) -> [{'hmac-sha2-256', [{macs,hmac}, {hashs,sha256}]}, {'hmac-sha2-512', [{macs,hmac}, {hashs,sha512}]}, {'hmac-sha1', [{macs,hmac}, {hashs,sha}]}, + {'hmac-sha1-96', [{macs,hmac}, {hashs,sha}]}, {'AEAD_AES_128_GCM', [{ciphers,aes_128_gcm}]}, {'AEAD_AES_256_GCM', [{ciphers,aes_256_gcm}]} ] I can't test it because I have no ssh server at hand that has that old mac. But it works with erlang client to erlang server. Could you patch your Erlang client, test and tell me it it works? /Hans ________________________________ Fr?n: J?zsef B?rces Skickat: den 8 oktober 2019 14:36 Till: Hans Nilsson R ; Erlang Questions ?mne: RE: ssh:connect "Key exchange failed" Hi Hans, I tried it with several OTP releases up to 21. I have not tried OTP 22. Here is what I received with telnet: Q?Cb8????x|q?ob?gss-group1-sha1-toWM5Slw5Ew8Mqkay+al2g==,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1ssh-rsa,ssh-dss aes256-cbc,aes192-cbc,aes128-cbc aes256-cbc,aes192-cbc,aes128-cbchmac-sha1-96,hmac-md5-96hmac-sha1-96,hmac-md5-96 none,zlib none,zlib es,i-default es,i-defaultConnection closed by foreign host. And these are the default algorithms: [{kex,['ecdh-sha2-nistp384','ecdh-sha2-nistp521', 'ecdh-sha2-nistp256','diffie-hellman-group-exchange-sha256', 'diffie-hellman-group16-sha512', 'diffie-hellman-group18-sha512', 'diffie-hellman-group14-sha256', 'diffie-hellman-group14-sha1', 'diffie-hellman-group-exchange-sha1']}, {public_key,['ecdsa-sha2-nistp384','ecdsa-sha2-nistp521', 'ecdsa-sha2-nistp256','ssh-rsa','rsa-sha2-256', 'rsa-sha2-512','ssh-dss']}, {cipher,[{client2server,['aes256-gcm@REDACTED', 'aes256-ctr','aes192-ctr','aes128-gcm@REDACTED', 'aes128-ctr','aes128-cbc','3des-cbc']}, {server2client,['aes256-gcm@REDACTED','aes256-ctr', 'aes192-ctr','aes128-gcm@REDACTED','aes128-ctr', 'aes128-cbc','3des-cbc']}]}, {mac,[{client2server,['hmac-sha2-256','hmac-sha2-512', 'hmac-sha1']}, {server2client,['hmac-sha2-256','hmac-sha2-512', 'hmac-sha1']}]}, {compression,[{client2server,[none,'zlib@REDACTED',zlib]}, {server2client,[none,'zlib@REDACTED',zlib]}]}] 5> ssh_transport:supported_algorithms(mac). [{client2server,['hmac-sha2-256','hmac-sha2-512', 'hmac-sha1','AEAD_AES_128_GCM','AEAD_AES_256_GCM']}, {server2client,['hmac-sha2-256','hmac-sha2-512','hmac-sha1', 'AEAD_AES_128_GCM','AEAD_AES_256_GCM']}] Thanks, Jozsef From: Hans Nilsson R Sent: Tuesday, 8 October, 2019 10:47 To: J?zsef B?rces ; Erlang Questions Subject: Sv: ssh:connect "Key exchange failed" The "Key exchange failed" error indicates that there is no agreement in the first phase where different algorithms are negotiated. A common cause is that there are no common cipher algorithm available. For example the AES-256 CBC is not available in Erlang SSH. Which otp version do you use? What does the function ssh:default_algorithms() return? What algorithms does the server provide? (Check by telnet the server, type "SSH-2.0-xxx" and look at the funny characters that is returned. Or post them here) /Hans ________________________________ Fr?n: erlang-questions-bounces@REDACTED f?r J?zsef B?rces Skickat: den 8 oktober 2019 09:23 Till: Erlang Questions ?mne: [erlang-questions] ssh:connect "Key exchange failed" Hi, I need to connect to an ssh server but ssh:connect always returns "Key exchange failed". I can connect with KiTTY, and checking the KiTTY logs I suspect that the problem is the 1024-bit RSA key. Could you please check the KiTTY log below what is not supported by the Erlang SSH client? Is there any way to add the missing support to the Erlang SSH client so that I could connect to this server? Thanks, Jozsef 2019-10-07 14:37:58 Starting 42264 from XXXXX@REDACTED 2019-10-07 14:38:07 Connecting to 10.XXX.XXX.XXX port 22 2019-10-07 14:38:07 We claim version: SSH-2.0-PuTTY_KiTTY 2019-10-07 14:38:07 Server version: SSH-2.0-Sun_SSH_1.1.5 2019-10-07 14:38:07 Using SSH protocol version 2 2019-10-07 14:38:08 Doing Diffie-Hellman group exchange 2019-10-07 14:38:08 Doing Diffie-Hellman key exchange with hash SHA-1 2019-10-07 14:38:08 Server also has ssh-dss host key, but we don't know it 2019-10-07 14:38:08 Host key fingerprint is: 2019-10-07 14:38:08 ssh-rsa 1024 XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX 2019-10-07 14:38:16 Initialised AES-256 CBC client->server encryption 2019-10-07 14:38:16 Initialised HMAC-SHA1-96 client->server MAC algorithm 2019-10-07 14:38:16 Initialised AES-256 CBC server->client encryption 2019-10-07 14:38:16 Initialised HMAC-SHA1-96 server->client MAC algorithm 2019-10-07 14:38:22 Using SSPI from SECUR32.DLL 2019-10-07 14:38:22 Attempting GSSAPI authentication 2019-10-07 14:38:22 GSSAPI authentication initialisation failed 2019-10-07 14:38:22 The target was not recognized. 2019-10-07 14:38:22 Attempting keyboard-interactive authentication 2019-10-07 14:38:30 Access granted 2019-10-07 14:38:30 Opening session as main channel 2019-10-07 14:38:30 Opened main channel 2019-10-07 14:38:30 Allocated pty (ospeed 38400bps, ispeed 38400bps) 2019-10-07 14:38:31 Started a shell/command -------------- next part -------------- An HTML attachment was scrubbed... URL: From jozsef.berces@REDACTED Tue Oct 8 17:27:29 2019 From: jozsef.berces@REDACTED (=?iso-8859-1?Q?J=F3zsef_B=E9rces?=) Date: Tue, 8 Oct 2019 15:27:29 +0000 Subject: [erlang-questions] ssh:connect "Key exchange failed" In-Reply-To: References: , Message-ID: Hi Hans, It works! Many thanks, Jozsef From: Hans Nilsson R Sent: Tuesday, 8 October, 2019 16:15 To: J?zsef B?rces ; Erlang Questions Subject: Sv: ssh:connect "Key exchange failed" I've found it. The server and client has no common mac algorithm. I tried to add 'hmac-sha1-96' to Erlang SSH: $ git diff diff --git a/lib/ssh/src/ssh_transport.erl b/lib/ssh/src/ssh_transport.erl index a85926354e..66d06c3360 100644 --- a/lib/ssh/src/ssh_transport.erl +++ b/lib/ssh/src/ssh_transport.erl @@ -171,6 +171,7 @@ supported_algorithms(mac) -> [{'hmac-sha2-256', [{macs,hmac}, {hashs,sha256}]}, {'hmac-sha2-512', [{macs,hmac}, {hashs,sha512}]}, {'hmac-sha1', [{macs,hmac}, {hashs,sha}]}, + {'hmac-sha1-96', [{macs,hmac}, {hashs,sha}]}, {'AEAD_AES_128_GCM', [{ciphers,aes_128_gcm}]}, {'AEAD_AES_256_GCM', [{ciphers,aes_256_gcm}]} ] I can't test it because I have no ssh server at hand that has that old mac. But it works with erlang client to erlang server. Could you patch your Erlang client, test and tell me it it works? /Hans ________________________________ Fr?n: J?zsef B?rces > Skickat: den 8 oktober 2019 14:36 Till: Hans Nilsson R >; Erlang Questions > ?mne: RE: ssh:connect "Key exchange failed" Hi Hans, I tried it with several OTP releases up to 21. I have not tried OTP 22. Here is what I received with telnet: Q?Cb8???~x|q?ob'gss-group1-sha1-toWM5Slw5Ew8Mqkay+al2g==,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1ssh-rsa,ssh-dss aes256-cbc,aes192-cbc,aes128-cbc aes256-cbc,aes192-cbc,aes128-cbchmac-sha1-96,hmac-md5-96hmac-sha1-96,hmac-md5-96 none,zlib none,zlib es,i-default es,i-defaultConnection closed by foreign host. And these are the default algorithms: [{kex,['ecdh-sha2-nistp384','ecdh-sha2-nistp521', 'ecdh-sha2-nistp256','diffie-hellman-group-exchange-sha256', 'diffie-hellman-group16-sha512', 'diffie-hellman-group18-sha512', 'diffie-hellman-group14-sha256', 'diffie-hellman-group14-sha1', 'diffie-hellman-group-exchange-sha1']}, {public_key,['ecdsa-sha2-nistp384','ecdsa-sha2-nistp521', 'ecdsa-sha2-nistp256','ssh-rsa','rsa-sha2-256', 'rsa-sha2-512','ssh-dss']}, {cipher,[{client2server,['aes256-gcm@REDACTED', 'aes256-ctr','aes192-ctr','aes128-gcm@REDACTED', 'aes128-ctr','aes128-cbc','3des-cbc']}, {server2client,['aes256-gcm@REDACTED','aes256-ctr', 'aes192-ctr','aes128-gcm@REDACTED','aes128-ctr', 'aes128-cbc','3des-cbc']}]}, {mac,[{client2server,['hmac-sha2-256','hmac-sha2-512', 'hmac-sha1']}, {server2client,['hmac-sha2-256','hmac-sha2-512', 'hmac-sha1']}]}, {compression,[{client2server,[none,'zlib@REDACTED',zlib]}, {server2client,[none,'zlib@REDACTED',zlib]}]}] 5> ssh_transport:supported_algorithms(mac). [{client2server,['hmac-sha2-256','hmac-sha2-512', 'hmac-sha1','AEAD_AES_128_GCM','AEAD_AES_256_GCM']}, {server2client,['hmac-sha2-256','hmac-sha2-512','hmac-sha1', 'AEAD_AES_128_GCM','AEAD_AES_256_GCM']}] Thanks, Jozsef From: Hans Nilsson R > Sent: Tuesday, 8 October, 2019 10:47 To: J?zsef B?rces >; Erlang Questions > Subject: Sv: ssh:connect "Key exchange failed" The "Key exchange failed" error indicates that there is no agreement in the first phase where different algorithms are negotiated. A common cause is that there are no common cipher algorithm available. For example the AES-256 CBC is not available in Erlang SSH. Which otp version do you use? What does the function ssh:default_algorithms() return? What algorithms does the server provide? (Check by telnet the server, type "SSH-2.0-xxx" and look at the funny characters that is returned. Or post them here) /Hans ________________________________ Fr?n: erlang-questions-bounces@REDACTED > f?r J?zsef B?rces > Skickat: den 8 oktober 2019 09:23 Till: Erlang Questions > ?mne: [erlang-questions] ssh:connect "Key exchange failed" Hi, I need to connect to an ssh server but ssh:connect always returns "Key exchange failed". I can connect with KiTTY, and checking the KiTTY logs I suspect that the problem is the 1024-bit RSA key. Could you please check the KiTTY log below what is not supported by the Erlang SSH client? Is there any way to add the missing support to the Erlang SSH client so that I could connect to this server? Thanks, Jozsef 2019-10-07 14:37:58 Starting 42264 from XXXXX@REDACTED 2019-10-07 14:38:07 Connecting to 10.XXX.XXX.XXX port 22 2019-10-07 14:38:07 We claim version: SSH-2.0-PuTTY_KiTTY 2019-10-07 14:38:07 Server version: SSH-2.0-Sun_SSH_1.1.5 2019-10-07 14:38:07 Using SSH protocol version 2 2019-10-07 14:38:08 Doing Diffie-Hellman group exchange 2019-10-07 14:38:08 Doing Diffie-Hellman key exchange with hash SHA-1 2019-10-07 14:38:08 Server also has ssh-dss host key, but we don't know it 2019-10-07 14:38:08 Host key fingerprint is: 2019-10-07 14:38:08 ssh-rsa 1024 XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX 2019-10-07 14:38:16 Initialised AES-256 CBC client->server encryption 2019-10-07 14:38:16 Initialised HMAC-SHA1-96 client->server MAC algorithm 2019-10-07 14:38:16 Initialised AES-256 CBC server->client encryption 2019-10-07 14:38:16 Initialised HMAC-SHA1-96 server->client MAC algorithm 2019-10-07 14:38:22 Using SSPI from SECUR32.DLL 2019-10-07 14:38:22 Attempting GSSAPI authentication 2019-10-07 14:38:22 GSSAPI authentication initialisation failed 2019-10-07 14:38:22 The target was not recognized. 2019-10-07 14:38:22 Attempting keyboard-interactive authentication 2019-10-07 14:38:30 Access granted 2019-10-07 14:38:30 Opening session as main channel 2019-10-07 14:38:30 Opened main channel 2019-10-07 14:38:30 Allocated pty (ospeed 38400bps, ispeed 38400bps) 2019-10-07 14:38:31 Started a shell/command -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Tue Oct 8 18:53:28 2019 From: ingela.andin@REDACTED (Ingela Andin) Date: Tue, 8 Oct 2019 18:53:28 +0200 Subject: [erlang-questions] Wildcard certificate rejected with hostname_check_failed In-Reply-To: References: Message-ID: You need to use the ssl option: {customize_hostname_check, [{match_fun, public_key:pkix_verify_hostname_match_fun(https)}]} to allow wildcard certs! Regards Ingela Erlang/OTP team - Ericsson AB Den tors 3 okt. 2019 kl 00:22 skrev D?niel Szoboszlay : > Hi, > > I'm trying to connect to the server foobar.example.com with ss from an > Erlang node. The server presents a wildcard certificate with an > ?'id-ce-subjectAltName' extension that looks like this: > > {'Extension', {2,5,29,17}, false, > [{dNSName, "*.example.com"}, > {dNSName, "example.com"} > ]} > > The peer verification (default one) fails with hostname_check_failed. I > traced the problem back to public_key:verify_hostname_match_default0/2 > which gets called like this: > > public_key:verify_hostname_match_default0( > {dns_id, "foobar.example.com"}, > {dNSName, "*.example.com"}) > > This check fails. The function has one clause for matching a dns_id > against a dNSName, and it requires an exact match. It also has a clause > that allows wildcards, but that's only for matching a (non-wrapped) fqdn > against a cn tuple: > https://github.com/erlang/otp/blob/5bf0a8a2a18a8e883792692bdc060e37552b0d08/lib/public_key/src/public_key.erl#L1672-L1675 > > Why isn't the wildcard matching enabled for the dns_id against dNSName case > too? The fqdn against cn scenario is only used as a fallback when the > certificate doesn't contain an ?'id-ce-subjectAltName' extension. But as > far as I can tell wildcard certificates typically have the extension, > instead of (or besides) using a wildcard in the CN (see for example the > certificate presented by https://blog.hu/). > > The public_key User's Guide > > says: > > In case where the Presented IDs are fetched from the Subject certificate >> field, the names may contain wildcard characters. The function handles this >> as defined in chapter 6.4.3 in RFC 6125 >> . > > > But as I understand, chapter 6.4.3 speaks about presented ids *in general*. > Where does this restriction to the subject field come from? > > Thanks, > Daniel > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From krzysztof.jurewicz@REDACTED Tue Oct 8 22:02:05 2019 From: krzysztof.jurewicz@REDACTED (Krzysztof Jurewicz) Date: Tue, 08 Oct 2019 22:02:05 +0200 Subject: [erlang-questions] [ANN] Ercoin 1.0.0 beta 1 (plus IBO and bug bounty) Message-ID: <87bluruhtu.fsf@gmail.com> Ercoin 1.0.0 beta 1 has been released. This version brings the announcement of Initial Burn Offering which will be used to obtain initial distribution. We are also pleased to announce bug bounty programme (with rewards up to $1,000). For the full announcement and other resources, see https://ercoin.tech/news/ercoin-1.0.0-beta.1.html . The main reason for which this version is named beta, not release candidate, is that we may want to put information about genesis data (including its checksum) into version 1.0.0. This data however is not known yet ? it will be generated after the IBO (and after fee voting period). If there are planned commits, then by definition it cannot be named a release candidate (though OTP policy is different in this regard). As usual, feedback is welcome. -- Krzysztof From hans.r.nilsson@REDACTED Wed Oct 9 10:00:27 2019 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Wed, 9 Oct 2019 08:00:27 +0000 Subject: [erlang-questions] ssh:connect "Key exchange failed" In-Reply-To: References: , , Message-ID: Great! Thanks for reporting and testing it. I'll add this to maint and master in github. /Hans ________________________________ Fr?n: J?zsef B?rces Skickat: den 8 oktober 2019 17:27 Till: Hans Nilsson R ; Erlang Questions ?mne: RE: ssh:connect "Key exchange failed" Hi Hans, It works! Many thanks, Jozsef From: Hans Nilsson R Sent: Tuesday, 8 October, 2019 16:15 To: J?zsef B?rces ; Erlang Questions Subject: Sv: ssh:connect "Key exchange failed" I've found it. The server and client has no common mac algorithm. I tried to add 'hmac-sha1-96' to Erlang SSH: $ git diff diff --git a/lib/ssh/src/ssh_transport.erl b/lib/ssh/src/ssh_transport.erl index a85926354e..66d06c3360 100644 --- a/lib/ssh/src/ssh_transport.erl +++ b/lib/ssh/src/ssh_transport.erl @@ -171,6 +171,7 @@ supported_algorithms(mac) -> [{'hmac-sha2-256', [{macs,hmac}, {hashs,sha256}]}, {'hmac-sha2-512', [{macs,hmac}, {hashs,sha512}]}, {'hmac-sha1', [{macs,hmac}, {hashs,sha}]}, + {'hmac-sha1-96', [{macs,hmac}, {hashs,sha}]}, {'AEAD_AES_128_GCM', [{ciphers,aes_128_gcm}]}, {'AEAD_AES_256_GCM', [{ciphers,aes_256_gcm}]} ] I can't test it because I have no ssh server at hand that has that old mac. But it works with erlang client to erlang server. Could you patch your Erlang client, test and tell me it it works? /Hans ________________________________ Fr?n: J?zsef B?rces > Skickat: den 8 oktober 2019 14:36 Till: Hans Nilsson R >; Erlang Questions > ?mne: RE: ssh:connect "Key exchange failed" Hi Hans, I tried it with several OTP releases up to 21. I have not tried OTP 22. Here is what I received with telnet: Q?Cb8????x|q?ob?gss-group1-sha1-toWM5Slw5Ew8Mqkay+al2g==,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1ssh-rsa,ssh-dss aes256-cbc,aes192-cbc,aes128-cbc aes256-cbc,aes192-cbc,aes128-cbchmac-sha1-96,hmac-md5-96hmac-sha1-96,hmac-md5-96 none,zlib none,zlib es,i-default es,i-defaultConnection closed by foreign host. And these are the default algorithms: [{kex,['ecdh-sha2-nistp384','ecdh-sha2-nistp521', 'ecdh-sha2-nistp256','diffie-hellman-group-exchange-sha256', 'diffie-hellman-group16-sha512', 'diffie-hellman-group18-sha512', 'diffie-hellman-group14-sha256', 'diffie-hellman-group14-sha1', 'diffie-hellman-group-exchange-sha1']}, {public_key,['ecdsa-sha2-nistp384','ecdsa-sha2-nistp521', 'ecdsa-sha2-nistp256','ssh-rsa','rsa-sha2-256', 'rsa-sha2-512','ssh-dss']}, {cipher,[{client2server,['aes256-gcm@REDACTED', 'aes256-ctr','aes192-ctr','aes128-gcm@REDACTED', 'aes128-ctr','aes128-cbc','3des-cbc']}, {server2client,['aes256-gcm@REDACTED','aes256-ctr', 'aes192-ctr','aes128-gcm@REDACTED','aes128-ctr', 'aes128-cbc','3des-cbc']}]}, {mac,[{client2server,['hmac-sha2-256','hmac-sha2-512', 'hmac-sha1']}, {server2client,['hmac-sha2-256','hmac-sha2-512', 'hmac-sha1']}]}, {compression,[{client2server,[none,'zlib@REDACTED',zlib]}, {server2client,[none,'zlib@REDACTED',zlib]}]}] 5> ssh_transport:supported_algorithms(mac). [{client2server,['hmac-sha2-256','hmac-sha2-512', 'hmac-sha1','AEAD_AES_128_GCM','AEAD_AES_256_GCM']}, {server2client,['hmac-sha2-256','hmac-sha2-512','hmac-sha1', 'AEAD_AES_128_GCM','AEAD_AES_256_GCM']}] Thanks, Jozsef From: Hans Nilsson R > Sent: Tuesday, 8 October, 2019 10:47 To: J?zsef B?rces >; Erlang Questions > Subject: Sv: ssh:connect "Key exchange failed" The "Key exchange failed" error indicates that there is no agreement in the first phase where different algorithms are negotiated. A common cause is that there are no common cipher algorithm available. For example the AES-256 CBC is not available in Erlang SSH. Which otp version do you use? What does the function ssh:default_algorithms() return? What algorithms does the server provide? (Check by telnet the server, type "SSH-2.0-xxx" and look at the funny characters that is returned. Or post them here) /Hans ________________________________ Fr?n: erlang-questions-bounces@REDACTED > f?r J?zsef B?rces > Skickat: den 8 oktober 2019 09:23 Till: Erlang Questions > ?mne: [erlang-questions] ssh:connect "Key exchange failed" Hi, I need to connect to an ssh server but ssh:connect always returns "Key exchange failed". I can connect with KiTTY, and checking the KiTTY logs I suspect that the problem is the 1024-bit RSA key. Could you please check the KiTTY log below what is not supported by the Erlang SSH client? Is there any way to add the missing support to the Erlang SSH client so that I could connect to this server? Thanks, Jozsef 2019-10-07 14:37:58 Starting 42264 from XXXXX@REDACTED 2019-10-07 14:38:07 Connecting to 10.XXX.XXX.XXX port 22 2019-10-07 14:38:07 We claim version: SSH-2.0-PuTTY_KiTTY 2019-10-07 14:38:07 Server version: SSH-2.0-Sun_SSH_1.1.5 2019-10-07 14:38:07 Using SSH protocol version 2 2019-10-07 14:38:08 Doing Diffie-Hellman group exchange 2019-10-07 14:38:08 Doing Diffie-Hellman key exchange with hash SHA-1 2019-10-07 14:38:08 Server also has ssh-dss host key, but we don't know it 2019-10-07 14:38:08 Host key fingerprint is: 2019-10-07 14:38:08 ssh-rsa 1024 XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX 2019-10-07 14:38:16 Initialised AES-256 CBC client->server encryption 2019-10-07 14:38:16 Initialised HMAC-SHA1-96 client->server MAC algorithm 2019-10-07 14:38:16 Initialised AES-256 CBC server->client encryption 2019-10-07 14:38:16 Initialised HMAC-SHA1-96 server->client MAC algorithm 2019-10-07 14:38:22 Using SSPI from SECUR32.DLL 2019-10-07 14:38:22 Attempting GSSAPI authentication 2019-10-07 14:38:22 GSSAPI authentication initialisation failed 2019-10-07 14:38:22 The target was not recognized. 2019-10-07 14:38:22 Attempting keyboard-interactive authentication 2019-10-07 14:38:30 Access granted 2019-10-07 14:38:30 Opening session as main channel 2019-10-07 14:38:30 Opened main channel 2019-10-07 14:38:30 Allocated pty (ospeed 38400bps, ispeed 38400bps) 2019-10-07 14:38:31 Started a shell/command -------------- next part -------------- An HTML attachment was scrubbed... URL: From hello@REDACTED Wed Oct 9 15:26:44 2019 From: hello@REDACTED (Adam Lindberg) Date: Wed, 9 Oct 2019 15:26:44 +0200 Subject: [erlang-questions] Who uses Erlang for research? In-Reply-To: <20190902101039.GA21127@hec.corelatus.se> References: <20190902101039.GA21127@hec.corelatus.se> Message-ID: Hi Matthias, Erlang has been extensively used in the LightKone EU project, and I know also several of the project partners use it in other projects as well. More information here: https://www.lightkone.eu I can tell for a fact that we at Stritzinger GmbH (https://stritzinger.com) use it for both development and research. Cheers, Adam > On 2. Sep 2019, at 12:10, Matthias Lang wrote: > > Hi, > > I wanted to update FAQ 1.6: "Who uses Erlang for research (and > teaching)?". > > A decade or two ago, my uninvolved view of _research_ was "Uppsala, > Chalmers, Kent and Spain, and they all know each other, and then there > are some smaller efforts elsewhere." > > Looking at the ICFP 2019, the actual conference has nothing involving > Erlang, but the Erlang workshop has both familiar names and new > faces. Being a workshop, it's about applications, so maybe that's what > a settled-down Erlang is about now. > > Thoughts? > > Is there something major I've missed? > > (The 'teaching' part of the FAQ is unfortunate; it turned into an > ad-hoc list of universities where someone mailed me. I intend to chop > the list because it's such a small, arbitrary selection. I expect > Erlang to pop up quite frequently in courses about distributed/concurrent > systems, so just mentioning a few seems misleading.) > > Matthias > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From t@REDACTED Wed Oct 9 17:38:12 2019 From: t@REDACTED (Tristan Sloughter) Date: Wed, 09 Oct 2019 09:38:12 -0600 Subject: [erlang-questions] [ANN] Adopting Erlang Message-ID: <97ad3494-28dd-4ac3-8a02-2a1fd52c8a53@www.fastmail.com> Adopting Erlang (https://adoptingerlang.org/), a booksite for documenting how to do production Erlang, integrate within a company and build a team, has grown by a few chapters since the site went live a couple months ago, so I felt it was time to announce on the mailing list :). The most recently released chapter is on building Docker images https://adoptingerlang.org/docs/production/docker/, coming soon will be a chapter on Kubernetes and one on handling dependencies with rebar3. And there were already a number of important and interesting topics covered, see Hard Things to Get Right (https://adoptingerlang.org/docs/development/hard_to_get_right/) in particular. Below is the original announcement for Adopting Erlang that was posted on elixirforum.org. --- Adopting Erlang is a collaborative effort between myself (Tristan), Fred Hebert and Evan Vigil-McClanahan. It is an ongoing effort to gather all the resources that will help you use Erlang in a business. The booksite is divided in three sections focusing particularly on Erlang/OTP?s higher level concepts in the current open source ecosystem, how to use it in production (while setting up a pipeline for continuous development and delivery), and how to build a team when you?re starting from scratch. The booksite is currently a work in progress, and updates will regularly be posted, aiming for a chapter per month. You can come back here to check for new material, or follow the authors on social media to get notifications, at least until we add some RSS Feed in here. As more chapters become available we will comment on this thread with an update, however here?s what we have planned so far: Development Supervision Trees Dependencies Umbrella Projects Configuration Documentation Testing Hard Things to Get Right Unicode Time SSL Production Releases Docker Kubernetes Operations Remote access Metrics Logging Tracing Team Building Who to Put on the Team Repository Structures Processes How to Hire Read it online here: http://adoptingerlang.org/ Comments and fixes for the text can be made on the repository https://github.com/adoptingerlang/adoptingerlang From lloyd@REDACTED Wed Oct 9 18:39:23 2019 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Wed, 9 Oct 2019 12:39:23 -0400 Subject: [erlang-questions] [ANN] Adopting Erlang In-Reply-To: <97ad3494-28dd-4ac3-8a02-2a1fd52c8a53@www.fastmail.com> References: <97ad3494-28dd-4ac3-8a02-2a1fd52c8a53@www.fastmail.com> Message-ID: <48D53821-E323-4D70-891C-67F1F666120D@writersglen.com> Hi Tristan, I much look forward to reading your book. I?ve had questions recently about how to release Erlang apps into the wild. There seems to be little information on the web or in the literature that explores security issues. I?ve also recently explored Linux containers and wondered about synergies with Erlang. So, thanks for this work. Best wishes, LRP Sent from my iPad > On Oct 9, 2019, at 11:38 AM, Tristan Sloughter wrote: > > > Adopting Erlang (https://adoptingerlang.org/), a booksite for documenting how to do production Erlang, integrate within a company and build a team, has grown by a few chapters since the site went live a couple months ago, so I felt it was time to announce on the mailing list :). > > The most recently released chapter is on building Docker images https://adoptingerlang.org/docs/production/docker/, coming soon will be a chapter on Kubernetes and one on handling dependencies with rebar3. And there were already a number of important and interesting topics covered, see Hard Things to Get Right (https://adoptingerlang.org/docs/development/hard_to_get_right/) in particular. > > Below is the original announcement for Adopting Erlang that was posted on elixirforum.org. > > --- > > Adopting Erlang is a collaborative effort between myself (Tristan), Fred Hebert and Evan Vigil-McClanahan. It is an ongoing effort to gather all the resources that will help you use Erlang in a business. The booksite is divided in three sections focusing particularly on Erlang/OTP?s higher level concepts in the current open source ecosystem, how to use it in production (while setting up a pipeline for continuous development and delivery), and how to build a team when you?re starting from scratch. > > The booksite is currently a work in progress, and updates will regularly be posted, aiming for a chapter per month. You can come back here to check for new material, or follow the authors on social media to get notifications, at least until we add some RSS Feed in here. > > As more chapters become available we will comment on this thread with an update, however here?s what we have planned so far: > > Development > Supervision Trees > Dependencies > Umbrella Projects > Configuration > Documentation > Testing > Hard Things to Get Right > Unicode > Time > SSL > Production > Releases > Docker > Kubernetes > Operations > Remote access > Metrics > Logging > Tracing > Team Building > Who to Put on the Team > Repository Structures > Processes > How to Hire > > Read it online here: http://adoptingerlang.org/ > > Comments and fixes for the text can be made on the repository https://github.com/adoptingerlang/adoptingerlang > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From agutierrez@REDACTED Thu Oct 10 11:27:50 2019 From: agutierrez@REDACTED (Asier Gutierrez Rubio) Date: Thu, 10 Oct 2019 09:27:50 +0000 Subject: [erlang-questions] Erlang/OTP diameter relay doesn't work Message-ID: <1570699669588.15187@megalabs.ru> Hi there, We are trying to build a simple diameter relay agent for our company which will connect diameter clients to a number of different billing diameter servers. Right now we are just testing the system with one client and one server. Unfortunately, we are unable to get even the relay example from ericsson repository working. https://github.com/erlang/otp/tree/master/lib/diameter/examples/code We modified slightly the example to connect to our diameter server on a remote address and listen to the local port TCP/3868. The relay agent connects successfully to our diameter server, performing a full CER/CEA exchange and subsequent DWR/DWA. Our client successfully connects to the relay agent through CER/CEA, but when the client sends a Credit Control Request (AVP Request-Type:4, Requested_Action: 2, Command-Code: 272), we get 3001 DIAMETER_COMMAND_UNSUPPORTED consistently, every single time. Peer_up callback is called, but no handle_request or pick_peer callback is called. When we try to use the same client with the server directly, everything works fine and we get 2001 SUCCESS. We are sure we are missing something, maybe a parameter in the transport options, a callback that we need to implement or something like that. Could you give us a hint? Thanks in advance, Asier ________________________________ ?????????? ? ???? ????????? ????????????? ????????????? ??? ?????????? ???, ??????? ??? ??????????. ? ????????? ????? ??????????? ???????????????? ??????????, ??????? ?? ????? ???? ???????? ??? ???????????? ???-???? ????? ?????????. ???? ?? ?? ??????? ????? ?????????, ?? ?????????????, ?????????????, ??????????? ??? ??????????????? ?????????? ????????? ??? ??? ????? - ????????? ? ?????????. ???? ?? ???????? ??? ????????? ????????, ??????????, ??????????????? ???????? ??????????? ?? ???? ? ??????? ?? ???? ?????????? ???? ????????? ? ????? ????????? ??? ????? ? ??????????. The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. The contents may not be disclosed or used by anyone other than the addressee. If you are not the intended recipient(s), any use, disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this communication in error please notify us immediately by responding to this email and then delete the e-mail and all attachments and any copies thereof. -------------- next part -------------- An HTML attachment was scrubbed... URL: From anders.gs.svensson@REDACTED Thu Oct 10 12:17:44 2019 From: anders.gs.svensson@REDACTED (Anders Svensson GS) Date: Thu, 10 Oct 2019 10:17:44 +0000 Subject: [erlang-questions] Erlang/OTP diameter relay doesn't work In-Reply-To: References: Message-ID: <23967.1349.704180.756532@gargle.gargle.HOWL> Hi Asier. Assuming you've configured the relay application as in the example, the only reason you should get 3001 is if your CCR doesn't set the P-bit, in which case it has to be locally processed. Anders erlang-questions-request@REDACTED writes: > > We are trying to build a simple diameter relay agent for our company which will connect diameter clients to a number of different billing diameter servers. Right now we are just testing the system with one client and one server. > > > Unfortunately, we are unable to get even the relay example from ericsson repository working. > We modified slightly the example to connect to our diameter server on a remote address and listen to the local port TCP/3868. The relay agent connects successfully to our diameter server, performing a full CER/CEA exchange and subsequent DWR/DWA. Our client successfully connects to the relay agent through CER/CEA, but when the client sends a Credit Control Request (AVP Request-Type:4, Requested_Action: 2, Command-Code: 272), we get 3001 DIAMETER_COMMAND_UNSUPPORTED consistently, every single time. Peer_up callback is called, but no handle_request or pick_peer callback is called. When we try to use the same client with the server directly, everything works fine and we get 2001 SUCCESS. > > > We are sure we are missing something, maybe a parameter in the transport options, a callback that we need to implement or something like that. > > > Could you give us a hint? > > > Thanks in advance, > > Asier From otp@REDACTED Thu Oct 10 12:29:12 2019 From: otp@REDACTED (Erlang/OTP) Date: Thu, 10 Oct 2019 12:29:12 +0200 Subject: [erlang-questions] Patch Package OTP 22.1.3 Released Message-ID: <20191010102912.GA6259@erix.ericsson.se> Patch Package: OTP 22.1.3 Git Tag: OTP-22.1.3 Date: 2019-10-10 Trouble Report Id: OTP-16019 Seq num: System: OTP Release: 22 Application: inets-7.1.1 Predecessor: OTP 22.1.2 Check out the git tag OTP-22.1.3, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- inets-7.1.1 ----------------------------------------------------- --------------------------------------------------------------------- The inets-7.1.1 application can be applied independently of other applications on a full OTP 22 installation. --- Improvements and New Features --- OTP-16019 Application(s): inets Add HTTP server error logging vi logger Full runtime dependencies of inets-7.1.1: erts-6.0, kernel-3.0, mnesia-4.12, runtime_tools-1.8.14, ssl-5.3.4, stdlib-3.5 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From agutierrez@REDACTED Thu Oct 10 17:07:46 2019 From: agutierrez@REDACTED (Asier Gutierrez Rubio) Date: Thu, 10 Oct 2019 15:07:46 +0000 Subject: [erlang-questions] Erlang/OTP diameter relay doesn't work In-Reply-To: <23967.1349.704180.756532@gargle.gargle.HOWL> References: , <23967.1349.704180.756532@gargle.gargle.HOWL> Message-ID: <1570720065906.86256@megalabs.ru> Hi Anders, Thanks for the advice!!! You were right, we didn't have the P-bit set. Now it works fine! ________________________________________ From: Anders Svensson GS Sent: Thursday, October 10, 2019 1:17 PM To: erlang-questions@REDACTED Cc: Asier Gutierrez Rubio Subject: Re: Erlang/OTP diameter relay doesn't work Hi Asier. Assuming you've configured the relay application as in the example, the only reason you should get 3001 is if your CCR doesn't set the P-bit, in which case it has to be locally processed. Anders erlang-questions-request@REDACTED writes: > > We are trying to build a simple diameter relay agent for our company which will connect diameter clients to a number of different billing diameter servers. Right now we are just testing the system with one client and one server. > > > Unfortunately, we are unable to get even the relay example from ericsson repository working. > We modified slightly the example to connect to our diameter server on a remote address and listen to the local port TCP/3868. The relay agent connects successfully to our diameter server, performing a full CER/CEA exchange and subsequent DWR/DWA. Our client successfully connects to the relay agent through CER/CEA, but when the client sends a Credit Control Request (AVP Request-Type:4, Requested_Action: 2, Command-Code: 272), we get 3001 DIAMETER_COMMAND_UNSUPPORTED consistently, every single time. Peer_up callback is called, but no handle_request or pick_peer callback is called. When we try to use the same client with the server directly, everything works fine and we get 2001 SUCCESS. > > > We are sure we are missing something, maybe a parameter in the transport options, a callback that we need to implement or something like that. > > > Could you give us a hint? > > > Thanks in advance, > > Asier ________________________________ ?????????? ? ???? ????????? ????????????? ????????????? ??? ?????????? ???, ??????? ??? ??????????. ? ????????? ????? ??????????? ???????????????? ??????????, ??????? ?? ????? ???? ???????? ??? ???????????? ???-???? ????? ?????????. ???? ?? ?? ??????? ????? ?????????, ?? ?????????????, ?????????????, ??????????? ??? ??????????????? ?????????? ????????? ??? ??? ????? - ????????? ? ?????????. ???? ?? ???????? ??? ????????? ????????, ??????????, ??????????????? ???????? ??????????? ?? ???? ? ??????? ?? ???? ?????????? ???? ????????? ? ????? ????????? ??? ????? ? ??????????. The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. The contents may not be disclosed or used by anyone other than the addressee. If you are not the intended recipient(s), any use, disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this communication in error please notify us immediately by responding to this email and then delete the e-mail and all attachments and any copies thereof. From svenolof@REDACTED Fri Oct 11 11:11:55 2019 From: svenolof@REDACTED (Sven-Olof Nystrom) Date: Fri, 11 Oct 2019 11:11:55 +0200 Subject: [erlang-questions] A subtyping system for Erlang Message-ID: <23968.18267.733787.407181@vranx.it.uu.se> Hello, I would like to announce a type system for Erlang. This type system offers guarantees for safety; a program that types should not fail (at least, it should not fail with a type error). Typing a program requires specifications and declarations of visible functions and visible data types. As Erlang data structures such as lists and tuples cannot be modified at run-time it is a good fit to base the type system on *subtyping*. Among other advantages, subtyping allows some values to be typed as the universal type. The implementation is written in Erlang. It uses a modified Erlang parser to process Erlang modules extended with function specifications and type declarations. The type system can type itself, of course. Below follows two examples of modules that type: -module(example1). -compile(export_all). %: +type list(X) = [] + [X|list(X)]. %: +func append :: list(X) * list(X) -> !list(X). append([A | B], C) -> [A | append(B, C)]; append([], C) -> C. %: +func dup :: list(integer()) -> !list(!integer()). dup(S) -> append(S, S). The module starts with a declaration of the recursive parametric data type "list". This is followed by a specification and a definition of the recursive function append. The function "dup" is only specified for lists of integers (but that can be changed). The exclamation marks force the type system to check that something interested is returned by the functions. Let's take a look at a less orthodox example. This example gives a hint of what the type system can express, even though it is perhaps not very practical: -module(tsil). -compile(export_all). %: +type tsil(X) = [tsil(X)|X] + []. %: +func dneppa :: tsil(X) * tsil(X) -> tsil(X). dneppa([], L) -> L; dneppa([L1|X], L2) -> [ dneppa(L1, L2) | X]. %: +type list(X) = [X|list(X)] + []. %: +func from_list :: list(X) -> tsil(X). from_list([]) -> []; from_list([X|L]) -> [from_list(L)|X]. %: +func to_list :: tsil(X) -> list(X). to_list([]) -> []; to_list([L|X]) -> [X|to_list(L)]. "tsil" is of course "list" backwards. "dneppa" concatenates two "tsil"s. Included are also functions that convert between "tsil"s and lists, just to show that the type system can reason about combinations of types that use the same data type constructors in different ways. Currently there is not much documentation of the system available, beside what I have written here and the README file that comes with the source. The download page also contains some paper that describe the theory. Download: http://user.it.uu.se/~svenolof/scfa/ yours, Sven-Olof Nystr?m -- Sven-Olof Nystrom Comp Sci Dept, Uppsala University, Box 337, S-751 05 Uppsala SWEDEN svenolof@REDACTED From svenolof@REDACTED Fri Oct 11 11:40:25 2019 From: svenolof@REDACTED (Sven-Olof Nystrom) Date: Fri, 11 Oct 2019 11:40:25 +0200 Subject: [erlang-questions] A subtyping system for Erlang In-Reply-To: <7F1BB126-FD9C-4866-8C7F-DEDD6A9AA720@niemier.pl> References: <23968.18267.733787.407181@vranx.it.uu.se> <7F1BB126-FD9C-4866-8C7F-DEDD6A9AA720@niemier.pl> Message-ID: <23968.19977.464702.829157@vranx.it.uu.se> ?ukasz Niemier writes: > Out of curiosity, why this is using custom type specification > format? What are type specs lacking to work with that type checker? > -- > > ?ukasz Niemier lukasz@REDACTED The short answer is that the Dialyzer specifications did not contain the information I needed. Consider for example the specification of the foldl function in the lists module: -spec foldl(Fun, Acc0, List) -> Acc1 when Fun :: fun((Elem :: T, AccIn) -> AccOut), Acc0 :: term(), Acc1 :: term(), AccIn :: term(), AccOut :: term(), List :: [T], T :: term(). foldl(F, Accu, [Hd|Tail]) -> foldl(F, F(Hd, Accu), Tail); foldl(F, Accu, []) when is_function(F, 2) -> Accu. Note that the type of Acc0 (second argument) is set to any term. I wanted (and needed) a more specific specification so that I could give a warning for programs that would have a type error at run: +func foldl :: (X*V -> V)*V*list(X) -> V. According to this specification, the accumulator must be the of the same type as the second argument as the function parameter. There are also other reasons. I wanted to have user-defined recursive data types. Since the one you see everywhere is the list type I wanted to make that one user-defined. Sven-Olof Sven-Olof Nystrom Comp Sci Dept, Uppsala University, Box 337, S-751 05 Uppsala SWEDEN svenolof@REDACTED From ebegumisa@REDACTED Fri Oct 11 13:39:01 2019 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Fri, 11 Oct 2019 21:39:01 +1000 Subject: [erlang-questions] A subtyping system for Erlang In-Reply-To: <23968.18267.733787.407181@vranx.it.uu.se> References: <23968.18267.733787.407181@vranx.it.uu.se> Message-ID: Hello, This looks appetizing and quite a feat. So I'm following the README instructions. Everything compiled fine. However, when I try to run the tests I get the a bunch of errors of the form "File operation error: ebadf. Target: ./con.beam. Function: get_file. Process: code_server" as shown in full shell output below. I get similar error output with the bad tests and when type checking the type checker. Any ideas? Thanks in advance. - Edmond - OUTPUT >>>> Erlang/OTP 21 [erts-10.3] [smp:4:4] [ds:4:4:10] [async-threads:1] Eshell V10.3 (abort with ^G) 1> code:add_path("../ebin/"). true 2> c("../test/test_scfa.erl"). ../test/test_scfa.erl:3: Warning: export_all flag enabled - all functions will be exported {ok,test_scfa} 3> test_scfa:good(["../test/good"]). Test g004 Specs: 1 Constraints: 7=ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: ./con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/kernel-6.3/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/stdlib-3.8/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/xmerl-1.3.19/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/wx-1.8.7/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/tools-3.1/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/tftp-1.0.1/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/syntax_tools-2.1.7/ebin/con.beam. Function: get_file. Process: code_server. ** exception error: undefined function con:variance/2 in function reach:right_reachable_support/4 (reach.erl, line 169) in call from reach:right_reachable_supports/4 (reach.erl, line 165) in call from reach:reachability/2 (reach.erl, line 83) in call from reach:simplifyx/3 (reach.erl, line 25) in call from poly:solve_and_simplify/2 (poly.erl, line 152) in call from poly:collapse_dag/4 (poly.erl, line 122) in call from poly:check_fnid_list/3 (poly.erl, line 82) =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/ssl-9.2/ebin/con.beam. Function: get_file. Process: code_server. 4> =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/ssh-4.7.4/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/snmp-5.2.12/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/sasl-3.3/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/runtime_tools-1.13.2/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/reltool-0.7.8/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/public_key-1.6.5/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/parsetools-2.1.8/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/otp_mibs-1.2.1/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/os_mon-2.4.7/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/odbc-2.12.3/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/observer-2.9/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/mnesia-4.15.6/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/megaco-3.18.4/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/jinterface-1.9.1/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/inets-7.0.6/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/hipe-3.18.3/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/ftp-1.0.2/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/eunit-2.3.7/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/et-1.6.4/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/erts-10.3/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/erl_interface-3.11/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/erl_docgen-0.9/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/eldap-1.2.6/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/edoc-0.10/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/diameter-2.2/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/dialyzer-3.3.2/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/debugger-4.2.6/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/crypto-4.4.1/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/compiler-7.3.2/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/common_test-1.17/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: c:/Program Files (x86)/erl10.3/lib/asn1-5.0.8/ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: ../ebin/con.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === File operation error: ebadf. Target: con.beam. Function: get_file. Process: code_server. On Fri, 11 Oct 2019 19:11:55 +1000, Sven-Olof Nystrom wrote: > > Hello, > > I would like to announce a type system for Erlang. > > This type system offers guarantees for safety; a program that types > should not fail (at least, it should not fail with a type > error). Typing a program requires specifications and declarations of > visible functions and visible data types. > > As Erlang data structures such as lists and tuples cannot be modified > at run-time it is a good fit to base the type system on > *subtyping*. Among other advantages, subtyping allows some values to > be typed as the universal type. > > The implementation is written in Erlang. It uses a modified Erlang > parser to process Erlang modules extended with function specifications > and type declarations. The type system can type itself, of course. > > > Below follows two examples of modules that type: > > -module(example1). > > -compile(export_all). > > %: +type list(X) = [] + [X|list(X)]. > > %: +func append :: list(X) * list(X) -> !list(X). > append([A | B], C) -> > [A | append(B, C)]; > append([], C) -> C. > > > %: +func dup :: list(integer()) -> !list(!integer()). > dup(S) -> > append(S, S). > > The module starts with a declaration of the recursive parametric data > type "list". This is followed by a specification and a definition of > the recursive function append. The function "dup" is only specified > for lists of integers (but that can be changed). The exclamation marks > force the type system to check that something interested is returned > by the functions. > > Let's take a look at a less orthodox example. This example gives a > hint of what the type system can express, even though it is perhaps > not very practical: > > -module(tsil). > > -compile(export_all). > > %: +type tsil(X) = [tsil(X)|X] + []. > %: +func dneppa :: tsil(X) * tsil(X) -> tsil(X). > > dneppa([], L) -> > L; > dneppa([L1|X], L2) -> > [ dneppa(L1, L2) | X]. > > %: +type list(X) = [X|list(X)] + []. > %: +func from_list :: list(X) -> tsil(X). > from_list([]) -> > []; > from_list([X|L]) -> > [from_list(L)|X]. > > > %: +func to_list :: tsil(X) -> list(X). > to_list([]) -> > []; > to_list([L|X]) -> > [X|to_list(L)]. > > "tsil" is of course "list" backwards. "dneppa" concatenates two > "tsil"s. Included are also functions that convert between "tsil"s and > lists, just to show that the type system can reason about combinations > of types that use the same data type constructors in different ways. > > > Currently there is not much documentation of the system available, > beside what I have written here and the README file that comes with > the source. The download page also contains some paper that describe > the theory. > > Download: > http://user.it.uu.se/~svenolof/scfa/ > > > yours, > > Sven-Olof Nystr?m > > > -- Using Opera's mail client: http://www.opera.com/mail/ From svenolof@REDACTED Fri Oct 11 14:15:42 2019 From: svenolof@REDACTED (Sven-Olof Nystrom) Date: Fri, 11 Oct 2019 14:15:42 +0200 Subject: [erlang-questions] A subtyping system for Erlang In-Reply-To: References: <23968.18267.733787.407181@vranx.it.uu.se> Message-ID: <23968.29294.701712.514321@vranx.it.uu.se> Edmond Begumisa writes: > Hello, > > This looks appetizing and quite a feat. > > So I'm following the README instructions. Everything compiled fine. > However, when I try to run the tests I get the a bunch of errors of > the form "File operation error: ebadf. Target: > ./con.beam. Function: get_file. Process: code_server" as shown in > full shell output below. > > I get similar error output with the bad tests and when type > checking the type checker. > > Any ideas? > > Thanks in advance. > Sorry, I have not come across this error. In the Erlang documentation I see: Typical error reasons: ebadf The file is not opened for reading. > 3> test_scfa:good(["../test/good"]). > > Test g004 > Specs: 1 > Constraints: 7=ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === > File operation error: ebadf. Target: ./con.beam. Function: get_file. > Process: code_server. Not sure what is going on here. The system could read the test program (g004.erl) from file, parse it and generate constraints. There should not be any file IO after that. And there is no reason (for the program) to read the file con.beam (I suppose the compiler has already loaded con.beam). > =ERROR REPORT==== 11-Oct-2019::21:24:47.715000 === > File operation error: ebadf. Target: c:/Program Files > (x86)/erl10.3/lib/syntax_tools-2.1.7/ebin/con.beam. Function: get_file. > Process: code_server. > ** exception error: undefined function con:variance/2 > in function reach:right_reachable_support/4 (reach.erl, line 169) > in call from reach:right_reachable_supports/4 (reach.erl, line 165) > in call from reach:reachability/2 (reach.erl, line 83) > in call from reach:simplifyx/3 (reach.erl, line 25) > in call from poly:solve_and_simplify/2 (poly.erl, line 152) > in call from poly:collapse_dag/4 (poly.erl, line 122) > in call from poly:check_fnid_list/3 (poly.erl, line 82) The subtyping system contains a module con.erl. If the name conflicts with some module in the standard library, I could rename it. But as far as I can tell, there is no module con.erl in the library "syntax_tools". Sven-Olof -- Sven-Olof Nystrom Comp Sci Dept, Uppsala University, Box 337, S-751 05 Uppsala SWEDEN svenolof@REDACTED From bjorn@REDACTED Fri Oct 11 14:29:48 2019 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Fri, 11 Oct 2019 14:29:48 +0200 Subject: [erlang-questions] A subtyping system for Erlang In-Reply-To: <23968.29294.701712.514321@vranx.it.uu.se> References: <23968.18267.733787.407181@vranx.it.uu.se> <23968.29294.701712.514321@vranx.it.uu.se> Message-ID: On Fri, Oct 11, 2019 at 2:15 PM Sven-Olof Nystrom wrote: > > The subtyping system contains a module con.erl. If the name conflicts > with some module in the standard library, I could rename it. But as > far as I can tell, there is no module con.erl in the library > "syntax_tools". On Windows, CON is the reserved named for the console. Since file names are case insensitive, "con" also refers to the console, as does "con" with any extension. Thus, "con.beam" is a reference to the console. You should rename your module. /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From ebegumisa@REDACTED Fri Oct 11 14:40:04 2019 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Fri, 11 Oct 2019 22:40:04 +1000 Subject: [erlang-questions] A subtyping system for Erlang In-Reply-To: References: <23968.18267.733787.407181@vranx.it.uu.se> <23968.29294.701712.514321@vranx.it.uu.se> Message-ID: Thanks Bj?rn and Sven, Indeed, I looked again and found that make compiled all files except con.beam (there's no ../ebin/con.beam -- I should have noticed that before, although make should have at least failed with an error instead of silently skipping over that target). And like Bj?rn said, from the shell... >>> 7> make:files(["con.erl"], [{outdir,"../ebin"}]). Recompile: con con.erl: bad file number error >>> The previous errors make perfect sense now. It's the code server crashing when trying to find con.beam in all the code paths because of the reserved name. - Edmond - On Fri, 11 Oct 2019 22:29:48 +1000, Bj?rn Gustavsson wrote: > On Fri, Oct 11, 2019 at 2:15 PM Sven-Olof Nystrom > wrote: >> >> The subtyping system contains a module con.erl. If the name conflicts >> with some module in the standard library, I could rename it. But as >> far as I can tell, there is no module con.erl in the library >> "syntax_tools". > > On Windows, CON is the reserved named for the console. Since file > names are case insensitive, "con" also refers to the console, as does > "con" with any extension. Thus, "con.beam" is a reference to the > console. You should rename your module. > > /Bj?rn > > -- Using Opera's mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Fri Oct 11 14:44:30 2019 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Fri, 11 Oct 2019 22:44:30 +1000 Subject: [erlang-questions] A subtyping system for Erlang In-Reply-To: References: <23968.18267.733787.407181@vranx.it.uu.se> <23968.29294.701712.514321@vranx.it.uu.se> Message-ID: On Fri, 11 Oct 2019 22:40:04 +1000, Edmond Begumisa wrote: > Thanks Bj?rn and Sven, > > Indeed, I looked again and found that make compiled all files except > con.beam (there's no ../ebin/con.beam -- I should have noticed that > before, although make should have at least failed with an error instead Or rather, erlc should have ... > of silently skipping over that target). And like Bj?rn said, from the > shell... > >>>> > > 7> make:files(["con.erl"], [{outdir,"../ebin"}]). > Recompile: con > con.erl: bad file number > error > >>>> > > The previous errors make perfect sense now. It's the code server > crashing when trying to find con.beam in all the code paths because of > the reserved name. > > - Edmond - > > On Fri, 11 Oct 2019 22:29:48 +1000, Bj?rn Gustavsson > wrote: > >> On Fri, Oct 11, 2019 at 2:15 PM Sven-Olof Nystrom >> wrote: >>> >>> The subtyping system contains a module con.erl. If the name conflicts >>> with some module in the standard library, I could rename it. But as >>> far as I can tell, there is no module con.erl in the library >>> "syntax_tools". >> >> On Windows, CON is the reserved named for the console. Since file >> names are case insensitive, "con" also refers to the console, as does >> "con" with any extension. Thus, "con.beam" is a reference to the >> console. You should rename your module. >> >> /Bj?rn >> >> > > -- Using Opera's mail client: http://www.opera.com/mail/ From svenolof@REDACTED Fri Oct 11 15:02:02 2019 From: svenolof@REDACTED (Sven-Olof Nystrom) Date: Fri, 11 Oct 2019 15:02:02 +0200 Subject: [erlang-questions] A subtyping system for Erlang In-Reply-To: References: <23968.18267.733787.407181@vranx.it.uu.se> <23968.29294.701712.514321@vranx.it.uu.se> Message-ID: <23968.32074.420438.232857@vranx.it.uu.se> Bj?rn Gustavsson writes: > On Fri, Oct 11, 2019 at 2:15 PM Sven-Olof Nystrom > wrote: > > > > The subtyping system contains a module con.erl. If the name > > conflicts with some module in the standard library, I could > > rename it. But as far as I can tell, there is no module con.erl > > in the library "syntax_tools". > > On Windows, CON is the reserved named for the console. Since file > names are case insensitive, "con" also refers to the console, as > does "con" with any extension. Thus, "con.beam" is a reference to > the console. You should rename your module. Thanks. I have made a new version available. Edmond Begumisa writes: > Indeed, I looked again and found that make compiled all files except > con.beam (there's no ../ebin/con.beam -- I should have noticed that > before, although make should have at least failed with an error instead of > silently skipping over that target). And like Bj?rn said, from the shell... Sorry about the problems. Sven-Olof -- Sven-Olof Nystrom Comp Sci Dept, Uppsala University, Box 337, S-751 05 Uppsala SWEDEN svenolof@REDACTED From ebegumisa@REDACTED Fri Oct 11 15:21:35 2019 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Fri, 11 Oct 2019 23:21:35 +1000 Subject: [erlang-questions] A subtyping system for Erlang In-Reply-To: <23968.32074.420438.232857@vranx.it.uu.se> References: <23968.18267.733787.407181@vranx.it.uu.se> <23968.29294.701712.514321@vranx.it.uu.se> <23968.32074.420438.232857@vranx.it.uu.se> Message-ID: On Fri, 11 Oct 2019 23:02:02 +1000, Sven-Olof Nystrom wrote: > Bj?rn Gustavsson writes: > > On Fri, Oct 11, 2019 at 2:15 PM Sven-Olof Nystrom > > wrote: > > > > > > The subtyping system contains a module con.erl. If the name > > > conflicts with some module in the standard library, I could > > > rename it. But as far as I can tell, there is no module con.erl > > > in the library "syntax_tools". > > > > On Windows, CON is the reserved named for the console. Since file > > names are case insensitive, "con" also refers to the console, as > > does "con" with any extension. Thus, "con.beam" is a reference to > > the console. You should rename your module. > > Thanks. I have made a new version available. > All good now. Tests run. Thanks for the swift fix. - Edmond - > Edmond Begumisa writes: > > Indeed, I looked again and found that make compiled all files except > > con.beam (there's no ../ebin/con.beam -- I should have noticed that > > before, although make should have at least failed with an error > instead of > > silently skipping over that target). And like Bj?rn said, from the > shell... > > Sorry about the problems. > > > Sven-Olof > -- Using Opera's mail client: http://www.opera.com/mail/ From massimo.cesaro@REDACTED Fri Oct 11 16:22:32 2019 From: massimo.cesaro@REDACTED (Massimo Cesaro) Date: Fri, 11 Oct 2019 16:22:32 +0200 Subject: [erlang-questions] Parsing UUIDs with Leex Message-ID: Hello, I'm developing the transpiler for an IoT oriented DSL to Erlang. The idea is to take a domain specific language in input and create the erlang source code in output. In my erlang project, I'm using the venerable Leex and Yecc tools to build the lexer and parser. Using Erlang/OTP 22 [erts-10.5.2], so far, so good. Today I was trying to define a UUID data type in the DSL language using the following definition in the leex .xlr file: UUID = [0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12} and the corresponding rule: {UUID} : {token, {uuid, TokenChars, TokenLine}}. in the same file there is also the definition for an atom type: ATOM = [a-zA-Z0-9_]* with the rule {ATOM} : {token, {atom, list_to_atom(TokenChars), TokenLine}}. I found that the generated lexer cannot match this UUID 3eeb8daf-4e66-4c02-bb28-68934157e36e in input, but it matches the following tokens sequence: {atom,'3eeb8daf',25},{minus,25},{atom,'4e66',25},{minus,25},{atom,'4c02',25},{minus,25},{atom,bb28,25},{minus,25},{atom,'68934157e36e',25} The minus token is present because I'm implementing also unary and arithmetic expressions. I tested my definition with re: 7> re:run("3eeb8daf-4e66-4c02-bb28-68934157e36e", "[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}", [{capture,all,list},unicode]). {match,["3eeb8daf-4e66-4c02-bb28-68934157e36e"]} 8> re:run("3eeb8daf-4e66-4c02-bb28-68934157e36e", "[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}", [{capture,first,list},unicode]). {match,["3eeb8daf-4e66-4c02-bb28-68934157e36e"]} so I guess that at least the definition should be right. My workaround is to include the UUIDs in the source code between double quotes and then parsing UUIDs as strings, but I'm still wondering if I'm doing something wrong. Thanks, Massimo -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang.org@REDACTED Fri Oct 11 18:30:48 2019 From: erlang.org@REDACTED (Stanislaw Klekot) Date: Fri, 11 Oct 2019 18:30:48 +0200 Subject: [erlang-questions] Parsing UUIDs with Leex In-Reply-To: References: Message-ID: <20191011163048.GA24307@jarowit.net> On Fri, Oct 11, 2019 at 04:22:32PM +0200, Massimo Cesaro wrote: > Today I was trying to define a UUID data type in the DSL language using the > following definition in the leex .xlr file: > > UUID = > [0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12} > > and the corresponding rule: > {UUID} : {token, {uuid, TokenChars, TokenLine}}. Are you sure that {8} is not treated as a macro name? Also, you don't need to quote your dashes with a backslash in the regexp. -- Stanislaw Klekot From eric.pailleau@REDACTED Sat Oct 12 11:21:06 2019 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Sat, 12 Oct 2019 11:21:06 +0200 Subject: [erlang-questions] Parsing UUIDs with Leex In-Reply-To: References: Message-ID: Hi, Atom's definition is invalid. Should be : ATOM = [a-z][a-zA-Z0-9_]* Take also care of rule precedence. UUID rule should have higher precedence than ATOM rule. regards Le 11/10/2019 ? 16:22, Massimo Cesaro a ?crit?: > > ATOM = [a-zA-Z0-9_]* From foldl@REDACTED Sat Oct 12 12:15:26 2019 From: foldl@REDACTED (Zhengji Li) Date: Sat, 12 Oct 2019 10:15:26 +0000 Subject: [erlang-questions] crypto stream_state() used to be "self-contained" before OTP 22.0 Message-ID: Hi all, Could we infer that "State" is a "self-contained" piece of raw data that we can pass to different function calls? stream_encrypt(State, PlainText) -> {NewState, CipherText} Compare it with: file:write(IoDevice, Bytes) -> ok | {error, Reason} I think the answer should be "Yes". And things truly works before OTP 22.0. Although codes can be fixed for OTP 22 easily, my concern is that, if a function in OTP is declared as f(State, ...) -> {NewState, ...} Can we infer that the old State can be used again? Below is a simple test. ``` t() -> Key = <<12,137,124,112,128,97,39,1,112,248,35,27,186,168,219,244>>,? IVec = <<78,237,251,180,107,173,97,170,32,214,233,193,147,64,117, 126>>,? Data = <<1,2,3,4>>,? Enc0 = enc(Key, IVec, Data),? Dec0 = dec(Key, IVec, Enc0),? show({Data, Enc0, Dec0}),? ? Cipher = crypto:stream_init(aes_ctr, Key, IVec),? Enc1 = enc(Cipher, Data),? Dec1 = dec(Cipher, Enc1),? show({Data, Enc1, Dec1}).? % FAIL in OTP 22.0 ? enc(Key, IVec, Data) ->? Cipher = crypto:stream_init(aes_ctr, Key, IVec),? {_, Enc} = crypto:stream_encrypt(Cipher, Data),? Enc.? ? dec(Key, IVec, Data) ->? Cipher = crypto:stream_init(aes_ctr, Key, IVec),? {_, Dec} = crypto:stream_decrypt(Cipher, Data),? Dec.? ? enc(Cipher, Data) ->? {_, Enc} = crypto:stream_encrypt(Cipher, Data),? Enc.? ? dec(Cipher, Data) ->? {_, Dec} = crypto:stream_decrypt(Cipher, Data),? Dec.? ? show({A, B, A}) -> io:format("GOOD: ~p -> ~p -> ~p~n", [A,B,A]);? show({A, B, C}) -> io:format(" BAD: ~p -> ~p -> ~p~n", [A,B,C]). ``` -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimo.cesaro@REDACTED Sat Oct 12 14:39:18 2019 From: massimo.cesaro@REDACTED (Massimo Cesaro) Date: Sat, 12 Oct 2019 14:39:18 +0200 Subject: [erlang-questions] Parsing UUIDs with Leex In-Reply-To: <20191011163048.GA24307@jarowit.net> References: <20191011163048.GA24307@jarowit.net> Message-ID: On Fri, Oct 11, 2019 at 6:30 PM Stanislaw Klekot wrote: > On Fri, Oct 11, 2019 at 04:22:32PM +0200, Massimo Cesaro wrote: > > Today I was trying to define a UUID data type in the DSL language using > the > > following definition in the leex .xlr file: > > > > UUID = > > > [0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12} > > > > and the corresponding rule: > > {UUID} : {token, {uuid, TokenChars, TokenLine}}. > > Are you sure that {8} is not treated as a macro name? > Not really, I'm looking into leex source. > Also, you don't need to quote your dashes with a backslash in the > regexp. > > Yup, better safe than sorry. It didn't change the outcome, though. Best regards, Massimo > -- > Stanislaw Klekot > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ledest@REDACTED Sat Oct 12 17:05:57 2019 From: ledest@REDACTED (Led) Date: Sat, 12 Oct 2019 18:05:57 +0300 Subject: [erlang-questions] Parsing UUIDs with Leex In-Reply-To: References: Message-ID: Atom's definition is invalid. > Should be : > > ATOM = [a-z][a-zA-Z0-9_]* > > ATOM = [a-z][a-zA-Z0-9_@]* -------------- next part -------------- An HTML attachment was scrubbed... URL: From huss01@REDACTED Sat Oct 12 17:27:45 2019 From: huss01@REDACTED (=?UTF-8?Q?H=C3=A5kan_Huss?=) Date: Sat, 12 Oct 2019 17:27:45 +0200 Subject: [erlang-questions] Parsing UUIDs with Leex In-Reply-To: References: <20191011163048.GA24307@jarowit.net> Message-ID: Den l?r 12 okt. 2019 14:39Massimo Cesaro skrev: > > On Fri, Oct 11, 2019 at 6:30 PM Stanislaw Klekot > wrote: > >> On Fri, Oct 11, 2019 at 04:22:32PM +0200, Massimo Cesaro wrote: >> > Today I was trying to define a UUID data type in the DSL language using >> the >> > following definition in the leex .xlr file: >> > >> > UUID = >> > >> [0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12} >> > >> > and the corresponding rule: >> > {UUID} : {token, {uuid, TokenChars, TokenLine}}. >> >> Are you sure that {8} is not treated as a macro name? >> > > Not really, I'm looking into leex source. > Looking at the Leex docs for the supported regexes it seems that the {n} construct isn't supported. /H?kan -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Sat Oct 12 18:03:57 2019 From: eric.pailleau@REDACTED (PAILLEAU Eric) Date: Sat, 12 Oct 2019 18:03:57 +0200 Subject: [erlang-questions] Parsing UUIDs with Leex In-Reply-To: References: Message-ID: <9d047339-68ac-834e-f290-cf48717413ec@wanadoo.fr> To be exact, ATOM = ([a-z][a-zA-Z0-9_@]*|'[^']*') From massimo.cesaro@REDACTED Sun Oct 13 19:22:15 2019 From: massimo.cesaro@REDACTED (Massimo Cesaro) Date: Sun, 13 Oct 2019 19:22:15 +0200 Subject: [erlang-questions] Parsing UUIDs with Leex In-Reply-To: References: <20191011163048.GA24307@jarowit.net> Message-ID: Right, although if you read the leex.erl source code it seems that it could. Whatever. I'll stick to UUIDs as strings. Thank you all. Massimo On Sat, Oct 12, 2019 at 5:27 PM H?kan Huss wrote: > Den l?r 12 okt. 2019 14:39Massimo Cesaro skrev: > >> >> On Fri, Oct 11, 2019 at 6:30 PM Stanislaw Klekot >> wrote: >> >>> On Fri, Oct 11, 2019 at 04:22:32PM +0200, Massimo Cesaro wrote: >>> > Today I was trying to define a UUID data type in the DSL language >>> using the >>> > following definition in the leex .xlr file: >>> > >>> > UUID = >>> > >>> [0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12} >>> > >>> > and the corresponding rule: >>> > {UUID} : {token, {uuid, TokenChars, TokenLine}}. >>> >>> Are you sure that {8} is not treated as a macro name? >>> >> >> Not really, I'm looking into leex source. >> > > Looking at the Leex docs for the supported regexes it seems that the {n} > construct isn't supported. > > /H?kan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.w.andersson@REDACTED Tue Oct 15 12:43:13 2019 From: peter.w.andersson@REDACTED (Andersson, Peter) Date: Tue, 15 Oct 2019 10:43:13 +0000 Subject: [erlang-questions] Most efficient way to copy objects between Mnesia tables? Message-ID: Hey guys! I can think of a few different ways to copy the objects from one Mnesia table to another, but I'm looking for a most efficient option, in terms of time and process heap memory consumption (in case of a tradeoff, memory is most important). The tables in question are RAM only, not distributed and it's ok to perform dirty operations. No object conversion or matching required. Possibly dirty_first followed by iterations over dirty_read, copy and dirty_next, but I have a strong feeling that's thinking too much "inside the box". ?? Hit me with your best suggestions! /Peter This email may contain information which is privileged or protected against unauthorized disclosure or communication. If you are not the intended recipient, please notify the sender and delete this message and any attachments from your system without producing, distributing or retaining copies thereof or disclosing its contents to any other person. Telia Company processes emails and other files that may contain personal data in accordance with Telia Company?s Privacy Policy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Oct 16 10:07:38 2019 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 16 Oct 2019 10:07:38 +0200 Subject: [erlang-questions] [ANN] Cowboy 2.7.0 Message-ID: <93bffdc0-8b24-0e0b-09ad-682dd4913d5a@ninenines.eu> Hello! Cowboy 2.7.0 has been released! Cowboy is a small, fast and modern HTTP server for Erlang/OTP. Cowboy aims to provide a complete modern Web stack. This includes HTTP/1.1, HTTP/2, Websocket, Server-Sent Events and Webmachine-based REST. Cowboy 2.7 improves the HTTP/2 code with optimizations around the sending of DATA and WINDOW_UPDATE frames; graceful shutdown of the connection when the client is going away; and rate limiting mechanisms. New options and mechanisms have also been added to control the amount of memory Cowboy ends up using with both HTTP/1.1 and HTTP/2. Much, but not all, of this work was done to address HTTP/2 CVEs about potential denial of service. In addition, many of the experimental features introduced in previous releases have been marked stable and are now documented. See the announcement and migration guide for more details: * https://ninenines.eu/articles/cowboy-2.7.0/ * https://ninenines.eu/docs/en/cowboy/2.7/guide/migrating_from_2.6/ Cheers, -- Lo?c Hoguin https://ninenines.eu From max.lapshin@REDACTED Wed Oct 16 13:31:26 2019 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 16 Oct 2019 14:31:26 +0300 Subject: [erlang-questions] [ANN] Cowboy 2.7.0 In-Reply-To: <93bffdc0-8b24-0e0b-09ad-682dd4913d5a@ninenines.eu> References: <93bffdc0-8b24-0e0b-09ad-682dd4913d5a@ninenines.eu> Message-ID: reallly time to migrate from 1.0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From okaprinarjaya@REDACTED Wed Oct 16 13:55:09 2019 From: okaprinarjaya@REDACTED (I Gusti Ngurah Oka Prinarjaya) Date: Wed, 16 Oct 2019 18:55:09 +0700 Subject: [erlang-questions] [ANN] Cowboy 2.7.0 In-Reply-To: References: <93bffdc0-8b24-0e0b-09ad-682dd4913d5a@ninenines.eu> Message-ID: Awesome! Thank you for the hard work cheerss Pada tanggal Rab, 16 Okt 2019 18.31, Max Lapshin menulis: > reallly time to migrate from 1.0 > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.r.nilsson@REDACTED Wed Oct 16 14:20:44 2019 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Wed, 16 Oct 2019 12:20:44 +0000 Subject: [erlang-questions] crypto stream_state() used to be "self-contained" before OTP 22.0 In-Reply-To: References: Message-ID: The "State" could not be re-used in OTP-22.0 or later. /Hans Nilsson ________________________________ Fr?n: erlang-questions-bounces@REDACTED f?r Zhengji Li Skickat: den 12 oktober 2019 12:15 Till: Lista de Erlang ?mne: [erlang-questions] crypto stream_state() used to be "self-contained" before OTP 22.0 Hi all, Could we infer that "State" is a "self-contained" piece of raw data that we can pass to different function calls? stream_encrypt(State, PlainText) -> {NewState, CipherText} Compare it with: file:write(IoDevice, Bytes) -> ok | {error, Reason} I think the answer should be "Yes". And things truly works before OTP 22.0. Although codes can be fixed for OTP 22 easily, my concern is that, if a function in OTP is declared as f(State, ...) -> {NewState, ...} Can we infer that the old State can be used again? Below is a simple test. ``` t() -> Key = <<12,137,124,112,128,97,39,1,112,248,35,27,186,168,219,244>>,? IVec = <<78,237,251,180,107,173,97,170,32,214,233,193,147,64,117, 126>>,? Data = <<1,2,3,4>>,? Enc0 = enc(Key, IVec, Data),? Dec0 = dec(Key, IVec, Enc0),? show({Data, Enc0, Dec0}),? ? Cipher = crypto:stream_init(aes_ctr, Key, IVec),? Enc1 = enc(Cipher, Data),? Dec1 = dec(Cipher, Enc1),? show({Data, Enc1, Dec1}).? % FAIL in OTP 22.0 ? enc(Key, IVec, Data) ->? Cipher = crypto:stream_init(aes_ctr, Key, IVec),? {_, Enc} = crypto:stream_encrypt(Cipher, Data),? Enc.? ? dec(Key, IVec, Data) ->? Cipher = crypto:stream_init(aes_ctr, Key, IVec),? {_, Dec} = crypto:stream_decrypt(Cipher, Data),? Dec.? ? enc(Cipher, Data) ->? {_, Enc} = crypto:stream_encrypt(Cipher, Data),? Enc.? ? dec(Cipher, Data) ->? {_, Dec} = crypto:stream_decrypt(Cipher, Data),? Dec.? ? show({A, B, A}) -> io:format("GOOD: ~p -> ~p -> ~p~n", [A,B,A]);? show({A, B, C}) -> io:format(" BAD: ~p -> ~p -> ~p~n", [A,B,C]). ``` -------------- next part -------------- An HTML attachment was scrubbed... URL: From ostinelli@REDACTED Wed Oct 16 14:33:18 2019 From: ostinelli@REDACTED (Roberto Ostinelli) Date: Wed, 16 Oct 2019 14:33:18 +0200 Subject: [erlang-questions] Setting packet option in ranch Message-ID: All, I cannot see how to set the {packet, 4} option with ranch. ranch:start_listener(packet_test, packet_tcp, #{socket_opts => [{packet, 4}, {port, 4444}]}, packet_test, []), However I get a "Transport option {packet, 4} unknown or invalid" error. Any pointers? Thank you, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Oct 16 14:55:49 2019 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 16 Oct 2019 14:55:49 +0200 Subject: [erlang-questions] Setting packet option in ranch In-Reply-To: References: Message-ID: <595d5003-361e-cc45-0f95-7e42b4159902@ninenines.eu> Transport:setops(Socket, [{packet, 4}]) at the start of your protocol code. The reason you cannot set it when starting the listener is because that configuration ought to be independent of the protocol, and packet options are protocol-dependent. Cheers, On 16/10/2019 14:33, Roberto Ostinelli wrote: > All, > I cannot see how to set the {packet, 4}?option with ranch. > > ranch:start_listener(packet_test,?packet_tcp, #{socket_opts => [{packet, > 4}, {port, 4444}]},?packet_test, []), > > However I get a "Transport option {packet, 4} unknown or invalid" error. > > Any pointers? > > Thank you, > r. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin https://ninenines.eu From ostinelli@REDACTED Wed Oct 16 15:36:31 2019 From: ostinelli@REDACTED (Roberto Ostinelli) Date: Wed, 16 Oct 2019 15:36:31 +0200 Subject: [erlang-questions] Setting packet option in ranch In-Reply-To: <595d5003-361e-cc45-0f95-7e42b4159902@ninenines.eu> References: <595d5003-361e-cc45-0f95-7e42b4159902@ninenines.eu> Message-ID: Thank you Lo?c. On Wed, Oct 16, 2019 at 2:55 PM Lo?c Hoguin wrote: > Transport:setops(Socket, [{packet, 4}]) at the start of your protocol code. > > The reason you cannot set it when starting the listener is because that > configuration ought to be independent of the protocol, and packet > options are protocol-dependent. > > Cheers, > > On 16/10/2019 14:33, Roberto Ostinelli wrote: > > All, > > I cannot see how to set the {packet, 4} option with ranch. > > > > ranch:start_listener(packet_test, packet_tcp, #{socket_opts => [{packet, > > 4}, {port, 4444}]}, packet_test, []), > > > > However I get a "Transport option {packet, 4} unknown or invalid" error. > > > > Any pointers? > > > > Thank you, > > r. > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Lo?c Hoguin > https://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From duncan.attard.01@REDACTED Wed Oct 16 18:03:20 2019 From: duncan.attard.01@REDACTED (Duncan Paul Attard) Date: Wed, 16 Oct 2019 18:03:20 +0200 Subject: [erlang-questions] Who uses Erlang for research? In-Reply-To: References: <20190902101039.GA21127@hec.corelatus.se> Message-ID: Hi Matthias, At the University of Malta we teach Erlang in concurrent programming as part of a paradigms course. We also use Erlang to implement runtime monitors: this is a manifestation of a body of theoretical work on runtime verification that can be found at the project website TheoFoMon , a project hosted by Reykjav?k University in conjunction with the University of Malta. In particular, you might want to check out this book chapter , which explains in detail said tool. Best regards, Duncan > On 09 Oct 2019, at 15:26, Adam Lindberg wrote: > > Hi Matthias, > > Erlang has been extensively used in the LightKone EU project, and I know also several of the project partners use it in other projects as well. More information here: https://www.lightkone.eu > > I can tell for a fact that we at Stritzinger GmbH (https://stritzinger.com) use it for both development and research. > > Cheers, > Adam > >> On 2. Sep 2019, at 12:10, Matthias Lang wrote: >> >> Hi, >> >> I wanted to update FAQ 1.6: "Who uses Erlang for research (and >> teaching)?". >> >> A decade or two ago, my uninvolved view of _research_ was "Uppsala, >> Chalmers, Kent and Spain, and they all know each other, and then there >> are some smaller efforts elsewhere." >> >> Looking at the ICFP 2019, the actual conference has nothing involving >> Erlang, but the Erlang workshop has both familiar names and new >> faces. Being a workshop, it's about applications, so maybe that's what >> a settled-down Erlang is about now. >> >> Thoughts? >> >> Is there something major I've missed? >> >> (The 'teaching' part of the FAQ is unfortunate; it turned into an >> ad-hoc list of universities where someone mailed me. I intend to chop >> the list because it's such a small, arbitrary selection. I expect >> Erlang to pop up quite frequently in courses about distributed/concurrent >> systems, so just mentioning a few seems misleading.) >> >> Matthias >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From duncan.attard.01@REDACTED Thu Oct 17 09:00:45 2019 From: duncan.attard.01@REDACTED (Duncan Paul Attard) Date: Thu, 17 Oct 2019 09:00:45 +0200 Subject: [erlang-questions] Suspending Erlang Processes In-Reply-To: <8D9D5576-E5D8-42C4-90FD-FBAFDA7301B3@um.edu.mt> References: <5390547C-8ADF-4370-B702-0CF75BD1A91F@um.edu.mt> <8D9D5576-E5D8-42C4-90FD-FBAFDA7301B3@um.edu.mt> Message-ID: <6476E9D9-2880-4A31-8E4B-C9BA4BC26E77@um.edu.mt> Hi Kenneth, Rickard, I was wondering whether you have any suggestions regarding this please. All the best, Duncan > On 03 Oct 2019, at 11:27, Duncan Paul Attard wrote: > > Kenneth, Rickard, > > Let me give you a bit of context. > > I?m working on a runtime verification (RV) tool that focusses on components systems in an asynchronous setting. I?ve chosen Erlang because it nicely models this setting and also facilitates certain aspects in the development of said tool. Very briefly, in RV, the concept is that of instrumenting the system with other processes (called monitors in the RV community, but have nothing to do with Erlang monitors) that analyse the parts of the system (e.g., one process or a group of them, which I will refer to as a "system component") to detect and flag the infringement of some property specified over the component. > > These properties (which are written using a high-level logic such as Linear Temporal Logic, Hennessy-Milner Logic, etc.), define things like ?Process P cannot send message M to Q when such and such condition arises? or ?Process P must exit when a particular message M is sent to it?, etc. A monitor, or rather, the monitor source code, is synthesised from a property and ?attached? to the component to be monitored. The following is more or less the general workflow: > > 1. A property is written in a text file using one of the logics mentioned; > 2. The property is parsed and compiled to generate the monitor (in Erlang source code, in my particular case); > 3. The monitor is spawned as a process that analyses a system component of interest as this executes. > > The monitor needs to somehow acquire the runtime events emitted by processes, and this it does via the built-in Erlang tracing (i.e., the monitor is itself a tracer process). The important thing to note is that the monitors, despite being processes themselves, may be considered as a meta-layer over the system, and therefore, do not technically form part of the ?ordinary? implementation of the system. This means that monitors can be introduced or removed from the system as needed, and merely function as a second layer that strives to observe the system with *minimal* interference. > > This brings me to Kenneth?s point, that tracing is a tool intended for debugging/profiling purposes. I agree, and in fact, RV might be considered as a flavour of debugging or profiling that is done at runtime. It differs (amongst other things) from debugging and profiling, in that monitors are the product of autogenerated code resulting from *formal* logical properties. From what I gather, debugging or profiling obtains trace events in a similar way to the one I?m using for monitoring. I also understand and agree with you Kenneth that, if a system process is being monitored by one of my monitors, then it cannot be profiled or debugged due to the one-tracer limit imposed by the EVM. > > Also, the reason I?m not using ?dbg' but 'erlang:trace/3? directly is that I want the full flexibility of tracing (I might require it in later stages of my research). Way back when I started the project I was not aware of the extent of the functionality ?dbg? offers, and so to play it safe (and after reading Francesco and Simon?s book), I decided to go for the tracing BIFs. > > Finally, the reason I require different tracers (in my case, monitors) for different system processes (or groups) is that it makes the specification of correctness properties much more manageable. The gist of the idea is that it is far easier to specify a property over a restricted set of processes (e.g., just one process which exhibits *sequential* execution) than it is for a large number of processes, as then the property needs to account for all the possible interleavings of trace events exhibited by different processes. So in a sense, different monitors over different system components allow me to partition and view the otherwise whole trace of the system as a collection of separate traces for different components. Naturally, the monitors generated from smaller properties tend to be small and lightweight themselves, and are easier to work with. Moreover, this allows me to switch off certain monitors dynamically at runtime for system components that might not require monitoring anymore, while leaving others on. > > Since a system can be viewed as always starting from one root process, I attach (i.e., start tracing) a special root monitor to this system root process. The root monitor creates new monitors on the fly for certain child processes that are spawned by the root system process. Now, to collect trace events without loss, the root monitor is configured with ?set_on_spawn?, meaning that new children of the root system process are automatically traced by the root monitor at first. To spawn a dedicated monitor ?Mon_C' for some child process ?C?, the following is executed: > > 1. Root monitor ?Mon_R? is currently tracing the new child process ?C? ('set_on_spawn' flag was set on 'Mon_R'); > 2. The new monitor ?Mon_C? created for child process ?C? switches tracing *off* for ?C? (i.e., erlang:trace(Pid_C, false, ..)), so the (previous) monitor ?Mon_R' stops being the tracer of ?C?; > 3. New monitor ?Mon_C? switches tracing back *on* for child process ?C? and becomes its new tracer. > > To minimise trace event loss between steps 2 and 3, I was thinking of suspending child process ?C' before step 2, and resuming it after step 3 This way, ?C? is at least blocked, and cannot spawn new processes itself or send messages. I cannot however prevent other processes from sending ?C' messages, meaning that there might be a chance of ?receive? events being lost in the space of time between steps 2 and 3. Therefore, my suggestion still does not banish the problem but merely mitigates it, as steps 2 and 3 do not happen atomically. I wonder whether such a BIF could be realisable, such that the ownership of tracing can be transferred atomically between tracers without incurring any loss of trace events (between monitors ?Mon_R? and ?Mon_C? in my case). > > FYI, much of the work I?ve discussed has already been published in a previous paper we?ve written in the past. The paper can be found here: http://staff.um.edu.mt/afra1/papers/sefm17.pdf . If you?re interested please let me know. > > Many thanks for your help! > Duncan > > > > >> On 02 Oct 2019, at 09:11, Kenneth Lundin > wrote: >> >> As a follow up on Rickards answer I think it would be interesting if you can explain why you want different tracers per process? >> If we know what problem you want to solve we can most probably come with better suggestions. >> >> I also recommend that you use tracing via the dbg module which is intended to be a more user friendly API towards tracing. The trace BIFs might give some more detailed control but dbg has support for most use cases and makes it easier to do the right thing, at least that is the intention. >> >> Also worth mentioning is that the tracing mechanisms are really not intended to use to achieve a certain functionality which is part of the application, they are intended to be used temporarily for debugging/profiling purposes. Since there is only one tracer at the time the use of tracing as part of the "ordinary" implementation of an application there will be conflicts as soon as any tracing or profiling is needed and probably the intended functionality of the application will then be broken. >> >> /Kenneth, Erlang/OTP Ericsson >> >> On Tue, Oct 1, 2019 at 10:07 PM Rickard Green > wrote: >> >> >> On Mon, Sep 30, 2019 at 1:57 PM Duncan Paul Attard > wrote: >> > >> > I am tracing an Erlang process, say, `P` by invoking the BIF `erlang:trace(Pid_P, true, [set_on_spawn, procs, send, 'receive'])` from some process. As per the Erlang docs, the latter process becomes the tracer for `P`, which I shall call `Trc_Q`. >> > >> > Suppose now, that process `P` spawns a new process `Q`. Since the flag `set_on_spawn` was specified in the call to `erlang:trace/3` above, `Q` will automatically be traced by `Trc_P` as well. >> > >> > --- >> > >> > I want to spawn a **new** tracer, `Trc_Q`, and transfer the ownership of tracing `Q` to it, so that the resulting configuration will be that of process `P` being traced by tracer `Trc_P`, `Q` by `Trc_Q`. >> > >> >> Unfortunately I do not have any ideas on how to accomplish this. >> >> > However, Erlang permits **at most** one tracer per process, so I cannot achieve said configuration by invoking `erlang:trace(Pid_Q, true, ..)` from `Trc_Q`. The only way possible is to do it in two steps: >> > >> > 1. Tracer `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop `Trc_P` from tracing `Q`; >> > 2. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing `Q`. >> > >> > In the time span between steps **1.** and **2.** above, it might be possible that trace events by process `Q` are **lost** because at that moment, there is no tracer attached. One way of mitigating this is to perform the following: >> > >> > 1. Suspend process `Q` by calling `erlang:suspend_process(Pid_Q)` from `Trc_Q` (note that as per Erlang docs, `Trc_Q` remains blocked until `Q` is eventually suspended by the VM); >> > 2. `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop `Trc_P` from tracing `Q`; >> > 3. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing `Q`; >> > 4. Finally, `Trc_Q` calls `erlang:resume_process(Pid_Q)` so that `Q` can continue executing. >> > >> > From what I was able to find out, while `Q` is suspended, messages sent to it are queued, and when resumed, `Trc_Q` receives the `{trace, Pid_Q, receive, Msg}` trace events accordingly without any loss. >> > >> >> This is not a feature, it is a bug (introduced in erts 10.0, OTP 21.0) that will be fixed. The trace message should have been delivered even though the receiver was suspended. >> >> You cannot even rely on this behavior while this bug is present. If you (or any process in the system) send the suspended process a non-message signal (monitor, demonitor, link, unlink, exit, process_info, ...), the bug will be bypassed and the trace message will be delivered. >> >> > However, I am hesitant to use suspend/resume, since the Erlang docs explicitly say that these are to be used for *debugging purposes only*. >> >> Mission accomplished! :-) >> >> > Any idea as to why this is the case? >> > >> >> The language was designed with other communication primitives intended for use. Suspend/Resume was explicitly introduced for debugging purposes only, and not for usage by ordinary Erlang programs. They will most likely not disappear, but debug functionality in general are not treated as carefully by us at OTP as other ordinary functionality with regards to compatibility, etc. We for example removed the automatic deadlock prevention in suspend_process() that existed prior to erts 10.0 due to performance reasons. >> >> Regards, >> Rickard >> -- >> Rickard Green, Erlang/OTP, Ericsson AB >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth@REDACTED Thu Oct 17 09:48:30 2019 From: kenneth@REDACTED (Kenneth Lundin) Date: Thu, 17 Oct 2019 09:48:30 +0200 Subject: [erlang-questions] Suspending Erlang Processes In-Reply-To: <6476E9D9-2880-4A31-8E4B-C9BA4BC26E77@um.edu.mt> References: <5390547C-8ADF-4370-B702-0CF75BD1A91F@um.edu.mt> <8D9D5576-E5D8-42C4-90FD-FBAFDA7301B3@um.edu.mt> <6476E9D9-2880-4A31-8E4B-C9BA4BC26E77@um.edu.mt> Message-ID: Hi Duncan, My initial thought was that why do you need many tracers to achieve what you want. What do you say about the approach of having 1 global tracer which act as a dispatcher to all your monitors. As all trace messages have information about the process from which the trace event originates you can do a mapping between the message and the monitor and then distribute the message to that monitor which could act in the same way as if it was the tracer for a specific process or process group. /Regards Kenneth On Thu, Oct 17, 2019 at 9:01 AM Duncan Paul Attard < duncan.attard.01@REDACTED> wrote: > Hi Kenneth, Rickard, > > I was wondering whether you have any suggestions regarding this please. > > > All the best, > > Duncan > > > On 03 Oct 2019, at 11:27, Duncan Paul Attard > wrote: > > Kenneth, Rickard, > > Let me give you a bit of context. > > I?m working on a runtime verification (RV) tool that focusses on > components systems in an asynchronous setting. I?ve chosen Erlang because > it nicely models this setting and also facilitates certain aspects in the > development of said tool. Very briefly, in RV, the concept is that of > instrumenting the system with other processes (called monitors in the RV > community, but have nothing to do with Erlang monitors) that analyse the > parts of the system (e.g., one process or a group of them, which I will > refer to as a "system component") to detect and flag the infringement of > some property specified over the component. > > These properties (which are written using a high-level logic such as > Linear Temporal Logic, Hennessy-Milner Logic, etc.), define things like > ?Process P cannot send message M to Q when such and such condition arises? > or ?Process P must exit when a particular message M is sent to it?, etc. A > monitor, or rather, the monitor source code, is synthesised from a property > and ?attached? to the component to be monitored. The following is more or > less the general workflow: > > 1. A property is written in a text file using one of the logics mentioned; > 2. The property is parsed and compiled to generate the monitor (in Erlang > source code, in my particular case); > 3. The monitor is spawned as a process that analyses a system component of > interest as this executes. > > The monitor needs to somehow acquire the runtime events emitted by > processes, and this it does via the built-in Erlang tracing (i.e., the > monitor is itself a tracer process). The important thing to note is that > the monitors, despite being processes themselves, may be considered as a > meta-layer over the system, and therefore, do not technically form part of > the ?ordinary? implementation of the system. This means that monitors can > be introduced or removed from the system as needed, and merely function as > a second layer that strives to observe the system with *minimal* > interference. > > This brings me to Kenneth?s point, that tracing is a tool intended for > debugging/profiling purposes. I agree, and in fact, RV might be considered > as a flavour of debugging or profiling that is done at runtime. It differs > (amongst other things) from debugging and profiling, in that monitors are > the product of autogenerated code resulting from *formal* logical > properties. From what I gather, debugging or profiling obtains trace events > in a similar way to the one I?m using for monitoring. I also understand and > agree with you Kenneth that, if a system process is being monitored by one > of my monitors, then it cannot be profiled or debugged due to the > one-tracer limit imposed by the EVM. > > Also, the reason I?m not using ?dbg' but 'erlang:trace/3? directly is that > I want the full flexibility of tracing (I might require it in later stages > of my research). Way back when I started the project I was not aware of the > extent of the functionality ?dbg? offers, and so to play it safe (and after > reading Francesco and Simon?s book), I decided to go for the tracing BIFs. > > Finally, the reason I require different tracers (in my case, monitors) for > different system processes (or groups) is that it makes the specification > of correctness properties much more manageable. The gist of the idea is > that it is far easier to specify a property over a restricted set of > processes (e.g., just one process which exhibits *sequential* execution) > than it is for a large number of processes, as then the property needs to > account for all the possible interleavings of trace events exhibited by > different processes. So in a sense, different monitors over different > system components allow me to partition and view the otherwise whole trace > of the system as a collection of separate traces for different components. > Naturally, the monitors generated from smaller properties tend to be small > and lightweight themselves, and are easier to work with. Moreover, this > allows me to switch off certain monitors dynamically at runtime for system > components that might not require monitoring anymore, while leaving others > on. > > Since a system can be viewed as always starting from one root process, I > attach (i.e., start tracing) a special root monitor to this system root > process. The root monitor creates new monitors on the fly for certain child > processes that are spawned by the root system process. Now, to collect > trace events without loss, the root monitor is configured with > ?set_on_spawn?, meaning that new children of the root system process are > automatically traced by the root monitor at first. To spawn a dedicated > monitor ?Mon_C' for some child process ?C?, the following is executed: > > 1. Root monitor ?Mon_R? is currently tracing the new child process ?C? > ('set_on_spawn' flag was set on 'Mon_R'); > 2. The new monitor ?Mon_C? created for child process ?C? switches tracing > *off* for ?C? (i.e., erlang:trace(Pid_C, false, ..)), so the (previous) > monitor ?Mon_R' stops being the tracer of ?C?; > 3. New monitor ?Mon_C? switches tracing back *on* for child process ?C? > and becomes its new tracer. > > To minimise trace event loss between steps 2 and 3, I was thinking of > suspending child process ?C' before step 2, and resuming it after step 3 > This way, ?C? is at least blocked, and cannot spawn new processes itself or > send messages. I cannot however prevent other processes from sending ?C' > messages, meaning that there might be a chance of ?receive? events being > lost in the space of time between steps 2 and 3. Therefore, my suggestion > still does not banish the problem but merely mitigates it, as steps 2 and 3 > do not happen atomically. I wonder whether such a BIF could be realisable, > such that the ownership of tracing can be transferred atomically between > tracers without incurring any loss of trace events (between monitors > ?Mon_R? and ?Mon_C? in my case). > > FYI, much of the work I?ve discussed has already been published in a > previous paper we?ve written in the past. The paper can be found here: > http://staff.um.edu.mt/afra1/papers/sefm17.pdf. If you?re interested > please let me know. > > Many thanks for your help! > Duncan > > > > > On 02 Oct 2019, at 09:11, Kenneth Lundin wrote: > > As a follow up on Rickards answer I think it would be interesting if you > can explain why you want different tracers per process? > If we know what problem you want to solve we can most probably come with > better suggestions. > > I also recommend that you use tracing via the dbg module which is intended > to be a more user friendly API towards tracing. The trace BIFs might give > some more detailed control but dbg has support for most use cases and makes > it easier to do the right thing, at least that is the intention. > > Also worth mentioning is that the tracing mechanisms are really not > intended to use to achieve a certain functionality which is part of the > application, they are intended to be used temporarily for > debugging/profiling purposes. Since there is only one tracer at the time > the use of tracing as part of the "ordinary" implementation of an > application there will be conflicts as soon as any tracing or profiling is > needed and probably the intended functionality of the application will then > be broken. > > /Kenneth, Erlang/OTP Ericsson > > On Tue, Oct 1, 2019 at 10:07 PM Rickard Green wrote: > >> >> >> On Mon, Sep 30, 2019 at 1:57 PM Duncan Paul Attard < >> duncan.attard.01@REDACTED> wrote: >> > >> > I am tracing an Erlang process, say, `P` by invoking the BIF >> `erlang:trace(Pid_P, true, [set_on_spawn, procs, send, 'receive'])` from >> some process. As per the Erlang docs, the latter process becomes the tracer >> for `P`, which I shall call `Trc_Q`. >> > >> > Suppose now, that process `P` spawns a new process `Q`. Since the flag >> `set_on_spawn` was specified in the call to `erlang:trace/3` above, `Q` >> will automatically be traced by `Trc_P` as well. >> > >> > --- >> > >> > I want to spawn a **new** tracer, `Trc_Q`, and transfer the ownership >> of tracing `Q` to it, so that the resulting configuration will be that of >> process `P` being traced by tracer `Trc_P`, `Q` by `Trc_Q`. >> > >> >> Unfortunately I do not have any ideas on how to accomplish this. >> >> > However, Erlang permits **at most** one tracer per process, so I cannot >> achieve said configuration by invoking `erlang:trace(Pid_Q, true, ..)` from >> `Trc_Q`. The only way possible is to do it in two steps: >> > >> > 1. Tracer `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop >> `Trc_P` from tracing `Q`; >> > 2. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing >> `Q`. >> > >> > In the time span between steps **1.** and **2.** above, it might be >> possible that trace events by process `Q` are **lost** because at that >> moment, there is no tracer attached. One way of mitigating this is to >> perform the following: >> > >> > 1. Suspend process `Q` by calling `erlang:suspend_process(Pid_Q)` from >> `Trc_Q` (note that as per Erlang docs, `Trc_Q` remains blocked until `Q` is >> eventually suspended by the VM); >> > 2. `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop `Trc_P` from >> tracing `Q`; >> > 3. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing >> `Q`; >> > 4. Finally, `Trc_Q` calls `erlang:resume_process(Pid_Q)` so that `Q` >> can continue executing. >> > >> > From what I was able to find out, while `Q` is suspended, messages sent >> to it are queued, and when resumed, `Trc_Q` receives the `{trace, Pid_Q, >> receive, Msg}` trace events accordingly without any loss. >> > >> >> This is not a feature, it is a bug (introduced in erts 10.0, OTP 21.0) >> that will be fixed. The trace message should have been delivered even >> though the receiver was suspended. >> >> You cannot even rely on this behavior while this bug is present. If you >> (or any process in the system) send the suspended process a non-message >> signal (monitor, demonitor, link, unlink, exit, process_info, ...), the bug >> will be bypassed and the trace message will be delivered. >> >> > However, I am hesitant to use suspend/resume, since the Erlang docs >> explicitly say that these are to be used for *debugging purposes only*. >> >> Mission accomplished! :-) >> >> > Any idea as to why this is the case? >> > >> >> The language was designed with other communication primitives intended >> for use. Suspend/Resume was explicitly introduced for debugging purposes >> only, and not for usage by ordinary Erlang programs. They will most likely >> not disappear, but debug functionality in general are not treated as >> carefully by us at OTP as other ordinary functionality with regards to >> compatibility, etc. We for example removed the automatic deadlock >> prevention in suspend_process() that existed prior to erts 10.0 due to >> performance reasons. >> >> Regards, >> Rickard >> -- >> Rickard Green, Erlang/OTP, Ericsson AB >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From duncan.attard.01@REDACTED Thu Oct 17 14:34:57 2019 From: duncan.attard.01@REDACTED (Duncan Paul Attard) Date: Thu, 17 Oct 2019 14:34:57 +0200 Subject: [erlang-questions] Suspending Erlang Processes In-Reply-To: References: <5390547C-8ADF-4370-B702-0CF75BD1A91F@um.edu.mt> <8D9D5576-E5D8-42C4-90FD-FBAFDA7301B3@um.edu.mt> <6476E9D9-2880-4A31-8E4B-C9BA4BC26E77@um.edu.mt> Message-ID: <4E153A50-EEAA-40E7-B959-026A69E20DFE@um.edu.mt> Hi Kenneth, Thanks for your reply. I wanted different tracers since I would like to target specific processes, and not just all of them. I had initially thought about your approach. However I?m afraid that for my case, having a central tracer that would distribute the different trace events to different monitors (according to the originating PID) could mean that I have to collect trace events even for processes that I don?t need to monitor (I would just filter these out, albeit this is extra processing). My idea was to keep tracing to a minimum for the sake of performance by tracing processes selectively (and dynamically at runtime). A second shortcoming I see with this approach is that potentially, the central tracer might at times experience high loads due to the trace events that could collect in its mailbox while it is busy routing or filtering trace events: this in turn could keep recipient monitors waiting longer than necessary just to receive a handful of trace events. I suspect that this might also impact memory consumption. What is more, in the case where the central tracer fails, all events would be lost. Separate tracers could mitigate these two issues, since they are less likely to create a hotspot, and independent tracers may fail without hampering the progress of other tracers/monitors. With reference to my previous question, "I wonder whether such a BIF could be realisable, such that the ownership of tracing can be transferred atomically between tracers without incurring any loss of trace events?, would such a BIF be possible to implement in the foreseeable future? Best, Duncan > On 17 Oct 2019, at 09:48, Kenneth Lundin wrote: > > Hi Duncan, > > My initial thought was that why do you need many tracers to achieve what you want. > What do you say about the approach of having 1 global tracer which act as a dispatcher to all your monitors. > As all trace messages have information about the process from which the trace event originates you can do a mapping between > the message and the monitor and then distribute the message to that monitor which could act in the same way as if it was the tracer for a specific process or process group. > > /Regards Kenneth > > On Thu, Oct 17, 2019 at 9:01 AM Duncan Paul Attard > wrote: > Hi Kenneth, Rickard, > > I was wondering whether you have any suggestions regarding this please. > > > All the best, > > Duncan > > >> On 03 Oct 2019, at 11:27, Duncan Paul Attard > wrote: >> >> Kenneth, Rickard, >> >> Let me give you a bit of context. >> >> I?m working on a runtime verification (RV) tool that focusses on components systems in an asynchronous setting. I?ve chosen Erlang because it nicely models this setting and also facilitates certain aspects in the development of said tool. Very briefly, in RV, the concept is that of instrumenting the system with other processes (called monitors in the RV community, but have nothing to do with Erlang monitors) that analyse the parts of the system (e.g., one process or a group of them, which I will refer to as a "system component") to detect and flag the infringement of some property specified over the component. >> >> These properties (which are written using a high-level logic such as Linear Temporal Logic, Hennessy-Milner Logic, etc.), define things like ?Process P cannot send message M to Q when such and such condition arises? or ?Process P must exit when a particular message M is sent to it?, etc. A monitor, or rather, the monitor source code, is synthesised from a property and ?attached? to the component to be monitored. The following is more or less the general workflow: >> >> 1. A property is written in a text file using one of the logics mentioned; >> 2. The property is parsed and compiled to generate the monitor (in Erlang source code, in my particular case); >> 3. The monitor is spawned as a process that analyses a system component of interest as this executes. >> >> The monitor needs to somehow acquire the runtime events emitted by processes, and this it does via the built-in Erlang tracing (i.e., the monitor is itself a tracer process). The important thing to note is that the monitors, despite being processes themselves, may be considered as a meta-layer over the system, and therefore, do not technically form part of the ?ordinary? implementation of the system. This means that monitors can be introduced or removed from the system as needed, and merely function as a second layer that strives to observe the system with *minimal* interference. >> >> This brings me to Kenneth?s point, that tracing is a tool intended for debugging/profiling purposes. I agree, and in fact, RV might be considered as a flavour of debugging or profiling that is done at runtime. It differs (amongst other things) from debugging and profiling, in that monitors are the product of autogenerated code resulting from *formal* logical properties. From what I gather, debugging or profiling obtains trace events in a similar way to the one I?m using for monitoring. I also understand and agree with you Kenneth that, if a system process is being monitored by one of my monitors, then it cannot be profiled or debugged due to the one-tracer limit imposed by the EVM. >> >> Also, the reason I?m not using ?dbg' but 'erlang:trace/3? directly is that I want the full flexibility of tracing (I might require it in later stages of my research). Way back when I started the project I was not aware of the extent of the functionality ?dbg? offers, and so to play it safe (and after reading Francesco and Simon?s book), I decided to go for the tracing BIFs. >> >> Finally, the reason I require different tracers (in my case, monitors) for different system processes (or groups) is that it makes the specification of correctness properties much more manageable. The gist of the idea is that it is far easier to specify a property over a restricted set of processes (e.g., just one process which exhibits *sequential* execution) than it is for a large number of processes, as then the property needs to account for all the possible interleavings of trace events exhibited by different processes. So in a sense, different monitors over different system components allow me to partition and view the otherwise whole trace of the system as a collection of separate traces for different components. Naturally, the monitors generated from smaller properties tend to be small and lightweight themselves, and are easier to work with. Moreover, this allows me to switch off certain monitors dynamically at runtime for system components that might not require monitoring anymore, while leaving others on. >> >> Since a system can be viewed as always starting from one root process, I attach (i.e., start tracing) a special root monitor to this system root process. The root monitor creates new monitors on the fly for certain child processes that are spawned by the root system process. Now, to collect trace events without loss, the root monitor is configured with ?set_on_spawn?, meaning that new children of the root system process are automatically traced by the root monitor at first. To spawn a dedicated monitor ?Mon_C' for some child process ?C?, the following is executed: >> >> 1. Root monitor ?Mon_R? is currently tracing the new child process ?C? ('set_on_spawn' flag was set on 'Mon_R'); >> 2. The new monitor ?Mon_C? created for child process ?C? switches tracing *off* for ?C? (i.e., erlang:trace(Pid_C, false, ..)), so the (previous) monitor ?Mon_R' stops being the tracer of ?C?; >> 3. New monitor ?Mon_C? switches tracing back *on* for child process ?C? and becomes its new tracer. >> >> To minimise trace event loss between steps 2 and 3, I was thinking of suspending child process ?C' before step 2, and resuming it after step 3 This way, ?C? is at least blocked, and cannot spawn new processes itself or send messages. I cannot however prevent other processes from sending ?C' messages, meaning that there might be a chance of ?receive? events being lost in the space of time between steps 2 and 3. Therefore, my suggestion still does not banish the problem but merely mitigates it, as steps 2 and 3 do not happen atomically. I wonder whether such a BIF could be realisable, such that the ownership of tracing can be transferred atomically between tracers without incurring any loss of trace events (between monitors ?Mon_R? and ?Mon_C? in my case). >> >> FYI, much of the work I?ve discussed has already been published in a previous paper we?ve written in the past. The paper can be found here: http://staff.um.edu.mt/afra1/papers/sefm17.pdf . If you?re interested please let me know. >> >> Many thanks for your help! >> Duncan >> >> >> >> >>> On 02 Oct 2019, at 09:11, Kenneth Lundin > wrote: >>> >>> As a follow up on Rickards answer I think it would be interesting if you can explain why you want different tracers per process? >>> If we know what problem you want to solve we can most probably come with better suggestions. >>> >>> I also recommend that you use tracing via the dbg module which is intended to be a more user friendly API towards tracing. The trace BIFs might give some more detailed control but dbg has support for most use cases and makes it easier to do the right thing, at least that is the intention. >>> >>> Also worth mentioning is that the tracing mechanisms are really not intended to use to achieve a certain functionality which is part of the application, they are intended to be used temporarily for debugging/profiling purposes. Since there is only one tracer at the time the use of tracing as part of the "ordinary" implementation of an application there will be conflicts as soon as any tracing or profiling is needed and probably the intended functionality of the application will then be broken. >>> >>> /Kenneth, Erlang/OTP Ericsson >>> >>> On Tue, Oct 1, 2019 at 10:07 PM Rickard Green > wrote: >>> >>> >>> On Mon, Sep 30, 2019 at 1:57 PM Duncan Paul Attard > wrote: >>> > >>> > I am tracing an Erlang process, say, `P` by invoking the BIF `erlang:trace(Pid_P, true, [set_on_spawn, procs, send, 'receive'])` from some process. As per the Erlang docs, the latter process becomes the tracer for `P`, which I shall call `Trc_Q`. >>> > >>> > Suppose now, that process `P` spawns a new process `Q`. Since the flag `set_on_spawn` was specified in the call to `erlang:trace/3` above, `Q` will automatically be traced by `Trc_P` as well. >>> > >>> > --- >>> > >>> > I want to spawn a **new** tracer, `Trc_Q`, and transfer the ownership of tracing `Q` to it, so that the resulting configuration will be that of process `P` being traced by tracer `Trc_P`, `Q` by `Trc_Q`. >>> > >>> >>> Unfortunately I do not have any ideas on how to accomplish this. >>> >>> > However, Erlang permits **at most** one tracer per process, so I cannot achieve said configuration by invoking `erlang:trace(Pid_Q, true, ..)` from `Trc_Q`. The only way possible is to do it in two steps: >>> > >>> > 1. Tracer `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop `Trc_P` from tracing `Q`; >>> > 2. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing `Q`. >>> > >>> > In the time span between steps **1.** and **2.** above, it might be possible that trace events by process `Q` are **lost** because at that moment, there is no tracer attached. One way of mitigating this is to perform the following: >>> > >>> > 1. Suspend process `Q` by calling `erlang:suspend_process(Pid_Q)` from `Trc_Q` (note that as per Erlang docs, `Trc_Q` remains blocked until `Q` is eventually suspended by the VM); >>> > 2. `Trc_Q` calls `erlang:trace(Pid_Q, false, ..)` to stop `Trc_P` from tracing `Q`; >>> > 3. `Trc_Q` calls `erlang:trace(Pid_Q, true, ..)` again to start tracing `Q`; >>> > 4. Finally, `Trc_Q` calls `erlang:resume_process(Pid_Q)` so that `Q` can continue executing. >>> > >>> > From what I was able to find out, while `Q` is suspended, messages sent to it are queued, and when resumed, `Trc_Q` receives the `{trace, Pid_Q, receive, Msg}` trace events accordingly without any loss. >>> > >>> >>> This is not a feature, it is a bug (introduced in erts 10.0, OTP 21.0) that will be fixed. The trace message should have been delivered even though the receiver was suspended. >>> >>> You cannot even rely on this behavior while this bug is present. If you (or any process in the system) send the suspended process a non-message signal (monitor, demonitor, link, unlink, exit, process_info, ...), the bug will be bypassed and the trace message will be delivered. >>> >>> > However, I am hesitant to use suspend/resume, since the Erlang docs explicitly say that these are to be used for *debugging purposes only*. >>> >>> Mission accomplished! :-) >>> >>> > Any idea as to why this is the case? >>> > >>> >>> The language was designed with other communication primitives intended for use. Suspend/Resume was explicitly introduced for debugging purposes only, and not for usage by ordinary Erlang programs. They will most likely not disappear, but debug functionality in general are not treated as carefully by us at OTP as other ordinary functionality with regards to compatibility, etc. We for example removed the automatic deadlock prevention in suspend_process() that existed prior to erts 10.0 due to performance reasons. >>> >>> Regards, >>> Rickard >>> -- >>> Rickard Green, Erlang/OTP, Ericsson AB >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.devoy@REDACTED Thu Oct 17 15:11:17 2019 From: peter.devoy@REDACTED (Peter Devoy) Date: Thu, 17 Oct 2019 14:11:17 +0100 Subject: [erlang-questions] line_delimiter option in gen_tcp Message-ID: Hi list TL;DR: can gen_tcp packets be terminated by null byte? A few years ago I was looking at erlang for a pet project -- an XML socket server for an old multiplayer Flash game. The XML packets are terminated by null bytes and I swear I had gen_tcp configured and working with {packet, line} and {line_delimiter, \00}. Only now I can't find the code and gen_tcp does not appear to have a documented or undocumented line_delimiter option. I can only find that option referenced in inet. Am I imagening the whole thing? Is there some way to achieve packet delimitation by null byte using gen_tcp? I am an erlang noob so any help would be much appreciated! Cheers Pete -------------- next part -------------- An HTML attachment was scrubbed... URL: From leonard.boyce@REDACTED Thu Oct 17 18:59:09 2019 From: leonard.boyce@REDACTED (Leonard B) Date: Thu, 17 Oct 2019 12:59:09 -0400 Subject: [erlang-questions] line_delimiter option in gen_tcp In-Reply-To: References: Message-ID: Hi Pete, Maybe you're looking for inet:set_opts/2 ? Leonard On Thu, Oct 17, 2019 at 11:51 AM Peter Devoy wrote: > Hi list > > TL;DR: can gen_tcp packets be terminated by null byte? > > A few years ago I was looking at erlang for a pet project -- an XML socket > server for an old multiplayer Flash game. > > The XML packets are terminated by null bytes and I swear I had gen_tcp > configured and working with {packet, line} and {line_delimiter, \00}. > > Only now I can't find the code and gen_tcp does not appear to have a > documented or undocumented line_delimiter option. I can only find that > option referenced in inet. > > Am I imagening the whole thing? Is there some way to achieve packet > delimitation by null byte using gen_tcp? > > I am an erlang noob so any help would be much appreciated! > > Cheers > > > Pete > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vincefoley@REDACTED Thu Oct 17 20:39:26 2019 From: vincefoley@REDACTED (Vince Foley) Date: Thu, 17 Oct 2019 11:39:26 -0700 Subject: [erlang-questions] Match error in `hackney_pool:queue_out` Message-ID: Hi folks, I've been seeing a strange match error in hackney recently.. ** (MatchError) no match of right hand side value: {:empty, {[], []}} (hackney) /app/deps/hackney/src/hackney_pool.erl:509: :hackney_pool.queue_out/2 Anyone seen this or have any clues as to what might be happening? I wrote up more details in this ticket: https://github.com/benoitc/hackney/issues/594 hackney 1.15.2 erlang 22.1.3 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andras.Bekes@REDACTED Fri Oct 18 16:33:40 2019 From: Andras.Bekes@REDACTED (Bekes, Andras G) Date: Fri, 18 Oct 2019 14:33:40 +0000 Subject: [erlang-questions] Segfault with Erlang R22 Message-ID: <649d87fb8e0346f7ad0a039e483a46af@morganstanley.com> Hi All, After upgrading to Erlang R22, my software crashes the Erlang VM with Segmentation fault. It happens rarely, only after several days of test workload, so I can't really reproduce. I made more than 10 core dumps so far, loaded them into gdb, and all of them died at these 2 crash points: Program terminated with signal 11, Segmentation fault. #0 process_main (x_reg_array=0x20002, f_reg_array=0x2ade29280590) at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:4064 4064 if (is_not_tuple(r(0))) { or #0 process_main (x_reg_array=0x20002, f_reg_array=0x2) at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:5252 5252 c_p->seq_trace_lastcnt = unsigned_val(SEQ_TRACE_TOKEN_SERIAL(c_p)); The software is not doing any tracing when the crash happens, nor does it have any NIFs. What should be the next step of investigation? Thanks Andras G. Bekes ________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikpelinux@REDACTED Fri Oct 18 20:02:35 2019 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Fri, 18 Oct 2019 20:02:35 +0200 Subject: [erlang-questions] Segfault with Erlang R22 In-Reply-To: <649d87fb8e0346f7ad0a039e483a46af@morganstanley.com> References: <649d87fb8e0346f7ad0a039e483a46af@morganstanley.com> Message-ID: On Fri, Oct 18, 2019 at 4:34 PM Bekes, Andras G wrote: > > Hi All, > > > > After upgrading to Erlang R22, my software crashes the Erlang VM with Segmentation fault. > > It happens rarely, only after several days of test workload, so I can?t really reproduce. > > > > I made more than 10 core dumps so far, loaded them into gdb, and all of them died at these 2 crash points: > > > > Program terminated with signal 11, Segmentation fault. > > #0 process_main (x_reg_array=0x20002, f_reg_array=0x2ade29280590) at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:4064 > > 4064 if (is_not_tuple(r(0))) { > > > > or > > > > #0 process_main (x_reg_array=0x20002, f_reg_array=0x2) at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:5252 > > 5252 c_p->seq_trace_lastcnt = unsigned_val(SEQ_TRACE_TOKEN_SERIAL(c_p)); > > > > The software is not doing any tracing when the crash happens, nor does it have any NIFs. I think you should open a bug report in the erlang bug tracker. The second crash site above is in the remove_message() function, in a block where ERL_MESSAGE_TOKEN(msgp) is neither NIL nor am_undefined, but SEQ_TRACE_TOKEN(c_p) is invalid (not the expected 5-tuple). Maybe printing *c_p in gdb when that happens could shed some light. /Mikael From curtis@REDACTED Fri Oct 18 21:52:33 2019 From: curtis@REDACTED (Curtis J Schofield) Date: Fri, 18 Oct 2019 19:52:33 +0000 Subject: [erlang-questions] SSL Out of Order Cert Chain Question (9.2) Message-ID: Dear Erlang Questions: SSL 9.0.2 mentions a patch to fix out of order cert chains In SSL 9.2 we have a root CA and an out of order cert chain for host hooks.glip.com. When we try to verify peer with the out of order cert chain we get 'Unknown CA'. Is this expected behaviour for Erlang SSL 9.2 with verify_peer ? The http://erlang.org/doc/apps/ssl/notes.html#ssl-9.0.2 notes mention that other care may need to be taken to ensure compatibility. Reproduce error: https://github.com/robotarmy/out-of-order-ssl Thank you, Curtis and Team DevEco Sent through ProtonMail Encrypted Email Channel. From peter.devoy@REDACTED Sat Oct 19 02:18:48 2019 From: peter.devoy@REDACTED (Peter Devoy) Date: Sat, 19 Oct 2019 01:18:48 +0100 Subject: [erlang-questions] line_delimiter option in gen_tcp In-Reply-To: References: Message-ID: Hi Scott, Leonard, Thanks for your help. I have now experimented further with inet:setopts/2 and, although it feels like it should work, it still is not behaving as I would expect. I am going to try and write the most simple programme possible that exemplifies the behaviour and post it on StackOverflow and/or here if it still doesn't work as expected. Kind regards Pete -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.devoy@REDACTED Sat Oct 19 15:00:48 2019 From: peter.devoy@REDACTED (Peter Devoy) Date: Sat, 19 Oct 2019 14:00:48 +0100 Subject: [erlang-questions] line_delimiter option in gen_tcp In-Reply-To: References: Message-ID: Hi list Figured it out. For completeness and reference, my mistakes were two-fold: 1) Setting options on Listen socket instead of Accept socket. 2) Did not pattern match for "ok" on inet:setopts/2 so didn't realise typing was strict: inet:setopts(AcceptSocket, [{line_delimiter, [0]}]). %fails inet:setopts(AcceptSocket, [{line_delimiter, "\0"}]). %fails Eventually I got it to work with char notation but not before this mistake: inet:setopts(AcceptSocket, [{line_delimiter, $0}]). %ascii numeral 0 (fails) inet:setopts(AcceptSocket, [{line_delimiter, $\0}]). %ascii NUL (works!) Final proof: https://gist.github.com/PeteDevoy/30476594a1488c78106af1efe7de44bd Thanks again Pete -------------- next part -------------- An HTML attachment was scrubbed... URL: From v@REDACTED Sat Oct 19 15:39:44 2019 From: v@REDACTED (Valentin Micic) Date: Sat, 19 Oct 2019 15:39:44 +0200 Subject: [erlang-questions] Binary pattern matching Message-ID: My sincere apology if this has been discussed before? I vaguely remember that a number of years ago, one was able to write something like this: some_function( Offset, <<_:Offset/binary-unit:8, Value/binary>> ) -> ? without (Erlang R21.1) compiler complaining with: variable Offset unbound Was I dreaming? Thanks in advance V/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gomoripeti@REDACTED Sat Oct 19 19:00:01 2019 From: gomoripeti@REDACTED (=?UTF-8?B?UGV0aSBHw7Ztw7ZyaQ==?=) Date: Sat, 19 Oct 2019 12:00:01 -0500 Subject: [erlang-questions] Binary pattern matching In-Reply-To: References: Message-ID: hi Valentin, Maybe you remember the case when Offset was bound within the binary? <> On Sat, Oct 19, 2019 at 8:39 AM Valentin Micic wrote: > My sincere apology if this has been discussed before? > I vaguely remember that a number of years ago, one was able to write > something like this: > > *some_function( Offset, <<_:Offset/binary-unit:8, Value/binary>> ) -> ?* > > without (Erlang R21.1) compiler complaining with: *variable Offset > unbound* > > Was I dreaming? > > Thanks in advance > > V/ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From v@REDACTED Sat Oct 19 19:23:54 2019 From: v@REDACTED (Valentin Micic) Date: Sat, 19 Oct 2019 19:23:54 +0200 Subject: [erlang-questions] Binary pattern matching In-Reply-To: References: Message-ID: <87910FD6-B21D-4240-BBF3-9C14A334D0F8@micic.co.za> Indeed, Peti, you are correct? Thanks for the reminder. In fact, this makes a perfect sense :-) Kind regards V/ > On 19 Oct 2019, at 19:00, Peti G?m?ri wrote: > > hi Valentin, > > Maybe you remember the case when Offset was bound within the binary? > <> > > On Sat, Oct 19, 2019 at 8:39 AM Valentin Micic > wrote: > My sincere apology if this has been discussed before? > I vaguely remember that a number of years ago, one was able to write something like this: > > some_function( Offset, <<_:Offset/binary-unit:8, Value/binary>> ) -> ? > > without (Erlang R21.1) compiler complaining with: variable Offset unbound > > Was I dreaming? > > Thanks in advance > > V/ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Sat Oct 19 19:51:31 2019 From: ingela.andin@REDACTED (Ingela Andin) Date: Sat, 19 Oct 2019 19:51:31 +0200 Subject: [erlang-questions] SSL Out of Order Cert Chain Question (9.2) In-Reply-To: References: Message-ID: Hi! "Unknown CA" means that you did not have the ROOT certificate of the chian in your "trusted store" (cacerts option). If you do not own the ROOT certificate you can not trust the chain. Regards Ingela Erlang/OTP Team - Ericsson AB Den fre 18 okt. 2019 kl 21:52 skrev Curtis J Schofield : > Dear Erlang Questions: > > > SSL 9.0.2 mentions a patch to fix out of order cert chains > > In SSL 9.2 we have a root CA and an out of order cert chain > for host hooks.glip.com. > > When we try to verify peer with the out of order cert > chain we get 'Unknown CA'. > > Is this expected behaviour for Erlang SSL 9.2 with verify_peer ? > > The http://erlang.org/doc/apps/ssl/notes.html#ssl-9.0.2 notes > mention that other care may need to be taken to ensure compatibility. > > Reproduce error: > > https://github.com/robotarmy/out-of-order-ssl > > Thank you, > Curtis and Team DevEco > > > > > Sent through ProtonMail Encrypted Email Channel. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From curtis@REDACTED Sun Oct 20 01:34:49 2019 From: curtis@REDACTED (Curtis J Schofield) Date: Sat, 19 Oct 2019 23:34:49 +0000 Subject: [erlang-questions] SSL Out of Order Cert Chain Question (9.2) In-Reply-To: References: Message-ID: <9ZVU-GXPquw0qkkxp1hrLF5qWBG0x8GWKm-ytdhfqeeQIl4dqt_pheziUaWjsXsQBcAhlWFRkur6-qAGXkLkyUgTMU7Y3QL5afob0aN_9Pg=@ram9.cc> Hi! Thank you. I included the root cert in the example. The root cert is id1 in cert chain - this is evident in the other file. It seems because the root cert is out of order - the cert chain is invalid - IIRC this may be true for tls1.2 - however the negotiation is at TLS1.2 Thank you for your consideration! Sent from ProtonMail Mobile On Sat, Oct 19, 2019 at 10:51 AM, Ingela Andin wrote: > Hi! > > "Unknown CA" means that you did not have the ROOT certificate of the chian in your "trusted store" (cacerts option). > If you do not own the ROOT certificate you can not trust the chain. > > Regards Ingela Erlang/OTP Team - Ericsson AB > > Den fre 18 okt. 2019 kl 21:52 skrev Curtis J Schofield : > >> Dear Erlang Questions: >> >> SSL 9.0.2 mentions a patch to fix out of order cert chains >> >> In SSL 9.2 we have a root CA and an out of order cert chain >> for host hooks.glip.com. >> >> When we try to verify peer with the out of order cert >> chain we get 'Unknown CA'. >> >> Is this expected behaviour for Erlang SSL 9.2 with verify_peer ? >> >> The http://erlang.org/doc/apps/ssl/notes.html#ssl-9.0.2 notes >> mention that other care may need to be taken to ensure compatibility. >> >> Reproduce error: >> >> https://github.com/robotarmy/out-of-order-ssl >> >> Thank you, >> Curtis and Team DevEco >> >> Sent through ProtonMail Encrypted Email Channel. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From klg.amit@REDACTED Mon Oct 21 22:25:43 2019 From: klg.amit@REDACTED (Amit K) Date: Mon, 21 Oct 2019 23:25:43 +0300 Subject: [erlang-questions] Supporting a port number in spawn/4 Message-ID: Hi all, I am very new to Erlang, am considering to use it in a project and I have some security concerns. I can see it's quite easy to configure TLS for the node-to-node communication, but making the name-to-port resolution service (epmd) secure seem a bit too complex to me, such as the one suggested here: https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd.html So I was thinking, seeing that there are already options to: 1. Start a distributed node without epmd (-start_epmd false) 2. Limit a node's port numbers to a specific range (via inet_dist_listen_min & inet_dist_listen_max). Wouldn't it be nice if we could also specify a predefined port to spawn/4, to complete that picture? That is allow spawn to look like: spawn("Name@REDACTED:Port", Mod, Func, ArgList). Then when spawn sees that a port was provided, it can completely skip the "epmd resolution" part and proceed with connecting to the target node via the provided port. Note: I realize that the "Name" becomes slightly redundant when the Port is explicit. However this can still be useful - it would be good if the implementation will also verify that the port belongs to the provided name at the receiving side, so that a node will not accidentally process a message that wasn't meant for it. Again, I'm a complete newbie to Erlang in general, so I may be missing something essential here :) But I would love to know what that is, if that's the case, or hear your thoughts in general otherwise :) Thanks! Amit -------------- next part -------------- An HTML attachment was scrubbed... URL: From list1@REDACTED Tue Oct 22 09:54:13 2019 From: list1@REDACTED (Grzegorz Junka) Date: Tue, 22 Oct 2019 08:54:13 +0100 Subject: [erlang-questions] Supporting a port number in spawn/4 In-Reply-To: References: Message-ID: <3c29721c-5936-c854-92b2-f76fb41b5d19@gjunka.com> On 21/10/2019 21:25, Amit K wrote: > Hi all, > > I am very new to Erlang, am considering to use it in a project and I > have some security concerns. > I can see it's quite easy to configure TLS for the node-to-node > communication, but making the name-to-port resolution service (epmd) > secure seem a bit too complex to me, such as the one suggested here: > https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd.html > > So I was thinking, seeing that there are already options to: > 1. Start a distributed node without epmd (-start_epmd false) > 2. Limit a node's port numbers to a specific range (via > inet_dist_listen_min &inet_dist_listen_max). > > Wouldn't it be nice if we could also specify a predefined port to > spawn/4, to complete that picture? That is allow spawn to look like: > spawn("Name@REDACTED:Port", Mod, Func, ArgList). > Then when spawn sees that a port was provided, it can completely skip > the "epmd resolution" part and proceed with connecting to the target > node via the provided port. > Note: I realize that the "Name" becomes slightly redundant when the > Port is explicit. However this can still be useful - it would be good > if the implementation will also verify that the port belongs to the > provided name at the receiving side, so that a node will not > accidentally process a message that wasn't meant for it. > > Again, I'm a complete newbie to Erlang in general, so I may be missing > something essential here :) But I would love to know what that is, if > that's the case, or hear your thoughts in general otherwise :) > Hi Amit, There is also another option, run any communication between nodes via IP tunnels . There are some tools to automate that . They are mostly used between docker containers or pods but it's just a detail, equally well they can support a microarchitecture build on Erlang nodes. Regards Greg -------------- next part -------------- An HTML attachment was scrubbed... URL: From buday.gergely@REDACTED Tue Oct 22 09:51:23 2019 From: buday.gergely@REDACTED (Gergely Buday) Date: Tue, 22 Oct 2019 09:51:23 +0200 Subject: [erlang-questions] Observability Message-ID: <013601d588ad$81c37dd0$854a7970$@uni-eszterhazy.hu> Hi, I came across a white paper on observability by Wavefront people, a marketing material: "Guide to Finding and Resolving Microservices Bottlenecks with Modern Observability" I was curious whether there are tools for observability in Erlang. I have found this page: https://erlef.org/wg/observability and this empty github repository: https://github.com/erlef/eef-observability-wg What is the state of the art for observability tools for Erlang / BEAM languages? - Gergely From klg.amit@REDACTED Tue Oct 22 10:53:42 2019 From: klg.amit@REDACTED (Amit K) Date: Tue, 22 Oct 2019 11:53:42 +0300 Subject: [erlang-questions] Supporting a port number in spawn/4 In-Reply-To: <3c29721c-5936-c854-92b2-f76fb41b5d19@gjunka.com> References: <3c29721c-5936-c854-92b2-f76fb41b5d19@gjunka.com> Message-ID: Hi Greg, Thank you for your feedback! In general, I think it makes sense to have a solution for this that's built in to OTP and not have to use an external proxy method. Going more along the route of what you suggest, a sensible alternative would be for epmd.exe to be able to accept TLS settings like the Erlang node (erl.exe) does, and then communication with it will also be protected by TLS. a quick look at the code makes me guess that the reason that was not done originally is that epmd is a small & separate utility written in C and therefore it doesn't have the "crypto" framework like the other OTP parts do. Regards, Amit On Tue, Oct 22, 2019 at 10:54 AM Grzegorz Junka wrote: > > On 21/10/2019 21:25, Amit K wrote: > > Hi all, > > I am very new to Erlang, am considering to use it in a project and I have > some security concerns. > I can see it's quite easy to configure TLS for the node-to-node > communication, but making the name-to-port resolution service (epmd) secure > seem a bit too complex to me, such as the one suggested here: > https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd.html > > So I was thinking, seeing that there are already options to: > 1. Start a distributed node without epmd (-start_epmd false) > 2. Limit a node's port numbers to a specific range (via inet_dist_listen_min > & inet_dist_listen_max). > > Wouldn't it be nice if we could also specify a predefined port to spawn/4, > to complete that picture? That is allow spawn to look like: > spawn("Name@REDACTED:Port", Mod, Func, ArgList). > Then when spawn sees that a port was provided, it can completely skip the > "epmd resolution" part and proceed with connecting to the target node via > the provided port. > Note: I realize that the "Name" becomes slightly redundant when the Port > is explicit. However this can still be useful - it would be good if the > implementation will also verify that the port belongs to the provided name > at the receiving side, so that a node will not accidentally process a > message that wasn't meant for it. > > Again, I'm a complete newbie to Erlang in general, so I may be missing > something essential here :) But I would love to know what that is, if > that's the case, or hear your thoughts in general otherwise :) > > > Hi Amit, > > There is also another option, run any communication between nodes via IP > tunnels . There are some tools > to automate that > . > They are mostly used between docker containers or pods but it's just a > detail, equally well they can support a microarchitecture build on Erlang > nodes. > > Regards > Greg > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Tue Oct 22 13:49:40 2019 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 22 Oct 2019 13:49:40 +0200 Subject: [erlang-questions] Supporting a port number in spawn/4 In-Reply-To: References: <3c29721c-5936-c854-92b2-f76fb41b5d19@gjunka.com> Message-ID: On Tue, Oct 22, 2019 at 1:02 PM Amit K wrote: > Hi Greg, > > Thank you for your feedback! > In general, I think it makes sense to have a solution for this that's > built in to OTP and not have to use an external proxy method. > Going more along the route of what you suggest, a sensible alternative > would be for epmd.exe to be able to accept TLS settings like the Erlang > node (erl.exe) does, and then communication with it will also be protected > by TLS. a quick look at the code makes me guess that the reason that was > not done originally is that epmd is a small & separate utility written in C > and therefore it doesn't have the "crypto" framework like the other OTP > parts do. > I'm kind of curious about what type of attacks you are trying to protect against? You mention "man in the middle", however, I cannot see how someone could exploit that to do anything but disrupt the distributed erlang connections. Though that could only be a lack of imagination on my part, and when it comes to security we must be extra careful. Lukas > > Regards, > Amit > > > > On Tue, Oct 22, 2019 at 10:54 AM Grzegorz Junka wrote: > >> >> On 21/10/2019 21:25, Amit K wrote: >> >> Hi all, >> >> I am very new to Erlang, am considering to use it in a project and I have >> some security concerns. >> I can see it's quite easy to configure TLS for the node-to-node >> communication, but making the name-to-port resolution service (epmd) secure >> seem a bit too complex to me, such as the one suggested here: >> https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd.html >> >> So I was thinking, seeing that there are already options to: >> 1. Start a distributed node without epmd (-start_epmd false) >> 2. Limit a node's port numbers to a specific range (via inet_dist_listen_min >> & inet_dist_listen_max). >> >> Wouldn't it be nice if we could also specify a predefined port to >> spawn/4, to complete that picture? That is allow spawn to look like: >> spawn("Name@REDACTED:Port", Mod, Func, ArgList). >> Then when spawn sees that a port was provided, it can completely skip the >> "epmd resolution" part and proceed with connecting to the target node via >> the provided port. >> Note: I realize that the "Name" becomes slightly redundant when the Port >> is explicit. However this can still be useful - it would be good if the >> implementation will also verify that the port belongs to the provided name >> at the receiving side, so that a node will not accidentally process a >> message that wasn't meant for it. >> >> Again, I'm a complete newbie to Erlang in general, so I may be missing >> something essential here :) But I would love to know what that is, if >> that's the case, or hear your thoughts in general otherwise :) >> >> >> Hi Amit, >> >> There is also another option, run any communication between nodes via IP >> tunnels . There are some tools >> to automate that >> . >> They are mostly used between docker containers or pods but it's just a >> detail, equally well they can support a microarchitecture build on Erlang >> nodes. >> >> Regards >> Greg >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From klg.amit@REDACTED Tue Oct 22 14:10:50 2019 From: klg.amit@REDACTED (Amit K) Date: Tue, 22 Oct 2019 15:10:50 +0300 Subject: [erlang-questions] Supporting a port number in spawn/4 In-Reply-To: References: <3c29721c-5936-c854-92b2-f76fb41b5d19@gjunka.com> Message-ID: Hi Lukas, It's a great question! And actually, choosing between disabling epmd completely and protecting communication via TLS, will have somewhat different security benefits. 1. *Protecting epmd communication via TLS* - This only protects us from an attacker that wants to perform epmd impersonation, because TLS here is authenticating only the server node to the client node. Then perhaps if we have two Erlang nodes on the same machine, let's say Node1 and Node2, an attacker can make the client node connect to Node2 instead of Node1 by returning the wrong port by a simple manipulation of the plain epmd traffic, and in that way expose sensitive data to the client node, which exists in Node2 but not in Node1. Maybe it's a bit far fetched but from my limited knowledge it looks like a possible attack scenario. Another thing that the encryption is good for here is probably in preventing a covert channel. 2. *Disabling epmd completely* - This approach eliminates a lot more attack vectors actually. Just as one simple example, assume for a moment that we have a buffer overflow bug in epmd, then using this approach you will not be affected by it. By completely disabling an unnecessary service (if it's really unnecessary of course, in some situations I assume epmd is very much required!) the attack surface reduces much more so it's difficult to list in detail everything we'll be protecting against. Nevertheless, we all know that it's a good security practice to disable unrequired services whenever you can :) and especially ones that listen to requests from the outside world. Regards, Amit On Tue, Oct 22, 2019 at 2:49 PM Lukas Larsson wrote: > > > On Tue, Oct 22, 2019 at 1:02 PM Amit K wrote: > >> Hi Greg, >> >> Thank you for your feedback! >> In general, I think it makes sense to have a solution for this that's >> built in to OTP and not have to use an external proxy method. >> Going more along the route of what you suggest, a sensible alternative >> would be for epmd.exe to be able to accept TLS settings like the Erlang >> node (erl.exe) does, and then communication with it will also be protected >> by TLS. a quick look at the code makes me guess that the reason that was >> not done originally is that epmd is a small & separate utility written in C >> and therefore it doesn't have the "crypto" framework like the other OTP >> parts do. >> > > I'm kind of curious about what type of attacks you are trying to protect > against? You mention "man in the middle", however, I cannot see how someone > could exploit that to do anything but disrupt the distributed erlang > connections. Though that could only be a lack of imagination on my part, > and when it comes to security we must be extra careful. > > Lukas > > >> >> Regards, >> Amit >> >> >> >> On Tue, Oct 22, 2019 at 10:54 AM Grzegorz Junka wrote: >> >>> >>> On 21/10/2019 21:25, Amit K wrote: >>> >>> Hi all, >>> >>> I am very new to Erlang, am considering to use it in a project and I >>> have some security concerns. >>> I can see it's quite easy to configure TLS for the node-to-node >>> communication, but making the name-to-port resolution service (epmd) secure >>> seem a bit too complex to me, such as the one suggested here: >>> https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd.html >>> >>> So I was thinking, seeing that there are already options to: >>> 1. Start a distributed node without epmd (-start_epmd false) >>> 2. Limit a node's port numbers to a specific range (via inet_dist_listen_min >>> & inet_dist_listen_max). >>> >>> Wouldn't it be nice if we could also specify a predefined port to >>> spawn/4, to complete that picture? That is allow spawn to look like: >>> spawn("Name@REDACTED:Port", Mod, Func, ArgList). >>> Then when spawn sees that a port was provided, it can completely skip >>> the "epmd resolution" part and proceed with connecting to the target node >>> via the provided port. >>> Note: I realize that the "Name" becomes slightly redundant when the Port >>> is explicit. However this can still be useful - it would be good if the >>> implementation will also verify that the port belongs to the provided name >>> at the receiving side, so that a node will not accidentally process a >>> message that wasn't meant for it. >>> >>> Again, I'm a complete newbie to Erlang in general, so I may be missing >>> something essential here :) But I would love to know what that is, if >>> that's the case, or hear your thoughts in general otherwise :) >>> >>> >>> Hi Amit, >>> >>> There is also another option, run any communication between nodes via IP >>> tunnels . There are some tools >>> to automate that >>> . >>> They are mostly used between docker containers or pods but it's just a >>> detail, equally well they can support a microarchitecture build on Erlang >>> nodes. >>> >>> Regards >>> Greg >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oltarasenko@REDACTED Tue Oct 22 14:24:44 2019 From: oltarasenko@REDACTED (Oleg Tarasenko) Date: Tue, 22 Oct 2019 14:24:44 +0200 Subject: [erlang-questions] Supporting a port number in spawn/4 In-Reply-To: References: <3c29721c-5936-c854-92b2-f76fb41b5d19@gjunka.com> Message-ID: Hey Amit, Actually I am maintaining a small project which is dedicated to the topic above: https://github.com/oltarasenko/epmdless See also the article regarding running it in docker: https://www.erlang-solutions.com/blog/running-distributed-erlang-elixir-applications-on-docker.html Hopefully it could be interesting for you as well, Oleg On Tue, Oct 22, 2019 at 2:11 PM Amit K wrote: > Hi Lukas, > > It's a great question! > And actually, choosing between disabling epmd completely and protecting > communication via TLS, will have somewhat different security benefits. > > 1. *Protecting epmd communication via TLS* - This only protects us from > an attacker that wants to perform epmd impersonation, because TLS here is > authenticating only the server node to the client node. > Then perhaps if we have two Erlang nodes on the same machine, let's say > Node1 and Node2, an attacker can make the client node connect to Node2 > instead of Node1 by returning the wrong port by a simple manipulation of > the plain epmd traffic, and in that way expose sensitive data to the client > node, which exists in Node2 but not in Node1. Maybe it's a bit far fetched > but from my limited knowledge it looks like a possible attack scenario. > Another thing that the encryption is good for here is probably in > preventing a covert channel. > > 2. *Disabling epmd completely* - This approach eliminates a lot more > attack vectors actually. > Just as one simple example, assume for a moment that we have a buffer > overflow bug in epmd, then using this approach you will not be affected by > it. By completely disabling an unnecessary service (if it's really > unnecessary of course, in some situations I assume epmd is very much > required!) the attack surface reduces much more so it's difficult to list > in detail everything we'll be protecting against. > Nevertheless, we all know that it's a good security practice to disable > unrequired services whenever you can :) and especially ones that listen to > requests from the outside world. > > Regards, > Amit > > On Tue, Oct 22, 2019 at 2:49 PM Lukas Larsson wrote: > >> >> >> On Tue, Oct 22, 2019 at 1:02 PM Amit K wrote: >> >>> Hi Greg, >>> >>> Thank you for your feedback! >>> In general, I think it makes sense to have a solution for this that's >>> built in to OTP and not have to use an external proxy method. >>> Going more along the route of what you suggest, a sensible alternative >>> would be for epmd.exe to be able to accept TLS settings like the Erlang >>> node (erl.exe) does, and then communication with it will also be protected >>> by TLS. a quick look at the code makes me guess that the reason that was >>> not done originally is that epmd is a small & separate utility written in C >>> and therefore it doesn't have the "crypto" framework like the other OTP >>> parts do. >>> >> >> I'm kind of curious about what type of attacks you are trying to protect >> against? You mention "man in the middle", however, I cannot see how someone >> could exploit that to do anything but disrupt the distributed erlang >> connections. Though that could only be a lack of imagination on my part, >> and when it comes to security we must be extra careful. >> >> Lukas >> >> >>> >>> Regards, >>> Amit >>> >>> >>> >>> On Tue, Oct 22, 2019 at 10:54 AM Grzegorz Junka >>> wrote: >>> >>>> >>>> On 21/10/2019 21:25, Amit K wrote: >>>> >>>> Hi all, >>>> >>>> I am very new to Erlang, am considering to use it in a project and I >>>> have some security concerns. >>>> I can see it's quite easy to configure TLS for the node-to-node >>>> communication, but making the name-to-port resolution service (epmd) secure >>>> seem a bit too complex to me, such as the one suggested here: >>>> https://www.erlang-solutions.com/blog/erlang-and-elixir-distribution-without-epmd.html >>>> >>>> So I was thinking, seeing that there are already options to: >>>> 1. Start a distributed node without epmd (-start_epmd false) >>>> 2. Limit a node's port numbers to a specific range (via inet_dist_listen_min >>>> & inet_dist_listen_max). >>>> >>>> Wouldn't it be nice if we could also specify a predefined port to >>>> spawn/4, to complete that picture? That is allow spawn to look like: >>>> spawn("Name@REDACTED:Port", Mod, Func, ArgList). >>>> Then when spawn sees that a port was provided, it can completely skip >>>> the "epmd resolution" part and proceed with connecting to the target node >>>> via the provided port. >>>> Note: I realize that the "Name" becomes slightly redundant when the >>>> Port is explicit. However this can still be useful - it would be good if >>>> the implementation will also verify that the port belongs to the provided >>>> name at the receiving side, so that a node will not accidentally process a >>>> message that wasn't meant for it. >>>> >>>> Again, I'm a complete newbie to Erlang in general, so I may be missing >>>> something essential here :) But I would love to know what that is, if >>>> that's the case, or hear your thoughts in general otherwise :) >>>> >>>> >>>> Hi Amit, >>>> >>>> There is also another option, run any communication between nodes via IP >>>> tunnels . There are some tools >>>> to automate that >>>> . >>>> They are mostly used between docker containers or pods but it's just a >>>> detail, equally well they can support a microarchitecture build on Erlang >>>> nodes. >>>> >>>> Regards >>>> Greg >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.r.nilsson@REDACTED Tue Oct 22 15:57:09 2019 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Tue, 22 Oct 2019 13:57:09 +0000 Subject: [erlang-questions] Patch Package OTP 22.1.4 Released Message-ID: Patch Package: OTP 22.1.4 Git Tag: OTP-22.1.4 Date: 2019-10-22 Trouble Report Id: OTP-16177, OTP-16199, OTP-16202 Seq num: ERL-1063 System: OTP Release: 22 Application: compiler-7.4.7, crypto-4.6.1, erts-10.5.3 Predecessor: OTP 22.1.3 Check out the git tag OTP-22.1.4, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- compiler-7.4.7 -------------------------------------------------- --------------------------------------------------------------------- The compiler-7.4.7 application can be applied independently of other applications on a full OTP 22 installation. --- Fixed Bugs and Malfunctions --- OTP-16199 Application(s): compiler Fixed a bug where the compiler could generate incorrect code for a 'receive' statement inside a 'try'. Full runtime dependencies of compiler-7.4.7: crypto-3.6, erts-9.0, hipe-3.12, kernel-4.0, stdlib-2.5 --------------------------------------------------------------------- --- crypto-4.6.1 ---------------------------------------------------- --------------------------------------------------------------------- The crypto-4.6.1 application can be applied independently of other applications on a full OTP 22 installation. --- Fixed Bugs and Malfunctions --- OTP-16202 Application(s): crypto FIxed a bug if the erlang emulator was linked with a very old cryptolib version (1.0.1 or earlier). The bug now fixed could have triggered a core dump if an unknown cipher name was used in crypto functions. Full runtime dependencies of crypto-4.6.1: erts-9.0, kernel-5.3, stdlib-3.4 --------------------------------------------------------------------- --- erts-10.5.3 ----------------------------------------------------- --------------------------------------------------------------------- The erts-10.5.3 application can be applied independently of other applications on a full OTP 22 installation. --- Fixed Bugs and Malfunctions --- OTP-16177 Application(s): erts Related Id(s): ERL-1063 Erlang/OTP can now be built on macOS Catalina (10.15). Full runtime dependencies of erts-10.5.3: kernel-6.1, sasl-3.3, stdlib-3.5 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From areege.alzubaidi@REDACTED Tue Oct 22 19:31:26 2019 From: areege.alzubaidi@REDACTED (Areege Al Zubaidi) Date: Tue, 22 Oct 2019 18:31:26 +0100 Subject: [erlang-questions] Call for talks - Code BEAM SF 2020 Message-ID: Hi all, The deadline for the call for talks at Code BEAM SF 2020 is looming! If you're interested in sharing your experience of working with Erlang or Elixir at Code BEAM SF, submit your talk before midnight tomorrow (23 October). We're excited to have expanded the themes of Code BEAM SF 2020 to include a new category: Web and APIs. The 2020 themes are: - The BEAM - Scalability & reliability - Case studies - Tools - Introduction to Erlang and Elixir - Web and APIs - Frameworks Code BEAM SF is the biggest and longest-running conference in North America to cover all of the BEAM languages, including Erlang and Elixir. Created for developers, by developers, Code BEAM SF is dedicated to bringing the best minds in the Erlang and Elixir communities together to SHARE. LEARN. INSPIRE. over two days. What you need to know about Code BEAM SF 2020 : - When - 05-06 March 2020 - Where - Hyatt Centric Fisherman's Wharf, San Francisco, CA - Programme committee - Miriam Pena, Fred Hebert, Anna Neyzberg, Francesco Cesarini, Bryan Hunt Our programme committee looks forward to reviewing your talk submission! See you in San Francisco, Areege -- *Areege Al Zubaidi* *Conferences Marketing Executive* *she/her* T: +44 (0) 2076550332 W: www.codesync.global A: Erlang Solutions Ltd, The Loom,14 Gower's Walk, London, E1 8PY -- Code Sync & Erlang Solutions Conferences Code BEAM Lite - Berlin: 11 October 2019 RabbitMQ Summit - London: 4 November 2019 Code Mesh LDN - London: 7-8 November 2019 Code BEAM Lite - Bangalore: 14 November 2019 Code BEAM Lite - Amsterdam: 28 November 2019 Lambda Days - Krak?w: 13-14 February 2020 Code BEAM SF - San Francisco: 5-6 March 2020 Code BEAM STO - Stockholm: 28-29 May 2020 Erlang Solutions cares about your data and privacy; please find all details about the basis for communicating with you and the way we process your data in our Privacy Policy . You can update your email preferences or opt-out from receiving Marketing emails here . -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.schultz@REDACTED Wed Oct 23 15:53:39 2019 From: andreas.schultz@REDACTED (Andreas Schultz) Date: Wed, 23 Oct 2019 15:53:39 +0200 Subject: [erlang-questions] OTP 22.1 socket.erl somehow breaks message delivery or scheduler Message-ID: Hi, After converting an application to socket.erl in OTP 22.1, the test suites started to fail with random timeouts. It took me a while to figure out that gen_server:calls arrived multiple seconds late in the server. I have a demonstration at https://gist.github.com/RoadRunnr/311a7679fff6fbdf367c455b960f1ba8. It implements a simple UDP echo server with socket.erl. The client uses gen_udp to send messages and wait for the response. The client also sends Erlang ping message to the server and expects to get a pong answer back. The socket.erl based server is supposed to not block (and as far as I can tell, it does not), it therefore should be able to answer the Erlang ping message all the time. There are also some simple busy loop process running to get some load. Without them the problem is not reproducible. The sample is failing in about 20% off the test runs, when it does the output is something like: $ ~/stest.escript Server Pid <0.78.0> Server Addr #{addr => {127,0,0,1},family => inet,port => 38959} ping timeout round trip Clnt/Srvr Srvr/Clnt ProcPing ... 85: ******** ns, ******** ns, 57675 ns, 42332 ns ... The failure happens because a 'ping' message is not see in time by the receive clause in the server process. It seems that either the process is not scheduled for some time (multiple seconds), or the scanning of the mailbox is missing the message. I have ruled out that the UDP messages are being dropped, otherwise the clients gen_udp:recv would never return. Does anyone have a clue what might cause this? Or point out where my sample is broken. Many thanks Andreas -- Andreas Schultz -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Wed Oct 23 16:14:55 2019 From: t@REDACTED (Tristan Sloughter) Date: Wed, 23 Oct 2019 08:14:55 -0600 Subject: [erlang-questions] Observability In-Reply-To: <013601d588ad$81c37dd0$854a7970$@uni-eszterhazy.hu> References: <013601d588ad$81c37dd0$854a7970$@uni-eszterhazy.hu> Message-ID: <745edebd-12c7-4c83-81c2-2759c520b1f1@www.fastmail.com> Hey Gergely, we will be working on documentation about this soon. But I'll quickly call out a couple projects, specifically for production observability, there are many others for tracing and profiling when it comes to developing and benchmarking: The telemetry is a project to have a shared library between all beam languages for instrumenting your code. Popular Elixir libraries have already begun including emitting telemetry events that can be handled and exported to multiple stats collectors (prometheus, statsd, etc) : https://github.com/beam-telemetry/telemetry In the same project as telemetry is the poller https://github.com/beam-telemetry/telemetry_poller which provides VM statistics in a way we can all benefit and share a common implementation. Then, there is the newly formed OpenTelemtry, https://opentelemetry.io, which is a successor to OpenTracing and OpenCensus which combines them into a single spec and implementation and is being worked on collectively by the people from both those former projects. OpenTelemetry (confusing name because the telemetry project was named first :) is a CNCF project is creating a common spec and protocol for recording metrics and distributed traces -- esp important for when you have multiple languages in your environment and want to trace across them which the W3C Trace Context spec is handling https://www.w3.org/TR/trace-context/ and is the default propagation mechanism for OpenTelemetry. We still need to update the website for the status of the Erlang implementation, https://github.com/open-telemetry/opentelemetry-erlang, which we hope to have in an alpha or beta state by mid-November. The plan is also for a telemetry handler which will take telemetry events and convert to OpenTelemetry spans and metrics. OpenTelemetry will then handle tracking the stats and spans and exporting to various services, which if the Erlang client doesn't natively support a certain service it will be easy enough to still send metrics/spans to them through the OpenTelemetry Collector https://github.com/open-telemetry/opentelemetry-collector/ Hope that helps some! It is certainly not the whole story for what exists for beam observability but it is what those of us in the WG are mainly focusing on at this time. Tristan On Tue, Oct 22, 2019, at 01:51, Gergely Buday wrote: > Hi, > > I came across a white paper on observability by Wavefront people, a > marketing material: > > "Guide to Finding and Resolving Microservices Bottlenecks with Modern > Observability" > > I was curious whether there are tools for observability in Erlang. I have > found this page: > > https://erlef.org/wg/observability > > and this empty github repository: > > https://github.com/erlef/eef-observability-wg > > What is the state of the art for observability tools for Erlang / BEAM > languages? > > - Gergely > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From tty.erlang@REDACTED Thu Oct 24 12:49:31 2019 From: tty.erlang@REDACTED (T Ty) Date: Thu, 24 Oct 2019 11:49:31 +0100 Subject: [erlang-questions] Efficient means of deleting Mnesia entries Message-ID: Good day, I have a Mnesia table which is fragmented. I created an index on an attribute and can efficiently select entries based on my criteria using match specs and mnesia:select. With these keys what is an efficient means of deleting this rather large Mnesia entries ? It is possible I might have 500,000 to 1.5M entries to delete. Using async_dirty I'm averaging 87s for 500,000 entries on SunOS 5.11 8GB 2 Core, OTP R17.3. Thanks, tty -------------- next part -------------- An HTML attachment was scrubbed... URL: From boyofmonk@REDACTED Thu Oct 24 13:23:35 2019 From: boyofmonk@REDACTED (Monk Boy) Date: Thu, 24 Oct 2019 19:23:35 +0800 Subject: [erlang-questions] cerl: insert-string Message-ID: Hi all: I have used cerl to debug beam(./bin/cerl -gdb), Run the debug compiled emulator in emacs and gdb. When I input ``` For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from beam.smp... (gdb) r Starting program: /home/memacs/workspace/github/otp/bin/x86_64-unknown-linux-gnu/beam.smp [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7fffb5e7f700 (LWP 8112)] [New Thread 0x7fffb40bf700 (LWP 8113)] [New Thread 0x7fffb5ef2700 (LWP 8114)] [Detaching after fork from child process 8115] [New Thread 0x7fffb36bf700 (LWP 8116)] [New Thread 0x7fffb35bc700 (LWP 8117)] [New Thread 0x7fffb34b9700 (LWP 8118)] [New Thread 0x7fffb317f700 (LWP 8119)] [New Thread 0x7fffb307c700 (LWP 8120)] [New Thread 0x7fffb2f79700 (LWP 8121)] [New Thread 0x7fffb2e76700 (LWP 8122)] [New Thread 0x7fffb2d73700 (LWP 8123)] [New Thread 0x7fffb2c70700 (LWP 8124)] [New Thread 0x7fffb2c1d700 (LWP 8125)] [New Thread 0x7fffb2bca700 (LWP 8126)] [New Thread 0x7fffb2b77700 (LWP 8127)] [New Thread 0x7fffb2b24700 (LWP 8128)] [New Thread 0x7fffb2ad1700 (LWP 8129)] [New Thread 0x7fffb2a7e700 (LWP 8130)] [New Thread 0x7fffb2a2b700 (LWP 8131)] [New Thread 0x7fffb29d8700 (LWP 8132)] [New Thread 0x7fffb2985700 (LWP 8133)] [New Thread 0x7fffb2932700 (LWP 8134)] [New Thread 0x7fffb28df700 (LWP 8135)] [New Thread 0x7fffb288c700 (LWP 8136)] [New Thread 0x7fffb2839700 (LWP 8137)] [New Thread 0x7fffb27e6700 (LWP 8138)] [New Thread 0x7fffb2793700 (LWP 8139)] [New Thread 0x7fffb2740700 (LWP 8140)] [New Thread 0x7fffb26ed700 (LWP 8141)] [New Thread 0x7fffb269a700 (LWP 8142)] [New Thread 0x7fffb2647700 (LWP 8143)] Thread 11 "7_scheduler" received signal SIG35, Real-time event 35. [Switching to Thread 0x7fffb2e76700 (LWP 8122)] 0x00007ffff7d14e3f in epoll_wait (epfd=8, events=events@REDACTED=0x7fffb72c8b60, maxevents=maxevents@REDACTED=512, timeout=timeout@REDACTED=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 30 ../sysdeps/unix/sysv/linux/epoll_wait.c: ?????????. (gdb) ``` I got: ``` {"init terminating in do_boot",no_or_multiple_root_variables} init terminating in do_boot (no_or_multiple_root_variables) Crash dump is being written to: erl_crash.dump... ``` I must input: (gdb) r -- -root /home/memacs/workspace/github/otp -progname /home/memacs/workspace/github/otp/bin/cerl -- -home /home/memacs -- Starting program: /home/memacs/workspace/github/otp/bin/x86_64-unknown-linux-gnu/beam.smp -- -root /home/memacs/workspace/github/otp -progname /home/memacs/workspace/github/otp/bin/cerl -- -home /home/memacs -- [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7fffb5e7f700 (LWP 8508)] [New Thread 0x7fffb40bf700 (LWP 8509)] [New Thread 0x7fffb5ef2700 (LWP 8510)] [Detaching after fork from child process 8511] [New Thread 0x7fffb36bf700 (LWP 8512)] [New Thread 0x7fffb35bc700 (LWP 8513)] [New Thread 0x7fffb34b9700 (LWP 8514)] [New Thread 0x7fffb31b6700 (LWP 8515)] [New Thread 0x7fffb33b6700 (LWP 8516)] [New Thread 0x7fffb2e7f700 (LWP 8517)] [New Thread 0x7fffb2d7c700 (LWP 8518)] [New Thread 0x7fffb2c79700 (LWP 8519)] [New Thread 0x7fffb32b3700 (LWP 8520)] [New Thread 0x7fffb3260700 (LWP 8521)] [New Thread 0x7fffb320d700 (LWP 8522)] [New Thread 0x7fffb2b76700 (LWP 8523)] [New Thread 0x7fffb2b23700 (LWP 8524)] [New Thread 0x7fffb2ad0700 (LWP 8525)] [New Thread 0x7fffb2a7d700 (LWP 8526)] [New Thread 0x7fffb2a2a700 (LWP 8527)] [New Thread 0x7fffb29d7700 (LWP 8528)] [New Thread 0x7fffb2984700 (LWP 8529)] [New Thread 0x7fffb2931700 (LWP 8530)] [New Thread 0x7fffb28de700 (LWP 8531)] [New Thread 0x7fffb288b700 (LWP 8532)] [New Thread 0x7fffb2838700 (LWP 8533)] [New Thread 0x7fffb27e5700 (LWP 8534)] [New Thread 0x7fffb2792700 (LWP 8535)] [New Thread 0x7fffb273f700 (LWP 8536)] [New Thread 0x7fffb26ec700 (LWP 8537)] [New Thread 0x7fffb2699700 (LWP 8538)] [New Thread 0x7fffb2646700 (LWP 8539)] ``` Last I find the cerl script ``` gdbcmd="(insert-string \"set args $beam_args\") \ (comint-send-input)" ``` But emacs has not the function (insert-string xxx) (C-h f) -------------- next part -------------- An HTML attachment was scrubbed... URL: From thp+erlang@REDACTED Thu Oct 24 12:04:44 2019 From: thp+erlang@REDACTED (Thomas Pircher) Date: Thu, 24 Oct 2019 11:04:44 +0100 Subject: [erlang-questions] Matching IP address in socket module Message-ID: <20191024100444.GA28388@tpccl.tty1.net> Hi, I'm trying to use the new socket module, and I'm having difficulties extracting the IP address from the socket:sockaddr_in4 record, as returned from socket:recvfrom. In my particular case I'm only dealing with IPv4 addresses for now. > receive_data(Sock, State) -> > case socket:recvfrom(Sock, 0, nowait) of > {ok, {From, Data}} -> > NewState = handle_data(From, Data, State), > receive_data(Sock, NewState); > {select, _SelectInfo} -> {ok, State}; > {error, Reason} -> {error, Reason} > end. The problem is I don't really know how to match the From address. I would like to match the IP address in the above statement, something like the following pseudo-code: > case socket:recvfrom(Sock, 0, nowait) of > {ok, {#socket:sockaddr_in4{family=inet, addr=From}, Data}} -> My (limited) understanding is that I would need to include a header file to use records from another module, but as far as I can see, the socket module does not define a header file. If need be I can extract the address in the body of the case statement. How would I get the sender address from the return value of the recvfrom function? Thanks, Thomas From markalanj@REDACTED Thu Oct 24 14:02:35 2019 From: markalanj@REDACTED (Mark Jones) Date: Thu, 24 Oct 2019 07:02:35 -0500 Subject: [erlang-questions] Matching IP address in socket module In-Reply-To: <20191024100444.GA28388@tpccl.tty1.net> References: <20191024100444.GA28388@tpccl.tty1.net> Message-ID: Thomas While I have not used the socket module, the documentation indicates that the Source variable returned from the recvfrom function is a map not a record. So you would just need to use map matching syntax to extract the data you want. Mark On Thu, Oct 24, 2019 at 6:42 AM Thomas Pircher wrote: > Hi, > > I'm trying to use the new socket module, and I'm having difficulties > extracting the IP address from the socket:sockaddr_in4 record, as > returned from socket:recvfrom. > In my particular case I'm only dealing with IPv4 addresses for now. > > > receive_data(Sock, State) -> > > case socket:recvfrom(Sock, 0, nowait) of > > {ok, {From, Data}} -> > > NewState = handle_data(From, Data, State), > > receive_data(Sock, NewState); > > {select, _SelectInfo} -> {ok, State}; > > {error, Reason} -> {error, Reason} > > end. > > The problem is I don't really know how to match the From address. I would > like > to match the IP address in the above statement, something like the > following > pseudo-code: > > > case socket:recvfrom(Sock, 0, nowait) of > > {ok, {#socket:sockaddr_in4{family=inet, addr=From}, Data}} -> > > My (limited) understanding is that I would need to include a header file > to use > records from another module, but as far as I can see, the socket module > does > not define a header file. > If need be I can extract the address in the body of the case statement. > > How would I get the sender address from the return value of the recvfrom > function? > > Thanks, > Thomas > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From micael.karlberg@REDACTED Thu Oct 24 15:07:55 2019 From: micael.karlberg@REDACTED (Micael Karlberg) Date: Thu, 24 Oct 2019 15:07:55 +0200 Subject: [erlang-questions] Matching IP address in socket module In-Reply-To: References: <20191024100444.GA28388@tpccl.tty1.net> Message-ID: case socket:recvfrom(Sock, 0, nowait) of {ok, {#{family := Domain, port := Port, addr := Addr}, Data}} -> . . . /BMK On 2019-10-24 14:02, Mark Jones wrote: > ? Thomas > > While I have not used the socket module, the documentation indicates that the Source variable > returned from the recvfrom function is a map not a record. So you would just need to use map > matching syntax to extract the data you want. > > Mark > > On Thu, Oct 24, 2019 at 6:42 AM Thomas Pircher > wrote: > > Hi, > > I'm trying to use the new socket module, and I'm having difficulties > extracting the IP address from the socket:sockaddr_in4 record, as > returned from socket:recvfrom. > In my particular case I'm only dealing with IPv4 addresses for now. > > > receive_data(Sock, State) -> > >? ? ?case socket:recvfrom(Sock, 0, nowait) of > >? ? ? ? ?{ok, {From, Data}} -> > >? ? ? ? ? ? ?NewState = handle_data(From, Data, State), > >? ? ? ? ? ? ?receive_data(Sock, NewState); > >? ? ? ? ?{select, _SelectInfo} -> {ok, State}; > >? ? ? ? ?{error, Reason} -> {error, Reason} > >? ? ?end. > > The problem is I don't really know how to match the From address. I would like > to match the IP address in the above statement, something like the following > pseudo-code: > > >? ? ?case socket:recvfrom(Sock, 0, nowait) of > >? ? ? ? ?{ok, {#socket:sockaddr_in4{family=inet, addr=From}, Data}} -> > > My (limited) understanding is that I would need to include a header file to use > records from another module, but as far as I can see, the socket module does > not define a header file. > If need be I can extract the address in the body of the case statement. > > How would I get the sender address from the return value of the recvfrom function? > > Thanks, > Thomas > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > https://protect2.fireeye.com/v1/url?k=27955f28-7b1c856f-27951fb3-0cc47ad93e96-051349a7d0786a77&q=1&e=dc31b8ad-08b4-405c-b0df-ce3ba6216af4&u=http%3A%2F%2Ferlang.org%2Fmailman%2Flistinfo%2Ferlang-questions > From z@REDACTED Thu Oct 24 17:23:04 2019 From: z@REDACTED (Danil Zagoskin) Date: Thu, 24 Oct 2019 18:23:04 +0300 Subject: OTP 22.1 ssl env protocol_version causes malformed_handshake_data alert Message-ID: Hi! While upgrading to OTP 22.1 I've hit an issue with ssl. The problem: when connecting to a TLS server with a client certificate, 'ssl_handshake:select_hashsign' may fail with '{error,badarg}', causing 'Handshake Failure - malformed_handshake_data' This happens because 'SupportedHashSigns = undefined', which comes from 'ssl:handle_options' in 'ssl_connection:init'. signature_algs = handle_option(signature_algs, Opts, undefined, Role, undefined, HighestVersion), ^^^ this returns 'undefined' because 'tls_v1:default_signature_algs(HighestVersion)' returns 'undefined' And the problem here was 'HighestVersion = {3,1}' which seemed wrong. When 'versions' option is not passed, HighestVersion is the head of list returned from 'tls_record:supported_protocol_versions()', and that function just maps 'protocol_version' environment parameter. If the first version specified in environment is low, the 'HighestVersion' will contain not a highest version. So, after setting default ssl protocol versions like this: 'application:set_env(ssl, protocol_version, [tlsv1,'tlsv1.1','tlsv1.2']).' the client raises an alert while handling '#certificate_request{}' from server. OTP 21 and earlier tolerated the version order in 'protocol_version' env, and the code was like this: signature_algs = handle_hashsigns_option(..., tls_version(RecordCb:highest_protocol_version(Versions))) If the new behaviour is intended, it may be useful to describe it in the man page: http://erlang.org/doc/man/ssl_app.html If 'HighestVersion' should always contain a highest version, the obvious (but may be not the best) fix is to sort default versions in 'ssl:handle_options', like it is done for specified 'versions'. Obvious workaround for this is to store 'protocol_version' env parameter starting from highest version. -- Danil Zagoskin | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andras.Bekes@REDACTED Thu Oct 24 16:57:42 2019 From: Andras.Bekes@REDACTED (Bekes, Andras G) Date: Thu, 24 Oct 2019 14:57:42 +0000 Subject: [erlang-questions] Segfault with Erlang R22 In-Reply-To: References: <649d87fb8e0346f7ad0a039e483a46af@morganstanley.com> Message-ID: <50f54b6f7f9b4e88ae3d1e540c881ba2@morganstanley.com> ?Hi Mikael, I filed a bug report in the bug tracker: https://bugs.erlang.org/browse/ERL-1074 Unfortunately printing *c_p did not reveal anything: (gdb) print c_p $1 = (gdb) print *c_p value has been optimized out What should be the next step? I can reliably produce 5-10 core dumps per week in my test system. -----Original Message----- From: Mikael Pettersson [mailto:mikpelinux@REDACTED] Sent: Friday, October 18, 2019 8:03 PM To: Bekes, Andras G (IST) Cc: Erlang Questions Subject: Re: [erlang-questions] Segfault with Erlang R22 On Fri, Oct 18, 2019 at 4:34 PM Bekes, Andras G wrote: > > Hi All, > > > > After upgrading to Erlang R22, my software crashes the Erlang VM with Segmentation fault. > > It happens rarely, only after several days of test workload, so I can?t really reproduce. > > > > I made more than 10 core dumps so far, loaded them into gdb, and all of them died at these 2 crash points: > > > > Program terminated with signal 11, Segmentation fault. > > #0 process_main (x_reg_array=0x20002, f_reg_array=0x2ade29280590) at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:4064 > > 4064 if (is_not_tuple(r(0))) { > > > > or > > > > #0 process_main (x_reg_array=0x20002, f_reg_array=0x2) at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:5252 > > 5252 c_p->seq_trace_lastcnt = unsigned_val(SEQ_TRACE_TOKEN_SERIAL(c_p)); > > > > The software is not doing any tracing when the crash happens, nor does it have any NIFs. I think you should open a bug report in the erlang bug tracker. The second crash site above is in the remove_message() function, in a block where ERL_MESSAGE_TOKEN(msgp) is neither NIL nor am_undefined, but SEQ_TRACE_TOKEN(c_p) is invalid (not the expected 5-tuple). Maybe printing *c_p in gdb when that happens could shed some light. /Mikael From thp+erlang@REDACTED Thu Oct 24 17:34:45 2019 From: thp+erlang@REDACTED (Thomas Pircher) Date: Thu, 24 Oct 2019 16:34:45 +0100 Subject: [erlang-questions] Matching IP address in socket module In-Reply-To: References: <20191024100444.GA28388@tpccl.tty1.net> Message-ID: <20191024153351.GB28388@tpccl.tty1.net> Micael Karlberg wrote: > case socket:recvfrom(Sock, 0, nowait) of > {ok, {#{family := Domain, > port := Port, > addr := Addr}, Data}} -> > . Hi Micael, Mark, thanks for your replies. The snippet above helped me getting the record matched. I'm quite impressed with the socket module, it seems to be pretty complete, at least for my application. Thanks, Thomas From frank.muller.erl@REDACTED Thu Oct 24 18:48:10 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Thu, 24 Oct 2019 18:48:10 +0200 Subject: [erlang-questions] Matching IP address in socket module In-Reply-To: <20191024153351.GB28388@tpccl.tty1.net> References: <20191024100444.GA28388@tpccl.tty1.net> <20191024153351.GB28388@tpccl.tty1.net> Message-ID: Hi Thomas Is the socket module faster than gen_tcp/gen_udp in your case? If yes, can you please share some numbers. /Frank > case socket:recvfrom(Sock, 0, nowait) of > > {ok, {#{family := Domain, > > port := Port, > > addr := Addr}, Data}} -> > > . > > Hi Micael, Mark, > > thanks for your replies. The snippet above helped me getting the record > matched. > > I'm quite impressed with the socket module, it seems to be pretty > complete, at least for my application. > > Thanks, > Thomas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikpelinux@REDACTED Thu Oct 24 19:09:42 2019 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Thu, 24 Oct 2019 19:09:42 +0200 Subject: [erlang-questions] Segfault with Erlang R22 In-Reply-To: <50f54b6f7f9b4e88ae3d1e540c881ba2@morganstanley.com> References: <649d87fb8e0346f7ad0a039e483a46af@morganstanley.com> <50f54b6f7f9b4e88ae3d1e540c881ba2@morganstanley.com> Message-ID: On Thu, Oct 24, 2019 at 4:57 PM Bekes, Andras G wrote: > > Hi Mikael, > > I filed a bug report in the bug tracker: https://bugs.erlang.org/browse/ERL-1074 > > Unfortunately printing *c_p did not reveal anything: > > (gdb) print c_p > $1 = > (gdb) print *c_p > value has been optimized out > > What should be the next step? > I can reliably produce 5-10 core dumps per week in my test system. I'd try to get a backtrace (bt command in gdb) from the crashed thread, then maybe print the c_p parameter via its raw value (print *(Process*)0x.....) if gdb insists that the value is optimized out. /Mikael > > -----Original Message----- > From: Mikael Pettersson [mailto:mikpelinux@REDACTED] > Sent: Friday, October 18, 2019 8:03 PM > To: Bekes, Andras G (IST) > Cc: Erlang Questions > Subject: Re: [erlang-questions] Segfault with Erlang R22 > > On Fri, Oct 18, 2019 at 4:34 PM Bekes, Andras G > wrote: > > > > Hi All, > > > > > > > > After upgrading to Erlang R22, my software crashes the Erlang VM with Segmentation fault. > > > > It happens rarely, only after several days of test workload, so I can?t really reproduce. > > > > > > > > I made more than 10 core dumps so far, loaded them into gdb, and all of them died at these 2 crash points: > > > > > > > > Program terminated with signal 11, Segmentation fault. > > > > #0 process_main (x_reg_array=0x20002, f_reg_array=0x2ade29280590) at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:4064 > > > > 4064 if (is_not_tuple(r(0))) { > > > > > > > > or > > > > > > > > #0 process_main (x_reg_array=0x20002, f_reg_array=0x2) at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:5252 > > > > 5252 c_p->seq_trace_lastcnt = unsigned_val(SEQ_TRACE_TOKEN_SERIAL(c_p)); > > > > > > > > The software is not doing any tracing when the crash happens, nor does it have any NIFs. > > I think you should open a bug report in the erlang bug tracker. > > The second crash site above is in the remove_message() function, in a > block where ERL_MESSAGE_TOKEN(msgp) > is neither NIL nor am_undefined, but SEQ_TRACE_TOKEN(c_p) is invalid > (not the expected 5-tuple). > > Maybe printing *c_p in gdb when that happens could shed some light. > > /Mikael > > -------------------------------------------------------------------------------- From sperber@REDACTED Thu Oct 24 20:44:43 2019 From: sperber@REDACTED (Michael Sperber) Date: Thu, 24 Oct 2019 20:44:43 +0200 Subject: 2nd Call for Contributions: BOB 2020 [Feb 28, Deadline Nov 8] Message-ID: Erlang submissions are very welcome at BOB! BOB Conference 2020 "What happens when we use what's best for a change?" http://bobkonf.de/2020/cfc.html Berlin, February 28 Call for Contributions Deadline: November 8, 2019 You are actively engaged in advanced software engineering methods, implement ambitious architectures and are open to cutting-edge innovation? Attend this conference, meet people that share your goals, and get to know the best software tools and technologies available today. We strive to offer a day full of new experiences and impressions that you can use to immediately improve your daily life as a software developer. If you share our vision and want to contribute, submit a proposal for a talk or tutorial! NOTE: The conference fee will be waived for presenters. Travel expenses will not be covered (for exceptions see "Speaker Grants"). Speaker Grants -------------- BOB has Speaker Grants available to support speakers from groups under-represented in technology. We specifically seek women speakers and speakers who are not be able to attend the conference for financial reasons. Shepherding ----------- The program committee offers shepherding to all speakers. Shepherding provides speakers assistance with preparing their sessions, as well as a review of the talk slides. Topics ------ We are looking for talks about best-of-breed software technology, e.g.: - functional programming - persistent data structures and database - event-based modelling and architectures - types - formal methods for correctness and robustness - abstractions for concurrency and parallelism - metaprogramming - probabilistic programming - math and programming - controlled side effects - beyond REST and SOAP - effective abstractions for data analytics - ... everything really that isn?t mainstream, but you think should be. Presenters should provide the audience with information that is practically useful for software developers. We're especially interested in experience reports. Other topics are also relevant, e.g.: - introductory talks on technical background - overviews of a given field - demos and how-tos Requirements ------------- We accept proposals for presentations of 45 minutes (40 minutes talk + 5 minutes questions), as well as 90 minute tutorials for beginners. The language of presentation should be either English or German. Your proposal should include (in your presentation language of choice): - An abstract of max. 1500 characters. - A short bio/cv - Contact information (including at least email address) - A list of 3-5 concrete ideas of how your work can be applied in a developer's daily life - additional material (websites, blogs, slides, videos of past presentations, ?) Submit here: https://bobcfc.active-group.de/en/bob2020/cfp Organisation ------------ - Direct questions to contact at bobkonf dot de - Proposal deadline: November 8, 2019 - Notification: November 22, 2019 - Program: December 6, 2019 Program Committee ----------------- (more information here: https://bobkonf.de/2020/programmkomitee.html) - Matthias Fischmann, Wire - Matthias Neubauer, SICK AG - Nicole Rauch, Softwareentwicklung und Entwicklungscoaching - Michael Sperber, Active Group - Stefan Wehr, factis research Scientific Advisory Board - Annette Bieniusa, TU Kaiserslautern - Torsten Grust, Uni T?bingen - Peter Thiemann, Uni Freiburg From boyofmonk@REDACTED Fri Oct 25 07:50:17 2019 From: boyofmonk@REDACTED (Monk Boy) Date: Fri, 25 Oct 2019 13:50:17 +0800 Subject: cerl: insert-string In-Reply-To: References: Message-ID: It works with change insert-string to insert My emacs: memacs@REDACTED:~/workspace/github/otp$ emacs --version GNU Emacs 26.1 Copyright (C) 2018 Free Software Foundation, Inc. GNU Emacs comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GNU Emacs under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING. https://github.com/emacs-mirror/emacs/blob/d0e2a341dd9a9a365fd311748df024ecb25b70ec/etc/NEWS.22#L4039 Monk Boy ?2019?10?24??? ??7:23??? > Hi all: > I have used cerl to debug beam(./bin/cerl -gdb), Run the debug > compiled emulator in emacs and gdb. When I input > > ``` > > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from beam.smp... > (gdb) r > Starting program: > /home/memacs/workspace/github/otp/bin/x86_64-unknown-linux-gnu/beam.smp > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > [New Thread 0x7fffb5e7f700 (LWP 8112)] > [New Thread 0x7fffb40bf700 (LWP 8113)] > [New Thread 0x7fffb5ef2700 (LWP 8114)] > [Detaching after fork from child process 8115] > [New Thread 0x7fffb36bf700 (LWP 8116)] > [New Thread 0x7fffb35bc700 (LWP 8117)] > [New Thread 0x7fffb34b9700 (LWP 8118)] > [New Thread 0x7fffb317f700 (LWP 8119)] > [New Thread 0x7fffb307c700 (LWP 8120)] > [New Thread 0x7fffb2f79700 (LWP 8121)] > [New Thread 0x7fffb2e76700 (LWP 8122)] > [New Thread 0x7fffb2d73700 (LWP 8123)] > [New Thread 0x7fffb2c70700 (LWP 8124)] > [New Thread 0x7fffb2c1d700 (LWP 8125)] > [New Thread 0x7fffb2bca700 (LWP 8126)] > [New Thread 0x7fffb2b77700 (LWP 8127)] > [New Thread 0x7fffb2b24700 (LWP 8128)] > [New Thread 0x7fffb2ad1700 (LWP 8129)] > [New Thread 0x7fffb2a7e700 (LWP 8130)] > [New Thread 0x7fffb2a2b700 (LWP 8131)] > [New Thread 0x7fffb29d8700 (LWP 8132)] > [New Thread 0x7fffb2985700 (LWP 8133)] > [New Thread 0x7fffb2932700 (LWP 8134)] > [New Thread 0x7fffb28df700 (LWP 8135)] > [New Thread 0x7fffb288c700 (LWP 8136)] > [New Thread 0x7fffb2839700 (LWP 8137)] > [New Thread 0x7fffb27e6700 (LWP 8138)] > [New Thread 0x7fffb2793700 (LWP 8139)] > [New Thread 0x7fffb2740700 (LWP 8140)] > [New Thread 0x7fffb26ed700 (LWP 8141)] > [New Thread 0x7fffb269a700 (LWP 8142)] > [New Thread 0x7fffb2647700 (LWP 8143)] > > Thread 11 "7_scheduler" received signal SIG35, Real-time event 35. > [Switching to Thread 0x7fffb2e76700 (LWP 8122)] > 0x00007ffff7d14e3f in epoll_wait (epfd=8, events=events@REDACTED=0x7fffb72c8b60, > maxevents=maxevents@REDACTED=512, timeout=timeout@REDACTED=-1) at > ../sysdeps/unix/sysv/linux/epoll_wait.c:30 > 30 ../sysdeps/unix/sysv/linux/epoll_wait.c: ?????????. > (gdb) > ``` > > I got: > ``` > {"init terminating in do_boot",no_or_multiple_root_variables} > init terminating in do_boot (no_or_multiple_root_variables) > > Crash dump is being written to: erl_crash.dump... > ``` > > > I must input: > (gdb) r -- -root /home/memacs/workspace/github/otp -progname > /home/memacs/workspace/github/otp/bin/cerl -- -home /home/memacs -- > Starting program: > /home/memacs/workspace/github/otp/bin/x86_64-unknown-linux-gnu/beam.smp -- > -root /home/memacs/workspace/github/otp -progname > /home/memacs/workspace/github/otp/bin/cerl -- -home /home/memacs -- > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > [New Thread 0x7fffb5e7f700 (LWP 8508)] > [New Thread 0x7fffb40bf700 (LWP 8509)] > [New Thread 0x7fffb5ef2700 (LWP 8510)] > [Detaching after fork from child process 8511] > [New Thread 0x7fffb36bf700 (LWP 8512)] > [New Thread 0x7fffb35bc700 (LWP 8513)] > [New Thread 0x7fffb34b9700 (LWP 8514)] > [New Thread 0x7fffb31b6700 (LWP 8515)] > [New Thread 0x7fffb33b6700 (LWP 8516)] > [New Thread 0x7fffb2e7f700 (LWP 8517)] > [New Thread 0x7fffb2d7c700 (LWP 8518)] > [New Thread 0x7fffb2c79700 (LWP 8519)] > [New Thread 0x7fffb32b3700 (LWP 8520)] > [New Thread 0x7fffb3260700 (LWP 8521)] > [New Thread 0x7fffb320d700 (LWP 8522)] > [New Thread 0x7fffb2b76700 (LWP 8523)] > [New Thread 0x7fffb2b23700 (LWP 8524)] > [New Thread 0x7fffb2ad0700 (LWP 8525)] > [New Thread 0x7fffb2a7d700 (LWP 8526)] > [New Thread 0x7fffb2a2a700 (LWP 8527)] > [New Thread 0x7fffb29d7700 (LWP 8528)] > [New Thread 0x7fffb2984700 (LWP 8529)] > [New Thread 0x7fffb2931700 (LWP 8530)] > [New Thread 0x7fffb28de700 (LWP 8531)] > [New Thread 0x7fffb288b700 (LWP 8532)] > [New Thread 0x7fffb2838700 (LWP 8533)] > [New Thread 0x7fffb27e5700 (LWP 8534)] > [New Thread 0x7fffb2792700 (LWP 8535)] > [New Thread 0x7fffb273f700 (LWP 8536)] > [New Thread 0x7fffb26ec700 (LWP 8537)] > [New Thread 0x7fffb2699700 (LWP 8538)] > [New Thread 0x7fffb2646700 (LWP 8539)] > ``` > > Last I find the cerl script > ``` > gdbcmd="(insert-string \"set args $beam_args\") \ > (comint-send-input)" > ``` > > But emacs has not the function (insert-string xxx) > (C-h f) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From micael.karlberg@REDACTED Fri Oct 25 12:39:07 2019 From: micael.karlberg@REDACTED (Micael Karlberg) Date: Fri, 25 Oct 2019 12:39:07 +0200 Subject: [erlang-questions] Matching IP address in socket module In-Reply-To: References: <20191024100444.GA28388@tpccl.tty1.net> <20191024153351.GB28388@tpccl.tty1.net> Message-ID: <64c9307c-f8ec-3599-3d21-f86fb09c9770@ericsson.com> Hi, Its early days still, but the goal is definitely that it should be faster. Here is some figures from a (time) test tool which is part of the test suite (its basically a ping-pong case), both server and client running on the same host (but in different VMs): Two transports: gen: gen_tcp sock: socket (tcp) The socket active mode is a simulation of gen_tcp active mode (no active = N). The tables are the exchange from the client side. With server side using gen (active = false): Transport Active Data gen false 10192 byte/ms, 154 msgs/ms gen true 10383 byte/ms, 157 msgs/ms gen once 6003 byte/ms, 90 msgs/ms sock false 14050 byte/ms, 212 msgs/ms sock true 14772 byte/ms, 223 msgs/ms sock once 14050 byte/ms, 210 msgs/ms With server side using gen (active = true): Transport Active Data gen false 9447 byte/ms, 143 msgs/ms gen true 22345 byte/ms, 338 msgs/ms gen once 5532 byte/ms, 83 msgs/ms sock false 15316 byte/ms, 232 msgs/ms sock true 23693 byte/ms, 358 msgs/ms sock once 22068 byte/ms, 334 msgs/ms With server side using sock (active = false, async = true): Transport Active Data gen false 11260 byte/ms, 170 msgs/ms gen true 22273 byte/ms, 337 msgs/ms gen once 7703 byte/ms, 116 msgs/ms sock false 15211 byte/ms, 230 msgs/ms sock true 24778 byte/ms, 375 msgs/ms sock once 23086 byte/ms, 349 msgs/ms With server side using sock (active = true, async = true): Transport Active Data gen false 11351 byte/ms, 171 msgs/ms gen true 22469 byte/ms, 340 msgs/ms gen once 7407 byte/ms, 112 msgs/ms sock false 15484 byte/ms, 234 msgs/ms sock true 24885 byte/ms, 377 msgs/ms sock once 23570 byte/ms, 357 msgs/ms There is of course a bit of overhead since the socket transport is trying to emulate (part of) gen_tcp. This is run on a Dell Precision T1700 running SLES 12 SP2 (not that that effects the relative performance). Run with a snapshot from the maint branch. This is obviously not a real use case, but it can be a guideline. Also no UDP test (at the moment). /BMK On 2019-10-24 18:48, Frank Muller wrote: > Hi Thomas > > Is the socket module faster than gen_tcp/gen_udp in your case? If yes, can you please share some > numbers. > > /Frank > > > case socket:recvfrom(Sock, 0, nowait) of > >? ? ? ?{ok, {#{family := Domain, > >? ? ? ? ? ? ? ?port? ?:= Port, > >? ? ? ? ? ? ? ?addr? ?:= Addr}, Data}} -> > >? ? ? ?. > > Hi Micael, Mark, > > thanks for your replies. The snippet above helped me getting the record > matched. > > I'm quite impressed with the socket module, it seems to be pretty > complete, at least for my application. > > Thanks, > Thomas > From frank.muller.erl@REDACTED Fri Oct 25 13:02:08 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 25 Oct 2019 13:02:08 +0200 Subject: [erlang-questions] Matching IP address in socket module In-Reply-To: <64c9307c-f8ec-3599-3d21-f86fb09c9770@ericsson.com> References: <20191024100444.GA28388@tpccl.tty1.net> <20191024153351.GB28388@tpccl.tty1.net> <64c9307c-f8ec-3599-3d21-f86fb09c9770@ericsson.com> Message-ID: Awesome, thanks! Le ven. 25 oct. 2019 ? 12:39, Micael Karlberg a ?crit : > Hi, > > Its early days still, but the goal is definitely that it should > be faster. > > Here is some figures from a (time) test tool which is part of the > test suite (its basically a ping-pong case), both server and client > running on the same host (but in different VMs): > > Two transports: > gen: gen_tcp > sock: socket (tcp) > > The socket active mode is a simulation of gen_tcp active mode > (no active = N). > > The tables are the exchange from the client side. > > With server side using gen (active = false): > > Transport Active Data > gen false 10192 byte/ms, 154 msgs/ms > gen true 10383 byte/ms, 157 msgs/ms > gen once 6003 byte/ms, 90 msgs/ms > sock false 14050 byte/ms, 212 msgs/ms > sock true 14772 byte/ms, 223 msgs/ms > sock once 14050 byte/ms, 210 msgs/ms > > > With server side using gen (active = true): > > Transport Active Data > gen false 9447 byte/ms, 143 msgs/ms > gen true 22345 byte/ms, 338 msgs/ms > gen once 5532 byte/ms, 83 msgs/ms > sock false 15316 byte/ms, 232 msgs/ms > sock true 23693 byte/ms, 358 msgs/ms > sock once 22068 byte/ms, 334 msgs/ms > > > With server side using sock (active = false, async = true): > > Transport Active Data > gen false 11260 byte/ms, 170 msgs/ms > gen true 22273 byte/ms, 337 msgs/ms > gen once 7703 byte/ms, 116 msgs/ms > sock false 15211 byte/ms, 230 msgs/ms > sock true 24778 byte/ms, 375 msgs/ms > sock once 23086 byte/ms, 349 msgs/ms > > > With server side using sock (active = true, async = true): > > Transport Active Data > gen false 11351 byte/ms, 171 msgs/ms > gen true 22469 byte/ms, 340 msgs/ms > gen once 7407 byte/ms, 112 msgs/ms > sock false 15484 byte/ms, 234 msgs/ms > sock true 24885 byte/ms, 377 msgs/ms > sock once 23570 byte/ms, 357 msgs/ms > > > There is of course a bit of overhead since the socket transport > is trying to emulate (part of) gen_tcp. > > This is run on a Dell Precision T1700 running SLES 12 SP2 (not that > that effects the relative performance). > Run with a snapshot from the maint branch. > > This is obviously not a real use case, but it can be a guideline. > Also no UDP test (at the moment). > > /BMK > > On 2019-10-24 18:48, Frank Muller wrote: > > Hi Thomas > > > > Is the socket module faster than gen_tcp/gen_udp in your case? If yes, > can you please share some > > numbers. > > > > /Frank > > > > > case socket:recvfrom(Sock, 0, nowait) of > > > {ok, {#{family := Domain, > > > port := Port, > > > addr := Addr}, Data}} -> > > > . > > > > Hi Micael, Mark, > > > > thanks for your replies. The snippet above helped me getting the > record > > matched. > > > > I'm quite impressed with the socket module, it seems to be pretty > > complete, at least for my application. > > > > Thanks, > > Thomas > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ratmapper@REDACTED Fri Oct 25 15:23:35 2019 From: ratmapper@REDACTED (Raimo Niskanen) Date: Fri, 25 Oct 2019 15:23:35 +0200 Subject: Nobody is unsubscribed Message-ID: The reason we changed mailing list servers was to get better DMARC and DKIM compliance. This is a test post for us to inspect its headers... -- Raimo Niskanen From hans.r.nilsson@REDACTED Fri Oct 25 16:25:25 2019 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Fri, 25 Oct 2019 14:25:25 +0000 Subject: Patch Package OTP 21.3.8.10 Released Message-ID: Patch Package: OTP 21.3.8.10 Git Tag: OTP-21.3.8.10 Date: 2019-10-25 Trouble Report Id: OTP-14849, OTP-16056 Seq num: ERL-545 System: OTP Release: 21 Application: ftp-1.0.2.1, ssh-4.7.6.2 Predecessor: OTP 21.3.8.9 Check out the git tag OTP-21.3.8.10, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- ftp-1.0.2.1 ----------------------------------------------------- --------------------------------------------------------------------- The ftp-1.0.2.1 application can be applied independently of other applications on a full OTP 21 installation. --- Fixed Bugs and Malfunctions --- OTP-16056 Application(s): ftp A possibly infinite loop when receiving messages divided in parts is removed. Full runtime dependencies of ftp-1.0.2.1: erts-7.0, kernel-6.0, stdlib-3.5 --------------------------------------------------------------------- --- ssh-4.7.6.2 ----------------------------------------------------- --------------------------------------------------------------------- The ssh-4.7.6.2 application can be applied independently of other applications on a full OTP 21 installation. --- Fixed Bugs and Malfunctions --- OTP-14849 Application(s): ssh Related Id(s): ERL-545 The ssh cli (e.g shell) server behaved strangely when characters were inserted in a string so that the last characters tried to wrap the line. Full runtime dependencies of ssh-4.7.6.2: crypto-4.2, erts-6.0, kernel-3.0, public_key-1.5.2, stdlib-3.3 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From v@REDACTED Fri Oct 25 16:36:59 2019 From: v@REDACTED (Valentin Micic) Date: Fri, 25 Oct 2019 16:36:59 +0200 Subject: Binary, List and Tuple Inequalities (Paradox?) Message-ID: <46D4876E-C091-4A16-9A2B-D91DA65986AE@micic.co.za> Hi all, Consider the following inequalities: (tsdb_1_1@REDACTED)2876> <<0,0,1>> < <<1,0,0>>. true As well as: (tsdb_1_1@REDACTED)2878> [0,0,1] < [1,0,0]. true The result (true) makes sense ? in both cases operands are of the same size, and the first (leftmost) element value of the left operand is less than first (again, leftmost) element value of the right operand. However: (tsdb_1_1@REDACTED)2877> <<0,0,1>> < <<1,0>>. true and (tsdb_1_1@REDACTED)2879> [0,0,1] < [1,0]. true This indicates that the actual length of the operands are not considered, for clearly, three octets cannot be less then two, right? This becomes even more confusing if you check how tuples are compared given the similar test-case: (tsdb_1_1@REDACTED)2886> {0,0,1} < {1,0,0}. true (tsdb_1_1@REDACTED)2887> {0,0,1} < {1,0}. false Here, the number of elements in the tuple are clearly taken into the consideration. Then, if one converts all three pairs of operands to its (external) binary representation, e.g. Binary: (tsdb_1_1@REDACTED)2916> term_to_binary( <<0,0,1>> ). <<131,109,0,0,0,3,0,0,1>> (tsdb_1_1@REDACTED)2917> term_to_binary( <<1,0>> ). <<131,109,0,0,0,2,1,0>> List: tsdb_1_1@REDACTED)2918> term_to_binary( [0,0,1] ). <<131,107,0,3,0,0,1>> (tsdb_1_1@REDACTED)2919> term_to_binary( [1,0] ). <<131,107,0,2,1,0>> Tuple: (tsdb_1_1@REDACTED)2920> term_to_binary( {0,0,1} ). <<131,104,3,97,0,97,0,97,1>> (tsdb_1_1@REDACTED)2921> term_to_binary( {1,0} ). <<131,104,2,97,1,97,0>> One could see that the number of ?elements? are known and available for comparison (I?ve highlighted this number using bold, yellow font) for all three situations. And yet, the different approaches between binary and lists on one side, and tuples, on another, appear to be deliberate as binary and list types are following the approach used by C standard library memcmp function, whereas tuples do not, e.g. int memcmp(const void *s1, const void *s2, size_t n); In other words, operands are reduced to some common size ?n', and then compared as if they were equal in length. I do understand this restriction in C -- if not for ?n?, the argument sizes are not known, and without it the execution may end up in a catastrophic failure (e.g. segment violation, hmm.. which can still happen anyway if ?n? is inadequate). So, THE QUESTION: Why would it be wrong to consider inequalities for binary() data types the same way we do it for tuples ? number of ?elements? first and ?numeric values? of individual elements ? second? In my view, this is not only not wrong, but it would be more logical, and this is why. Look at the following comparison: (tsdb_1_1@REDACTED)2957> <<0,0,1>> < <<0,0,1,0>>. true Well, this make sense, for less-is-less (e.g. three octets are less than four octets) and even if one considers <<0,0,1>> as a 24-bit integer value, and <<0,0,1,0>> as a 32-bit integer, one may ?normalise? the 24-bit integer to its 32-bit equivalent by adding a leading zero, and the comparison expression would still return true: (tsdb_1_1@REDACTED)2961> <<0,0,0,1>> < <<0,0,1,0>>. true Therefore, as expected, 1 < 256, and, thus, we may be forgiven if we have the same expectation when we write: (tsdb_1_1@REDACTED)2958> <<1>> < <<0,0,1,0>>. Right? Well, not so, because: (tsdb_1_1@REDACTED)2958> <<1>> < <<0,0,1,0>>. false Thus, "less is more", and therefore 1 appears to be greater-than-or-equal-to 256. Somehow. This could never happen if one considered a number of elements (in this case number of octets) before attempting to compare the individual element values. But wait, the "less-is-more" may easily become its opposite ? ?more is less?, because: (tsdb_1_1@REDACTED)2973> <<32, 97>> < <<97>>. true Yes, this approach may help us to sort a list of, say, surnames alphabetically, so we can have all the ?Ms? before the ?Ns?, regardless of how long the surname is, but is this actually logical for two arbitrary binaries? Seeing that we actually care about number of elements when we deal with tuples, why not extend the same approach to arbitrary binary values? Why do we presume that binaries are given as STRINGS when we compare them? When it comes to lists, the situation is even worse, because lists do, but then do not follow the STRING comparison semantic: (tsdb_1_1@REDACTED)2991> [32, 97] < [97]. true *** But! *** (tsdb_1_1@REDACTED)2992> [{32, 97}] < [97]. false (tsdb_1_1@REDACTED)2993> [{32, 97}] < [{97}]. false And just to be clear, I do not expect Erlang to change. Nor do I care about how lists are handled (well, at least not for the moment). My main concern is semantic behind comparing two arbitrary binary() operands. For one of my projects I?d like to employ tuple-like semantic (.e.g. number of octets first, values later) when comparing two arbitrary binaries and thus avoid ?less-is-more? and ?more-is-less? paradoxes. Is there any reason why I shouldn?t? (Okay, it may be a bit slower, but one may argue this approach may be even a bit faster for a long-enough binaries. But what else?) At any rate, I would appreciate your reasoning on the topic. What resonates better with you? Thanks in advance. V/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ratmapper@REDACTED Fri Oct 25 16:38:46 2019 From: ratmapper@REDACTED (Raimo Niskanen) Date: Fri, 25 Oct 2019 16:38:46 +0200 Subject: Nobody is unsubscribed In-Reply-To: References: Message-ID: To achieve DMARC compliance we have stopped changing the Subject: field and no longer add the mailing list footer to the messages. This is because From: Subject: and mail body among other fields are often DKIM signed, so if we should change them we would not pass DKIM signature check and thereby not be DMARC compliant. Sorry for the inconvenience, we do not make the rules... / Raimo Niskanen On Fri, Oct 25, 2019 at 3:23 PM Raimo Niskanen wrote: > > The reason we changed mailing list servers was to get better DMARC and > DKIM compliance. This is a test post for us to inspect its headers... > -- > Raimo Niskanen From as_rr@REDACTED Fri Oct 25 16:14:49 2019 From: as_rr@REDACTED (Andreas Schultz) Date: Fri, 25 Oct 2019 14:14:49 +0000 (UTC) Subject: Nobody is unsubscribed In-Reply-To: References: Message-ID: <989498580.453149.1572012889383@mail.yahoo.com> Works on gmail, but produces a SPF error on Yahoo: Authentication-Results: mta1148.mail.ir2.yahoo.com;? ?dkim=pass (ok) header.i=@gmail.com header.s=20161025;?spf=permerror smtp.mailfrom=@erlang.org;?dmarc=pass(p=none sp=quarantine dis=none) header.from=gmail.com;Received-SPF: permerror (encountered permanent error during SPF processing of domain of erlang.org) Am Freitag, 25. Oktober 2019, 06:24:16 GMT-7 hat Raimo Niskanen Folgendes geschrieben: The reason we changed mailing list servers was to get better DMARC and DKIM compliance.? This is a test post for us to inspect its headers... -- Raimo Niskanen -------------- next part -------------- An HTML attachment was scrubbed... URL: From csrl@REDACTED Fri Oct 25 16:58:30 2019 From: csrl@REDACTED (Chris Rempel) Date: Fri, 25 Oct 2019 16:58:30 +0200 Subject: Nobody is unsubscribed In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From be.dmitry@REDACTED Fri Oct 25 17:35:20 2019 From: be.dmitry@REDACTED (Dmitry Belyaev) Date: Sat, 26 Oct 2019 02:35:20 +1100 Subject: Binary, List and Tuple Inequalities (Paradox?) In-Reply-To: <46D4876E-C091-4A16-9A2B-D91DA65986AE@micic.co.za> References: <46D4876E-C091-4A16-9A2B-D91DA65986AE@micic.co.za> Message-ID: To me alphabetical ordering of lists (as legacy string representation) and binaries (as modern string representation) is natural, so it is expected that "aa" < "b" < "bb". Not sure about binaries ordering which first checks the binary size, like "b" < "aa" < "bb" - I personally cannot find a suitable application for such ordering. And about the lists. Their length is not known as for binaries. It can only be calculated with linear complexity. If you have a list 'aaa..a' of 50 elements and a list 'bbb..b' of 49, you'd have to make 49*2 iterations to determine which one is less. With the current implementation it is just 1 comparison. On 26 October 2019 1:36:59 am AEDT, Valentin Micic wrote: >Hi all, > >Consider the following inequalities: > >(tsdb_1_1@REDACTED)2876> <<0,0,1>> < <<1,0,0>>. >true > >As well as: > >(tsdb_1_1@REDACTED)2878> [0,0,1] < [1,0,0]. >true > >The result (true) makes sense ? in both cases operands are of the same >size, and the first (leftmost) element value of the left operand is >less than first (again, leftmost) element value of the right operand. > >However: > >(tsdb_1_1@REDACTED)2877> <<0,0,1>> < <<1,0>>. >true > >and > >(tsdb_1_1@REDACTED)2879> [0,0,1] < [1,0]. >true > >This indicates that the actual length of the operands are not >considered, for clearly, three octets cannot be less then two, right? > >This becomes even more confusing if you check how tuples are compared >given the similar test-case: > >(tsdb_1_1@REDACTED)2886> {0,0,1} < {1,0,0}. >true >(tsdb_1_1@REDACTED)2887> {0,0,1} < {1,0}. >false > >Here, the number of elements in the tuple are clearly taken into the >consideration. > >Then, if one converts all three pairs of operands to its (external) >binary representation, e.g. > >Binary: > >(tsdb_1_1@REDACTED)2916> term_to_binary( <<0,0,1>> ). ><<131,109,0,0,0,3,0,0,1>> >(tsdb_1_1@REDACTED)2917> term_to_binary( <<1,0>> ). ><<131,109,0,0,0,2,1,0>> > >List: > >tsdb_1_1@REDACTED)2918> term_to_binary( [0,0,1] ). ><<131,107,0,3,0,0,1>> >(tsdb_1_1@REDACTED)2919> term_to_binary( [1,0] ). ><<131,107,0,2,1,0>> > >Tuple: >(tsdb_1_1@REDACTED)2920> term_to_binary( {0,0,1} ). ><<131,104,3,97,0,97,0,97,1>> >(tsdb_1_1@REDACTED)2921> term_to_binary( {1,0} ). ><<131,104,2,97,1,97,0>> > >One could see that the number of ?elements? are known and available for >comparison (I?ve highlighted this number using bold, yellow font) for >all three situations. > >And yet, the different approaches between binary and lists on one side, >and tuples, on another, appear to be deliberate as binary and list >types are following the approach used by C standard library memcmp >function, whereas tuples do not, e.g. > >int memcmp(const void *s1, const void *s2, size_t n); >In other words, operands are reduced to some common size ?n', and then >compared as if they were equal in length. >I do understand this restriction in C -- if not for ?n?, the argument >sizes are not known, and without it the execution may end up in a >catastrophic failure (e.g. segment violation, hmm.. which can still >happen anyway if ?n? is inadequate). > >So, THE QUESTION: > >Why would it be wrong to consider inequalities for binary() data types >the same way we do it for tuples ? number of ?elements? first and >?numeric values? of individual elements ? second? > >In my view, this is not only not wrong, but it would be more logical, >and this is why. > >Look at the following comparison: > >(tsdb_1_1@REDACTED)2957> <<0,0,1>> < <<0,0,1,0>>. >true > > >Well, this make sense, for less-is-less (e.g. three octets are less >than four octets) and even if one considers <<0,0,1>> as a 24-bit >integer value, and <<0,0,1,0>> as a 32-bit integer, one may ?normalise? >the 24-bit integer to its 32-bit equivalent by adding a leading zero, >and the comparison expression would still return true: > >(tsdb_1_1@REDACTED)2961> <<0,0,0,1>> < <<0,0,1,0>>. >true > >Therefore, as expected, 1 < 256, and, thus, we may be forgiven if we >have the same expectation when we write: > >(tsdb_1_1@REDACTED)2958> <<1>> < <<0,0,1,0>>. > >Right? Well, not so, because: > >(tsdb_1_1@REDACTED)2958> <<1>> < <<0,0,1,0>>. >false > >Thus, "less is more", and therefore 1 appears to be >greater-than-or-equal-to 256. Somehow. > >This could never happen if one considered a number of elements (in this >case number of octets) before attempting to compare the individual >element values. > >But wait, the "less-is-more" may easily become its opposite ? ?more is >less?, because: > >(tsdb_1_1@REDACTED)2973> <<32, 97>> < <<97>>. >true > >Yes, this approach may help us to sort a list of, say, surnames >alphabetically, so we can have all the ?Ms? before the ?Ns?, regardless >of how long the surname is, but is this actually logical for two >arbitrary binaries? Seeing that we actually care about number of >elements when we deal with tuples, why not extend the same approach to >arbitrary binary values? > >Why do we presume that binaries are given as STRINGS when we compare >them? When it comes to lists, the situation is even worse, because >lists do, but then do not follow the STRING comparison semantic: > >(tsdb_1_1@REDACTED)2991> [32, 97] < [97]. >true > >*** But! *** > >(tsdb_1_1@REDACTED)2992> [{32, 97}] < [97]. >false > >(tsdb_1_1@REDACTED)2993> [{32, 97}] < [{97}]. >false > >And just to be clear, I do not expect Erlang to change. Nor do I care >about how lists are handled (well, at least not for the moment). > >My main concern is semantic behind comparing two arbitrary binary() >operands. For one of my projects I?d like to employ tuple-like semantic >(.e.g. number of octets first, values later) when comparing two >arbitrary binaries and thus avoid ?less-is-more? and ?more-is-less? >paradoxes. > >Is there any reason why I shouldn?t? (Okay, it may be a bit slower, but >one may argue this approach may be even a bit faster for a long-enough >binaries. But what else?) > >At any rate, I would appreciate your reasoning on the topic. What >resonates better with you? > > >Thanks in advance. > >V/ -- Kind regards, Dmitry Belyaev -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Fri Oct 25 17:43:35 2019 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 25 Oct 2019 11:43:35 -0400 Subject: Binary, List and Tuple Inequalities (Paradox?) In-Reply-To: References: <46D4876E-C091-4A16-9A2B-D91DA65986AE@micic.co.za> Message-ID: On Fri, Oct 25, 2019 at 11:35 AM Dmitry Belyaev wrote: > To me alphabetical ordering of lists (as legacy string representation) and > binaries (as modern string representation) is natural, so it is expected > that "aa" < "b" < "bb". > > As a general note, this is only true to the extent that you don't care that "ZOOM" is < than "apple" in this dictionary. It is anything but natural, I would say it's probably more "accidental". Actual string sorting tends to rely on a Unicode collation algorithm (a part of the spec that is still missing in the stdlib), since even string normalization won't be enough to ensure things are done safely with regards to human expectations. At this point only libraries such as ux support it well. Most of the orderings we have right now are often arbitrary and can have surprising sides to them. At this point for most data structures the important thing is not that the ordering is correct, but that at least one exists at all; this allows the creation of a bunch of data structures (usually trees and variants) to work on all data types, mixed or not. All atoms are smaller than all strings, even if atoms have a string component to them, for example. -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Fri Oct 25 19:15:10 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 25 Oct 2019 19:15:10 +0200 Subject: IP address/IP range auth module for Erlang Message-ID: Hi guys I?m looking for an Erlang module which let me authorize connexions based on an IP address or IP address range. Examples: . Rule 1: "51.121.92.25" Only one IP address is allowed: 51.121.92.25 . Rule 2: "51.121.*.25" These IPs are all allowed: 51.121.92.25 51.121.91.25 51.121.90.25 ? Would be great if the module can also validate the connection IP address. For now, I only deal with IP v4 addresses (but I?m open to support IP v6 in my code). Thanks in advance /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke@REDACTED Fri Oct 25 19:22:59 2019 From: luke@REDACTED (Luke Bakken) Date: Fri, 25 Oct 2019 10:22:59 -0700 Subject: IP address/IP range auth module for Erlang In-Reply-To: References: Message-ID: Hi Frank, You may be able to get some inspiration from this RabbitMQ plugin: https://github.com/gotthardp/rabbitmq-auth-backend-ip-range/ Thanks, Luke On Fri, Oct 25, 2019 at 10:15 AM Frank Muller wrote: > > Hi guys > > I?m looking for an Erlang module which let me authorize connexions based on an IP address or IP address range. > > Examples: > > . Rule 1: "51.121.92.25" > > Only one IP address is allowed: 51.121.92.25 > > . Rule 2: "51.121.*.25" > > These IPs are all allowed: > 51.121.92.25 > 51.121.91.25 > 51.121.90.25 > ? > > Would be great if the module can also validate the connection IP address. For now, I only deal with IP v4 addresses (but I?m open to support IP v6 in my code). > > Thanks in advance > /Frank From frank.muller.erl@REDACTED Fri Oct 25 20:10:28 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 25 Oct 2019 20:10:28 +0200 Subject: IP address/IP range auth module for Erlang In-Reply-To: References: Message-ID: Hi Luke I quickly reviewed the code, and found this section intriguing: https://github.com/gotthardp/rabbitmq-auth-backend-ip-range/blob/master/src/rabbit_auth_backend_ip_range.erl#L57-L60 How about just this instead: Addr == Mask, Anyway, thank you. If no one else has another alternative, I?m gonna hack this module a little bit to fit my needs. /Frank Le ven. 25 oct. 2019 ? 19:23, Luke Bakken a ?crit : > Hi Frank, > > You may be able to get some inspiration from this RabbitMQ plugin: > > https://github.com/gotthardp/rabbitmq-auth-backend-ip-range/ > > Thanks, > Luke > > On Fri, Oct 25, 2019 at 10:15 AM Frank Muller > wrote: > > > > Hi guys > > > > I?m looking for an Erlang module which let me authorize connexions based > on an IP address or IP address range. > > > > Examples: > > > > . Rule 1: "51.121.92.25" > > > > Only one IP address is allowed: 51.121.92.25 > > > > . Rule 2: "51.121.*.25" > > > > These IPs are all allowed: > > 51.121.92.25 > > 51.121.91.25 > > 51.121.90.25 > > ? > > > > Would be great if the module can also validate the connection IP > address. For now, I only deal with IP v4 addresses (but I?m open to support > IP v6 in my code). > > > > Thanks in advance > > /Frank > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marc@REDACTED Fri Oct 25 20:39:53 2019 From: marc@REDACTED (Marc Worrell) Date: Fri, 25 Oct 2019 20:39:53 +0200 Subject: IP address/IP range auth module for Erlang In-Reply-To: References: Message-ID: <274B96BE-50FD-4937-8979-651F31AA4104@worrell.nl> [2nd try, now from correct email address] Hi Frank, You can check zotonic_stdlib, there we have routines to check an IP address against IP address ranges. https://github.com/zotonic/z_stdlib/blob/master/src/z_ip_address.erl Cheers, Marc Worrell > On 25 Oct 2019, at 19:15, Frank Muller wrote: > > Hi guys > > I?m looking for an Erlang module which let me authorize connexions based on an IP address or IP address range. > > Examples: > > . Rule 1: "51.121.92.25" > > Only one IP address is allowed: 51.121.92.25 > > . Rule 2: "51.121.*.25" > > These IPs are all allowed: > 51.121.92.25 > 51.121.91.25 > 51.121.90.25 > ? > > Would be great if the module can also validate the connection IP address. For now, I only deal with IP v4 addresses (but I?m open to support IP v6 in my code). > > Thanks in advance > /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.muller.erl@REDACTED Fri Oct 25 22:41:29 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Fri, 25 Oct 2019 22:41:29 +0200 Subject: IP address/IP range auth module for Erlang In-Reply-To: <274B96BE-50FD-4937-8979-651F31AA4104@worrell.nl> References: <274B96BE-50FD-4937-8979-651F31AA4104@worrell.nl> Message-ID: Thanks Marc. Quickly checked the code and it looks clean and simple. Exactly what I was looking for. /Frank Le ven. 25 oct. 2019 ? 20:39, Marc Worrell a ?crit : > [2nd try, now from correct email address] > > Hi Frank, > > You can check zotonic_stdlib, there we have routines to check an IP > address against IP address ranges. > > https://github.com/zotonic/z_stdlib/blob/master/src/z_ip_address.erl > > Cheers, > > Marc Worrell > > > On 25 Oct 2019, at 19:15, Frank Muller wrote: > > Hi guys > > I?m looking for an Erlang module which let me authorize connexions based > on an IP address or IP address range. > > Examples: > > . Rule 1: "51.121.92.25" > > Only one IP address is allowed: 51.121.92.25 > > . Rule 2: "51.121.*.25" > > These IPs are all allowed: > 51.121.92.25 > 51.121.91.25 > 51.121.90.25 > ? > > Would be great if the module can also validate the connection IP address. > For now, I only deal with IP v4 addresses (but I?m open to support IP v6 in > my code). > > Thanks in advance > /Frank > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ratmapper@REDACTED Sat Oct 26 08:57:20 2019 From: ratmapper@REDACTED (Raimo Niskanen) Date: Sat, 26 Oct 2019 08:57:20 +0200 Subject: Nobody is unsubscribed In-Reply-To: References: Message-ID: It is mainly "the big ones" that have been affected by stricter DMARC policies. When a subscriber sending from e.g Yahoo gets received by Gmail then Gmail rejects that message since Yahoo's DMARC policy says so (also vice versa). So the list gets a bounce and eventually blocks the Gmail subscriber, if enough in a row happens to send with strict DMARC policies. So for some it has worked, some gets an annoying list probe every now and then, some do not get many posts, but the final nail in the coffin was Ericsson (Erlang/OTP's home corporation) that tightened its DMARC policy and at the same time told us to get our act together and stop sending "unhygienic e-mail". All the best / Raimo Den fre 25 okt. 2019 16:58Chris Rempel skrev: > Not having the subject contain [erlang-questions] or some other obvious > indicator is quite unfortunate. I guess many people were affected by not > being DMARC compliant? It seems to have been working just fine for quite > some time... ie it "works for me" as it was. > > That said, thanks for maintaining the list, and keeping it going. It is a > most useful resource. > > Chris > > *Sent:* Friday, October 25, 2019 at 7:38 AM > *From:* "Raimo Niskanen" > *To:* erlang-questions@REDACTED > *Subject:* Re: Nobody is unsubscribed > To achieve DMARC compliance we have stopped changing the Subject: > field and no longer add the mailing list footer to the messages. > > This is because From: Subject: and mail body among other fields are > often DKIM signed, so if we should change them we would not pass DKIM > signature check and thereby not be DMARC compliant. > > Sorry for the inconvenience, we do not make the rules... > / Raimo Niskanen > > On Fri, Oct 25, 2019 at 3:23 PM Raimo Niskanen > wrote: > > > > The reason we changed mailing list servers was to get better DMARC and > > DKIM compliance. This is a test post for us to inspect its headers... > > -- > > Raimo Niskanen > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtp11@REDACTED Sat Oct 26 11:41:54 2019 From: rtp11@REDACTED (rtp) Date: Sat, 26 Oct 2019 11:41:54 +0200 Subject: Nobody is unsubscribed In-Reply-To: References: Message-ID: <7652aa50-455a-2269-56ea-736c786be267@abwesend.de> What about using GPG for encrypting and signing E-Mails. A lot of problems would be solved at once. :-) Cheerio Ralf PS: Sorry, couldn't resist to write this mail. On 25.10.19 16:38, Raimo Niskanen wrote: > To achieve DMARC compliance we have stopped changing the Subject: > field and no longer add the mailing list footer to the messages. > > This is because From: Subject: and mail body among other fields are > often DKIM signed, so if we should change them we would not pass DKIM > signature check and thereby not be DMARC compliant. > > Sorry for the inconvenience, we do not make the rules... > / Raimo Niskanen > > On Fri, Oct 25, 2019 at 3:23 PM Raimo Niskanen wrote: >> >> The reason we changed mailing list servers was to get better DMARC and >> DKIM compliance. This is a test post for us to inspect its headers... >> -- >> Raimo Niskanen From klg.amit@REDACTED Sat Oct 26 22:38:11 2019 From: klg.amit@REDACTED (Amit K) Date: Sat, 26 Oct 2019 23:38:11 +0300 Subject: Zeroization of sensitive data in Erlang Message-ID: Hi all, Concerning the security practice of Zeroizing sensitive data from memory after you're finished with it, such as Cryptographic keys, passwords, etc, I wanted to ask if any of you ever had to implement such a thing in Erlang, and if so, how? :) Also note that often this is a requirement that you must fulfill if you want your product to pass a security standard evaluation such as the "Common Criteria" one (As an example requirement see FCS_CKM_EXT.4 here: https://www.commoncriteriaportal.org/files/ppfiles/pp_nd_v1.0.pdf). The problem of course is that in Erlang (and I suppose - FP in general) once a variable gets assigned, you are not allowed to change its value, and in particular of course overwrite it with zero bytes. One option I thought about is doing it with a NIF, but from my understanding a NIF isn't supposed to change its input values, it should treat them as constants, which understandable as well (basic FP language property). Any feedback can be helpful :) Regards, Amit -------------- next part -------------- An HTML attachment was scrubbed... URL: From dszoboszlay@REDACTED Sun Oct 27 00:42:09 2019 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Sun, 27 Oct 2019 00:42:09 +0200 Subject: Zeroization of sensitive data in Erlang In-Reply-To: References: Message-ID: Hi Amit, A NIF could use a resource object to implement a zeroable memory location. The Erlang program will only see a handle to the resource, which is immutable. But the contents of the object can be changed by the NIF (it's better explained in the docs ). So you could create an API that looks like this for the Erlang program: Handle = create_secret(Some, Input, Values), ok = use_secret(Handle), ok = destroy_secret(Handle), % The NIF zeroes out the memory in the resource object {error, destroyed} = use_secret(Handle). % The secret is no more at this point The big problem is that your secret shall never leave the native library. E.g. you cannot hide a cryptographic key behind a zeroable resource object and then pass it to the crypto library. As soon as you convert it into an Erlang term, it is copied onto the non-zeroed stack or heap or a process. Considering this, you may just as well use a port program to hold and work with the secret outside of the Erlang VM. Probably the only real solution within the VM would be to patch the GC to zero any reclaimed memory. Or maybe not even the GC, but the memory allocators, in case a secret is copied to some non-GC-d memory area, such as an ETS table. Cheers, Daniel On Sat, 26 Oct 2019 at 22:39, Amit K wrote: > Hi all, > > Concerning the security practice of Zeroizing sensitive data from memory > after you're finished with it, such as Cryptographic keys, passwords, etc, > I wanted to ask if any of you ever had to implement such a thing in Erlang, > and if so, how? :) > > Also note that often this is a requirement that you must fulfill if you > want your product to pass a security standard evaluation such as the > "Common Criteria" one (As an example requirement see FCS_CKM_EXT.4 here: > https://www.commoncriteriaportal.org/files/ppfiles/pp_nd_v1.0.pdf). > > The problem of course is that in Erlang (and I suppose - FP in general) > once a variable gets assigned, you are not allowed to change its value, and > in particular of course overwrite it with zero bytes. > > One option I thought about is doing it with a NIF, but from my > understanding a NIF isn't supposed to change its input values, it should > treat them as constants, which understandable as well (basic FP language > property). > > Any feedback can be helpful :) > > Regards, > > Amit > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From klg.amit@REDACTED Sun Oct 27 01:02:12 2019 From: klg.amit@REDACTED (Amit K) Date: Sun, 27 Oct 2019 01:02:12 +0200 Subject: Zeroization of sensitive data in Erlang In-Reply-To: References: Message-ID: Hi Daniel, Good to know about that workaround, thank you. However as you already pointed out, I really do want to be able to handle such "sensitive" variables in Erlang code and use the standard "crypto" library of the OTP. With regards to GC - that's a great direction I think, and perhaps a more finer grained approach would be to add a new "sensitive" flag for typed variables that the GC can recognize. Then only those variables get zeroized when the GC runs. Would probably be better for performance than always zeroizing everything that we free... BTW - so far I didn't find any FP language that can do something like that, and please correct me if I'm wrong. It would be cool if Erlang would be the first to the gold here (again!) :) On Sun, Oct 27, 2019 at 1:42 AM D?niel Szoboszlay wrote: > Hi Amit, > > A NIF could use a resource object to implement a zeroable memory location. > The Erlang program will only see a handle to the resource, which is > immutable. But the contents of the object can be changed by the NIF (it's > better explained in the docs ). > So you could create an API that looks like this for the Erlang program: > > Handle = create_secret(Some, Input, Values), > ok = use_secret(Handle), > ok = destroy_secret(Handle), % The NIF zeroes out the memory in the > resource object > {error, destroyed} = use_secret(Handle). % The secret is no more at this > point > > The big problem is that your secret shall never leave the native library. > E.g. you cannot hide a cryptographic key behind a zeroable resource object > and then pass it to the crypto library. As soon as you convert it into an > Erlang term, it is copied onto the non-zeroed stack or heap or a process. > Considering this, you may just as well use a port program to hold and work > with the secret outside of the Erlang VM. > > Probably the only real solution within the VM would be to patch the GC to > zero any reclaimed memory. Or maybe not even the GC, but the memory > allocators, in case a secret is copied to some non-GC-d memory area, such > as an ETS table. > > Cheers, > Daniel > > On Sat, 26 Oct 2019 at 22:39, Amit K wrote: > >> Hi all, >> >> Concerning the security practice of Zeroizing sensitive data from memory >> after you're finished with it, such as Cryptographic keys, passwords, etc, >> I wanted to ask if any of you ever had to implement such a thing in Erlang, >> and if so, how? :) >> >> Also note that often this is a requirement that you must fulfill if you >> want your product to pass a security standard evaluation such as the >> "Common Criteria" one (As an example requirement see FCS_CKM_EXT.4 here: >> https://www.commoncriteriaportal.org/files/ppfiles/pp_nd_v1.0.pdf). >> >> The problem of course is that in Erlang (and I suppose - FP in general) >> once a variable gets assigned, you are not allowed to change its value, and >> in particular of course overwrite it with zero bytes. >> >> One option I thought about is doing it with a NIF, but from my >> understanding a NIF isn't supposed to change its input values, it should >> treat them as constants, which understandable as well (basic FP language >> property). >> >> Any feedback can be helpful :) >> >> Regards, >> >> Amit >> >> >> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Sun Oct 27 02:16:43 2019 From: bob@REDACTED (Bob Ippolito) Date: Sat, 26 Oct 2019 17:16:43 -0700 Subject: Zeroization of sensitive data in Erlang In-Reply-To: References: Message-ID: With regard to other FP languages, Haskell has a crypto library called raaz that makes an effort to ensure that sensitive memory gets cleared eagerly. Other Haskell libraries might have similar functionality, it's a bit easier to do with GHC than with the Erlang VM for various reasons. http://hackage.haskell.org/package/raaz-0.2.1/docs/Raaz-Core-Types.html#v:allocaSecureAligned On Sat, Oct 26, 2019 at 4:02 PM Amit K wrote: > Hi Daniel, > > Good to know about that workaround, thank you. However as you already > pointed out, I really do want to be able to handle such "sensitive" > variables in Erlang code and use the standard "crypto" library of the OTP. > > With regards to GC - that's a great direction I think, and perhaps a more > finer grained approach would be to add a new "sensitive" flag for typed > variables that the GC can recognize. Then only those variables get zeroized > when the GC runs. > Would probably be better for performance than always zeroizing everything > that we free... > > BTW - so far I didn't find any FP language that can do something like > that, and please correct me if I'm wrong. It would be cool if Erlang would > be the first to the gold here (again!) :) > > On Sun, Oct 27, 2019 at 1:42 AM D?niel Szoboszlay > wrote: > >> Hi Amit, >> >> A NIF could use a resource object to implement a zeroable memory >> location. The Erlang program will only see a handle to the resource, which >> is immutable. But the contents of the object can be changed by the NIF >> (it's better explained in the docs >> ). So you could create an API >> that looks like this for the Erlang program: >> >> Handle = create_secret(Some, Input, Values), >> ok = use_secret(Handle), >> ok = destroy_secret(Handle), % The NIF zeroes out the memory in the >> resource object >> {error, destroyed} = use_secret(Handle). % The secret is no more at this >> point >> >> The big problem is that your secret shall never leave the native library. >> E.g. you cannot hide a cryptographic key behind a zeroable resource object >> and then pass it to the crypto library. As soon as you convert it into an >> Erlang term, it is copied onto the non-zeroed stack or heap or a process. >> Considering this, you may just as well use a port program to hold and work >> with the secret outside of the Erlang VM. >> >> Probably the only real solution within the VM would be to patch the GC to >> zero any reclaimed memory. Or maybe not even the GC, but the memory >> allocators, in case a secret is copied to some non-GC-d memory area, such >> as an ETS table. >> >> Cheers, >> Daniel >> >> On Sat, 26 Oct 2019 at 22:39, Amit K wrote: >> >>> Hi all, >>> >>> Concerning the security practice of Zeroizing sensitive data from memory >>> after you're finished with it, such as Cryptographic keys, passwords, etc, >>> I wanted to ask if any of you ever had to implement such a thing in Erlang, >>> and if so, how? :) >>> >>> Also note that often this is a requirement that you must fulfill if you >>> want your product to pass a security standard evaluation such as the >>> "Common Criteria" one (As an example requirement see FCS_CKM_EXT.4 here: >>> https://www.commoncriteriaportal.org/files/ppfiles/pp_nd_v1.0.pdf). >>> >>> The problem of course is that in Erlang (and I suppose - FP in general) >>> once a variable gets assigned, you are not allowed to change its value, and >>> in particular of course overwrite it with zero bytes. >>> >>> One option I thought about is doing it with a NIF, but from my >>> understanding a NIF isn't supposed to change its input values, it should >>> treat them as constants, which understandable as well (basic FP language >>> property). >>> >>> Any feedback can be helpful :) >>> >>> Regards, >>> >>> Amit >>> >>> >>> >>> >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From klg.amit@REDACTED Sun Oct 27 02:33:47 2019 From: klg.amit@REDACTED (Amit K) Date: Sun, 27 Oct 2019 02:33:47 +0200 Subject: Zeroization of sensitive data in Erlang In-Reply-To: References: Message-ID: Thank you Bob, Note though that the "gotcha" in the documentation that says: "...there can be strange situations in multi-threaded application where the memory is not wiped out..." is why I think ideally something like that should be built into the language and not provided as an external package, which at best will work only as long the run-time obeys certain rules which are bound to change over time, and at worst will work "most of the time" :) (as this is not acceptable in real security applications that need to go through security certifications). On Sun, Oct 27, 2019 at 2:16 AM Bob Ippolito wrote: > With regard to other FP languages, Haskell has a crypto library called > raaz that makes an effort to ensure that sensitive memory gets cleared > eagerly. Other Haskell libraries might have similar functionality, it's a > bit easier to do with GHC than with the Erlang VM for various reasons. > > > http://hackage.haskell.org/package/raaz-0.2.1/docs/Raaz-Core-Types.html#v:allocaSecureAligned > > On Sat, Oct 26, 2019 at 4:02 PM Amit K wrote: > >> Hi Daniel, >> >> Good to know about that workaround, thank you. However as you already >> pointed out, I really do want to be able to handle such "sensitive" >> variables in Erlang code and use the standard "crypto" library of the OTP. >> >> With regards to GC - that's a great direction I think, and perhaps a more >> finer grained approach would be to add a new "sensitive" flag for typed >> variables that the GC can recognize. Then only those variables get zeroized >> when the GC runs. >> Would probably be better for performance than always zeroizing everything >> that we free... >> >> BTW - so far I didn't find any FP language that can do something like >> that, and please correct me if I'm wrong. It would be cool if Erlang would >> be the first to the gold here (again!) :) >> >> On Sun, Oct 27, 2019 at 1:42 AM D?niel Szoboszlay >> wrote: >> >>> Hi Amit, >>> >>> A NIF could use a resource object to implement a zeroable memory >>> location. The Erlang program will only see a handle to the resource, which >>> is immutable. But the contents of the object can be changed by the NIF >>> (it's better explained in the docs >>> ). So you could create an API >>> that looks like this for the Erlang program: >>> >>> Handle = create_secret(Some, Input, Values), >>> ok = use_secret(Handle), >>> ok = destroy_secret(Handle), % The NIF zeroes out the memory in the >>> resource object >>> {error, destroyed} = use_secret(Handle). % The secret is no more at this >>> point >>> >>> The big problem is that your secret shall never leave the native >>> library. E.g. you cannot hide a cryptographic key behind a zeroable >>> resource object and then pass it to the crypto library. As soon as you >>> convert it into an Erlang term, it is copied onto the non-zeroed stack or >>> heap or a process. Considering this, you may just as well use a port >>> program to hold and work with the secret outside of the Erlang VM. >>> >>> Probably the only real solution within the VM would be to patch the GC >>> to zero any reclaimed memory. Or maybe not even the GC, but the memory >>> allocators, in case a secret is copied to some non-GC-d memory area, such >>> as an ETS table. >>> >>> Cheers, >>> Daniel >>> >>> On Sat, 26 Oct 2019 at 22:39, Amit K wrote: >>> >>>> Hi all, >>>> >>>> Concerning the security practice of Zeroizing sensitive data from >>>> memory after you're finished with it, such as Cryptographic keys, >>>> passwords, etc, I wanted to ask if any of you ever had to implement such a >>>> thing in Erlang, and if so, how? :) >>>> >>>> Also note that often this is a requirement that you must fulfill if you >>>> want your product to pass a security standard evaluation such as the >>>> "Common Criteria" one (As an example requirement see FCS_CKM_EXT.4 here: >>>> https://www.commoncriteriaportal.org/files/ppfiles/pp_nd_v1.0.pdf). >>>> >>>> The problem of course is that in Erlang (and I suppose - FP in general) >>>> once a variable gets assigned, you are not allowed to change its value, and >>>> in particular of course overwrite it with zero bytes. >>>> >>>> One option I thought about is doing it with a NIF, but from my >>>> understanding a NIF isn't supposed to change its input values, it should >>>> treat them as constants, which understandable as well (basic FP language >>>> property). >>>> >>>> Any feedback can be helpful :) >>>> >>>> Regards, >>>> >>>> Amit >>>> >>>> >>>> >>>> >>>> >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Sun Oct 27 02:24:41 2019 From: bob@REDACTED (Bob Ippolito) Date: Sat, 26 Oct 2019 18:24:41 -0700 Subject: Zeroization of sensitive data in Erlang In-Reply-To: References: Message-ID: Sure, that?s why I said it makes an effort. Most languages with threads (or even certain uses of signals) can have similar edge cases. Having the edge cases documented such that they can be avoided is likely the best you?re going to find anywhere. On Sat, Oct 26, 2019 at 17:34 Amit K wrote: > Thank you Bob, > Note though that the "gotcha" in the documentation that says: "...there > can be strange situations in multi-threaded application where the memory is > not wiped out..." is why I think ideally something like that should be > built into the language and not provided as an external package, which at > best will work only as long the run-time obeys certain rules which are > bound to change over time, and at worst will work "most of the time" :) (as > this is not acceptable in real security applications that need to go > through security certifications). > > On Sun, Oct 27, 2019 at 2:16 AM Bob Ippolito wrote: > >> With regard to other FP languages, Haskell has a crypto library called >> raaz that makes an effort to ensure that sensitive memory gets cleared >> eagerly. Other Haskell libraries might have similar functionality, it's a >> bit easier to do with GHC than with the Erlang VM for various reasons. >> >> >> http://hackage.haskell.org/package/raaz-0.2.1/docs/Raaz-Core-Types.html#v:allocaSecureAligned >> >> On Sat, Oct 26, 2019 at 4:02 PM Amit K wrote: >> >>> Hi Daniel, >>> >>> Good to know about that workaround, thank you. However as you already >>> pointed out, I really do want to be able to handle such "sensitive" >>> variables in Erlang code and use the standard "crypto" library of the OTP. >>> >>> With regards to GC - that's a great direction I think, and perhaps a >>> more finer grained approach would be to add a new "sensitive" flag for >>> typed variables that the GC can recognize. Then only those variables get >>> zeroized when the GC runs. >>> Would probably be better for performance than always zeroizing >>> everything that we free... >>> >>> BTW - so far I didn't find any FP language that can do something like >>> that, and please correct me if I'm wrong. It would be cool if Erlang would >>> be the first to the gold here (again!) :) >>> >>> On Sun, Oct 27, 2019 at 1:42 AM D?niel Szoboszlay >>> wrote: >>> >>>> Hi Amit, >>>> >>>> A NIF could use a resource object to implement a zeroable memory >>>> location. The Erlang program will only see a handle to the resource, which >>>> is immutable. But the contents of the object can be changed by the NIF >>>> (it's better explained in the docs >>>> ). So you could create an API >>>> that looks like this for the Erlang program: >>>> >>>> Handle = create_secret(Some, Input, Values), >>>> ok = use_secret(Handle), >>>> ok = destroy_secret(Handle), % The NIF zeroes out the memory in the >>>> resource object >>>> {error, destroyed} = use_secret(Handle). % The secret is no more at >>>> this point >>>> >>>> The big problem is that your secret shall never leave the native >>>> library. E.g. you cannot hide a cryptographic key behind a zeroable >>>> resource object and then pass it to the crypto library. As soon as you >>>> convert it into an Erlang term, it is copied onto the non-zeroed stack or >>>> heap or a process. Considering this, you may just as well use a port >>>> program to hold and work with the secret outside of the Erlang VM. >>>> >>>> Probably the only real solution within the VM would be to patch the GC >>>> to zero any reclaimed memory. Or maybe not even the GC, but the memory >>>> allocators, in case a secret is copied to some non-GC-d memory area, such >>>> as an ETS table. >>>> >>>> Cheers, >>>> Daniel >>>> >>>> On Sat, 26 Oct 2019 at 22:39, Amit K wrote: >>>> >>>>> Hi all, >>>>> >>>>> Concerning the security practice of Zeroizing sensitive data from >>>>> memory after you're finished with it, such as Cryptographic keys, >>>>> passwords, etc, I wanted to ask if any of you ever had to implement such a >>>>> thing in Erlang, and if so, how? :) >>>>> >>>>> Also note that often this is a requirement that you must fulfill if >>>>> you want your product to pass a security standard evaluation such as the >>>>> "Common Criteria" one (As an example requirement see FCS_CKM_EXT.4 here: >>>>> https://www.commoncriteriaportal.org/files/ppfiles/pp_nd_v1.0.pdf). >>>>> >>>>> The problem of course is that in Erlang (and I suppose - FP in >>>>> general) once a variable gets assigned, you are not allowed to change its >>>>> value, and in particular of course overwrite it with zero bytes. >>>>> >>>>> One option I thought about is doing it with a NIF, but from my >>>>> understanding a NIF isn't supposed to change its input values, it should >>>>> treat them as constants, which understandable as well (basic FP language >>>>> property). >>>>> >>>>> Any feedback can be helpful :) >>>>> >>>>> Regards, >>>>> >>>>> Amit >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.platt@REDACTED Sun Oct 27 07:25:21 2019 From: pablo.platt@REDACTED (pablo platt) Date: Sun, 27 Oct 2019 08:25:21 +0200 Subject: DTLS unexpected message certificate_verify Message-ID: Sometimes I'm getting the following error in a DTLS server running OTP 22.0.4. ssl/src/ssl_connection.erl:1323 generated SERVER ALERT: Fatal - Unexpected Message - {unexpected_msg,{internal,{certificate_verify,{sha256,rsa},<<158,...>>}}} Is it a bug in the ssl app or something I'm missing in my verify_fun? Should I just ignore the error? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pailleau@REDACTED Sun Oct 27 08:13:41 2019 From: eric.pailleau@REDACTED (Eric Pailleau) Date: Sun, 27 Oct 2019 08:13:41 +0100 Subject: Zeroization of sensitive data in Erlang In-Reply-To: References: Message-ID: <774ufvduq149fiqk9dl27c3h.1572160279411@email.android.com> Hi, Note that there is a sensitive flag for processes. Would be neat that variables used by sensitive processes get their memory zeroed. Envoy? depuis mon mobile ---- Amit K a ?crit ---- >Hi Daniel, > >Good to know about that workaround, thank you. However as you already >pointed out, I really do want to be able to handle such "sensitive" >variables in Erlang code and use the standard "crypto" library of the OTP. > >With regards to GC - that's a great direction I think, and perhaps a more >finer grained approach would be to add a new "sensitive" flag for typed >variables that the GC can recognize. Then only those variables get zeroized >when the GC runs. >Would probably be better for performance than always zeroizing everything >that we free... > >BTW - so far I didn't find any FP language that can do something like that, >and please correct me if I'm wrong. It would be cool if Erlang would be the >first to the gold here (again!) :) > >On Sun, Oct 27, 2019 at 1:42 AM D?niel Szoboszlay >wrote: > >> Hi Amit, >> >> A NIF could use a resource object to implement a zeroable memory location. >> The Erlang program will only see a handle to the resource, which is >> immutable. But the contents of the object can be changed by the NIF (it's >> better explained in the docs ). >> So you could create an API that looks like this for the Erlang program: >> >> Handle = create_secret(Some, Input, Values), >> ok = use_secret(Handle), >> ok = destroy_secret(Handle), % The NIF zeroes out the memory in the >> resource object >> {error, destroyed} = use_secret(Handle). % The secret is no more at this >> point >> >> The big problem is that your secret shall never leave the native library. >> E.g. you cannot hide a cryptographic key behind a zeroable resource object >> and then pass it to the crypto library. As soon as you convert it into an >> Erlang term, it is copied onto the non-zeroed stack or heap or a process. >> Considering this, you may just as well use a port program to hold and work >> with the secret outside of the Erlang VM. >> >> Probably the only real solution within the VM would be to patch the GC to >> zero any reclaimed memory. Or maybe not even the GC, but the memory >> allocators, in case a secret is copied to some non-GC-d memory area, such >> as an ETS table. >> >> Cheers, >> Daniel >> >> On Sat, 26 Oct 2019 at 22:39, Amit K wrote: >> >>> Hi all, >>> >>> Concerning the security practice of Zeroizing sensitive data from memory >>> after you're finished with it, such as Cryptographic keys, passwords, etc, >>> I wanted to ask if any of you ever had to implement such a thing in Erlang, >>> and if so, how? :) >>> >>> Also note that often this is a requirement that you must fulfill if you >>> want your product to pass a security standard evaluation such as the >>> "Common Criteria" one (As an example requirement see FCS_CKM_EXT.4 here: >>> https://www.commoncriteriaportal.org/files/ppfiles/pp_nd_v1.0.pdf). >>> >>> The problem of course is that in Erlang (and I suppose - FP in general) >>> once a variable gets assigned, you are not allowed to change its value, and >>> in particular of course overwrite it with zero bytes. >>> >>> One option I thought about is doing it with a NIF, but from my >>> understanding a NIF isn't supposed to change its input values, it should >>> treat them as constants, which understandable as well (basic FP language >>> property). >>> >>> Any feedback can be helpful :) >>> >>> Regards, >>> >>> Amit >>> >>> >>> >>> >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From v@REDACTED Sun Oct 27 13:16:11 2019 From: v@REDACTED (Valentin Micic) Date: Sun, 27 Oct 2019 14:16:11 +0200 Subject: Binary, List and Tuple Inequalities (Paradox?) In-Reply-To: References: <46D4876E-C091-4A16-9A2B-D91DA65986AE@micic.co.za> Message-ID: <13271C3C-DC77-443B-A472-C555B213B2FD@micic.co.za> Thanks for the replies? unsatisfactory as they me have been :-) Kind regards V/ > On 25 Oct 2019, at 17:43, Fred Hebert wrote: > > > > On Fri, Oct 25, 2019 at 11:35 AM Dmitry Belyaev > wrote: > To me alphabetical ordering of lists (as legacy string representation) and binaries (as modern string representation) is natural, so it is expected that "aa" < "b" < "bb". > > As a general note, this is only true to the extent that you don't care that "ZOOM" is < than "apple" in this dictionary. It is anything but natural, I would say it's probably more "accidental". Actual string sorting tends to rely on a Unicode collation algorithm (a part of the spec that is still missing in the stdlib), since even string normalization won't be enough to ensure things are done safely with regards to human expectations. At this point only libraries such as ux support it well. > > Most of the orderings we have right now are often arbitrary and can have surprising sides to them. At this point for most data structures the important thing is not that the ordering is correct, but that at least one exists at all; this allows the creation of a bunch of data structures (usually trees and variants) to work on all data types, mixed or not. All atoms are smaller than all strings, even if atoms have a string component to them, for example. -------------- next part -------------- An HTML attachment was scrubbed... URL: From klg.amit@REDACTED Sun Oct 27 17:34:20 2019 From: klg.amit@REDACTED (Amit K) Date: Sun, 27 Oct 2019 18:34:20 +0200 Subject: Zeroization of sensitive data in Erlang In-Reply-To: <774ufvduq149fiqk9dl27c3h.1572160279411@email.android.com> References: <774ufvduq149fiqk9dl27c3h.1572160279411@email.android.com> Message-ID: Hi Eric, I like your suggestion, but would it be a reasonable performance overhead, especially for those who are already using the sensitive flag for certain processes? If the answer is likely yes I may well try to implement this and possibly create a PR for that :) On Sun, Oct 27, 2019 at 9:21 AM Eric Pailleau wrote: > Hi, > > Note that there is a sensitive flag for processes. > > Would be neat that variables used by sensitive processes get their memory > zeroed. > > Envoy? depuis mon mobile > > > ---- Amit K a ?crit ---- > > Hi Daniel, > > Good to know about that workaround, thank you. However as you already > pointed out, I really do want to be able to handle such "sensitive" > variables in Erlang code and use the standard "crypto" library of the OTP. > > With regards to GC - that's a great direction I think, and perhaps a more > finer grained approach would be to add a new "sensitive" flag for typed > variables that the GC can recognize. Then only those variables get zeroized > when the GC runs. > Would probably be better for performance than always zeroizing everything > that we free... > > BTW - so far I didn't find any FP language that can do something like > that, and please correct me if I'm wrong. It would be cool if Erlang would > be the first to the gold here (again!) :) > > On Sun, Oct 27, 2019 at 1:42 AM D?niel Szoboszlay > wrote: > >> Hi Amit, >> >> A NIF could use a resource object to implement a zeroable memory >> location. The Erlang program will only see a handle to the resource, which >> is immutable. But the contents of the object can be changed by the NIF >> (it's better explained in the docs >> ). So you could create an API >> that looks like this for the Erlang program: >> >> Handle = create_secret(Some, Input, Values), >> ok = use_secret(Handle), >> ok = destroy_secret(Handle), % The NIF zeroes out the memory in the >> resource object >> {error, destroyed} = use_secret(Handle). % The secret is no more at this >> point >> >> The big problem is that your secret shall never leave the native library. >> E.g. you cannot hide a cryptographic key behind a zeroable resource object >> and then pass it to the crypto library. As soon as you convert it into an >> Erlang term, it is copied onto the non-zeroed stack or heap or a process. >> Considering this, you may just as well use a port program to hold and work >> with the secret outside of the Erlang VM. >> >> Probably the only real solution within the VM would be to patch the GC to >> zero any reclaimed memory. Or maybe not even the GC, but the memory >> allocators, in case a secret is copied to some non-GC-d memory area, such >> as an ETS table. >> >> Cheers, >> Daniel >> >> On Sat, 26 Oct 2019 at 22:39, Amit K wrote: >> >>> Hi all, >>> >>> Concerning the security practice of Zeroizing sensitive data from memory >>> after you're finished with it, such as Cryptographic keys, passwords, etc, >>> I wanted to ask if any of you ever had to implement such a thing in Erlang, >>> and if so, how? :) >>> >>> Also note that often this is a requirement that you must fulfill if you >>> want your product to pass a security standard evaluation such as the >>> "Common Criteria" one (As an example requirement see FCS_CKM_EXT.4 here: >>> https://www.commoncriteriaportal.org/files/ppfiles/pp_nd_v1.0.pdf). >>> >>> The problem of course is that in Erlang (and I suppose - FP in general) >>> once a variable gets assigned, you are not allowed to change its value, and >>> in particular of course overwrite it with zero bytes. >>> >>> One option I thought about is doing it with a NIF, but from my >>> understanding a NIF isn't supposed to change its input values, it should >>> treat them as constants, which understandable as well (basic FP language >>> property). >>> >>> Any feedback can be helpful :) >>> >>> Regards, >>> >>> Amit >>> >>> >>> >>> >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From thp+erlang@REDACTED Sun Oct 27 20:19:32 2019 From: thp+erlang@REDACTED (Thomas Pircher) Date: Sun, 27 Oct 2019 19:19:32 +0000 Subject: [erlang-questions] Matching IP address in socket module In-Reply-To: <20191024153351.GB28388@tpccl.tty1.net> References: <20191024100444.GA28388@tpccl.tty1.net> <20191024153351.GB28388@tpccl.tty1.net> Message-ID: <20191027191932.GA32310@tpccl.tty1.net> Thomas Pircher wrote: > I'm quite impressed with the socket module, it seems to be pretty > complete, at least for my application. Hi, I'm having another problem with the socket module, I'm hoping someone could help me understanding what is going on here. I'm attempting to send a packet from an arbitrary IP address. My understanding is that this can be done using the pktinfo structure on a Linux system. There is no direct support for this in the cmsghdr_send() type, so I thought I can use the #{level := ip, type := integer(), ?} construct instead. (I know this is non-portable.) Using the following code snippet I can get packets out of the system, but the pktinfo data seems to be ignored; the output interface is selected by the routing table (and not by the IfIndex variable) and the source IP address is not the one I set in the control header. -define(IP_PKTINFO, 8). send(DstIp, IfIndex, Data) -> {ok, Socket} = socket:open(inet, dgram), DstAddr = #{family => inet, addr => DstIp, port => 42001}, Ctrl = #{level => udp, type => ?IP_PKTINFO, data => <>}, MsgHdr = #{addr => DstAddr, ctrl => [Ctrl], flags => [], iov => [Data]}, Res = socket:sendmsg(Socket, MsgHdr), io:format("Sent message ~p~n", [Res]). Is there something I'm doing wrong? Thanks, Thomas From dszoboszlay@REDACTED Sun Oct 27 23:42:42 2019 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Sun, 27 Oct 2019 23:42:42 +0100 Subject: Zeroization of sensitive data in Erlang In-Reply-To: References: <774ufvduq149fiqk9dl27c3h.1572160279411@email.android.com> Message-ID: Hi Amit, Regarding the sensitive flag: - I like the idea of only zeroing memory of sensitive processes, it sounds like a sensible thing. - However, beware that this flag is not used widely enough. For example, I found no usage of it in the OTP source code (except for some test cases verifying it actually works), although applications like ssl could probably make good use of it. Similarly, on GitHub there are only 313 hits if you search for the flag (with a lot of false positives). So you'd definitely have to audit your libraries, such as your webserver, on whether they mark all processes with sensitive data as sensitive. - For the above reason, I wouldn't worry that much about performance impact on existing applications. But just to be on the safe side, you could add a command line switch for turning the new feature on, leaving it off by default. - What you should worry about are things like shared, reference-counted binaries. They may be shared between sensitive and non-sensitive processes, so do they have to be zeroed or not? What about the memory allocated by NIF-s in sensitive processes? Like the openssl data structures where crypto will copy the keys? I have a feeling that blindly zeroing every freed up memory is inefficient, but doable, while extending the sensitive concept to cover every aspect of the BEAM VM may not be feasible at all. Regarding more fine grained control via marking terms as sensitive instead of entire processes: I don't think this would be a good idea in practice. - The BEAM VM reserves a few low bits in each word of memory for type tags . To mark any term as sensitive, you'd need an unused tag bit, which you don't have. There is one unused tag bit combination for boxed types left, so you could in theory introduce a new, "sensitive data" term type, but that's not feasible to implement. For boxed types on x64, you could also use the top 16 bits of the pointer for meta-data, as currently the physical addresses are only 48 bit wide on this platform. But this also seems unreasonably complicated and isn't even portable to other architectures. - Even if you could tag each sensitive term, this wouldn't make the GC faster. The GC only visits (and copies) live terms. I'm pretty sure that reading in every dead term, checking whether they're sensitive and then selectively zeroing parts of the memory would be much-much slower in practice than just blindly zeroing over the entire memory area. We are talking about out of order CPU-s with precious, limited cache space here. - What may work instead is to maintain a list of sensitive terms on the heap for each process. This is probably not impossible to do (e.g. processes already maintain a list of shared, reference-counted binaries they use), but it seems impractically complicated. Daniel On Sun, 27 Oct 2019 at 17:34, Amit K wrote: > Hi Eric, > I like your suggestion, but would it be a reasonable performance overhead, > especially for those who are already using the sensitive flag for certain > processes? If the answer is likely yes I may well try to implement this and > possibly create a PR for that :) > > On Sun, Oct 27, 2019 at 9:21 AM Eric Pailleau > wrote: > >> Hi, >> >> Note that there is a sensitive flag for processes. >> >> Would be neat that variables used by sensitive processes get their memory >> zeroed. >> >> Envoy? depuis mon mobile >> >> >> ---- Amit K a ?crit ---- >> >> Hi Daniel, >> >> Good to know about that workaround, thank you. However as you already >> pointed out, I really do want to be able to handle such "sensitive" >> variables in Erlang code and use the standard "crypto" library of the OTP. >> >> With regards to GC - that's a great direction I think, and perhaps a more >> finer grained approach would be to add a new "sensitive" flag for typed >> variables that the GC can recognize. Then only those variables get zeroized >> when the GC runs. >> Would probably be better for performance than always zeroizing everything >> that we free... >> >> BTW - so far I didn't find any FP language that can do something like >> that, and please correct me if I'm wrong. It would be cool if Erlang would >> be the first to the gold here (again!) :) >> >> On Sun, Oct 27, 2019 at 1:42 AM D?niel Szoboszlay >> wrote: >> >>> Hi Amit, >>> >>> A NIF could use a resource object to implement a zeroable memory >>> location. The Erlang program will only see a handle to the resource, which >>> is immutable. But the contents of the object can be changed by the NIF >>> (it's better explained in the docs >>> ). So you could create an API >>> that looks like this for the Erlang program: >>> >>> Handle = create_secret(Some, Input, Values), >>> ok = use_secret(Handle), >>> ok = destroy_secret(Handle), % The NIF zeroes out the memory in the >>> resource object >>> {error, destroyed} = use_secret(Handle). % The secret is no more at this >>> point >>> >>> The big problem is that your secret shall never leave the native >>> library. E.g. you cannot hide a cryptographic key behind a zeroable >>> resource object and then pass it to the crypto library. As soon as you >>> convert it into an Erlang term, it is copied onto the non-zeroed stack or >>> heap or a process. Considering this, you may just as well use a port >>> program to hold and work with the secret outside of the Erlang VM. >>> >>> Probably the only real solution within the VM would be to patch the GC >>> to zero any reclaimed memory. Or maybe not even the GC, but the memory >>> allocators, in case a secret is copied to some non-GC-d memory area, such >>> as an ETS table. >>> >>> Cheers, >>> Daniel >>> >>> On Sat, 26 Oct 2019 at 22:39, Amit K wrote: >>> >>>> Hi all, >>>> >>>> Concerning the security practice of Zeroizing sensitive data from >>>> memory after you're finished with it, such as Cryptographic keys, >>>> passwords, etc, I wanted to ask if any of you ever had to implement such a >>>> thing in Erlang, and if so, how? :) >>>> >>>> Also note that often this is a requirement that you must fulfill if you >>>> want your product to pass a security standard evaluation such as the >>>> "Common Criteria" one (As an example requirement see FCS_CKM_EXT.4 here: >>>> https://www.commoncriteriaportal.org/files/ppfiles/pp_nd_v1.0.pdf). >>>> >>>> The problem of course is that in Erlang (and I suppose - FP in general) >>>> once a variable gets assigned, you are not allowed to change its value, and >>>> in particular of course overwrite it with zero bytes. >>>> >>>> One option I thought about is doing it with a NIF, but from my >>>> understanding a NIF isn't supposed to change its input values, it should >>>> treat them as constants, which understandable as well (basic FP language >>>> property). >>>> >>>> Any feedback can be helpful :) >>>> >>>> Regards, >>>> >>>> Amit >>>> >>>> >>>> >>>> >>>> >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo.niskanen@REDACTED Mon Oct 28 09:02:13 2019 From: raimo.niskanen@REDACTED (Raimo Niskanen) Date: Mon, 28 Oct 2019 08:02:13 +0000 Subject: Binary, List and Tuple Inequalities (Paradox?) Message-ID: <180ae745ba2acccfadafebe278a45155a68ab461.camel@ericsson.com> I'd like to defend the current term order of lists, tuples and binaries. Lists are compared a'la string comparison, which avoids having to traverse the whole list when comparing. Binaries are compared as lists since both are used for handling strings, and also to be compared like memcmp does it. Tuples are used for general complex terms. They can be compared size first whithout having to be fully traversed, as opposed to lists, which is useful in other contexts. You can thereby choose your data structure to select comparision. -- Raimo Niskanen From zxq9@REDACTED Mon Oct 28 10:34:55 2019 From: zxq9@REDACTED (zxq9) Date: Mon, 28 Oct 2019 18:34:55 +0900 Subject: Binary, List and Tuple Inequalities (Paradox?) In-Reply-To: <180ae745ba2acccfadafebe278a45155a68ab461.camel@ericsson.com> References: <180ae745ba2acccfadafebe278a45155a68ab461.camel@ericsson.com> Message-ID: <2320873.sPjQOCVjmy@burrito> On 2019?10?28? ??? 08:02:13 Raimo Niskanen wrote: > Lists are compared a'la string comparison, which avoids having to > traverse the whole list when comparing. Binaries are compared as lists > since both are used for handling strings, and also to be compared like > memcmp does it. > > Tuples are used for general complex terms. They can be compared size > first whithout having to be fully traversed, as opposed to lists, which > is useful in other contexts. You can thereby choose your data > structure to select comparision. Thanks for the explanation. Worth bookmarking for future reference. -Craig From micael.karlberg@REDACTED Mon Oct 28 12:28:08 2019 From: micael.karlberg@REDACTED (Micael Karlberg) Date: Mon, 28 Oct 2019 12:28:08 +0100 Subject: Patch Package OTP 22.1.5 Released Message-ID: <50848680-574d-61d1-f291-b5374c7a404b@ericsson.com> Patch Package: OTP 22.1.5 Git Tag: OTP-22.1.5 Date: 2019-10-28 Trouble Report Id: OTP-16207 Seq num: ERIERL-427 System: OTP Release: 22 Application: snmp-5.4.2 Predecessor: OTP 22.1.4 Check out the git tag OTP-22.1.5, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- snmp-5.4.2 ------------------------------------------------------ --------------------------------------------------------------------- The snmp-5.4.2 application can be applied independently of other applications on a full OTP 22 installation. --- Fixed Bugs and Malfunctions --- OTP-16207 Application(s): snmp Related Id(s): ERIERL-427 The agent discovery process has been made to work with snmptrapd.. Full runtime dependencies of snmp-5.4.2: crypto-3.3, erts-6.0, kernel-3.0, mnesia-4.12, runtime_tools-1.8.14, stdlib-2.5 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From frank.muller.erl@REDACTED Mon Oct 28 13:03:59 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Mon, 28 Oct 2019 13:03:59 +0100 Subject: Send VM Metrics directly to Grafana Message-ID: Hi guys While deploying one of our Erlang products in a highly secure environment (private bank), this partner required from us to export the VM metrics to Grafana. I know that a Prometheus Erlang client do exist, but our partner refused to deploy Prometheus (didn?t pass their pentesting phase). Question: is there a way to send the VM metrics directly to Grafana? Any other alternative will be welcomed. /Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerhard@REDACTED Mon Oct 28 13:21:48 2019 From: gerhard@REDACTED (Gerhard Lazu) Date: Mon, 28 Oct 2019 12:21:48 +0000 Subject: Send VM Metrics directly to Grafana In-Reply-To: References: Message-ID: Hi, Question: is there a way to send the VM metrics directly to Grafana? No. You need to use one of the supported data sources (Prometheus is just one of them): https://grafana.com/docs/features/datasources/ Would Pushgateway work in your case? https://prometheus.io/docs/practices/pushing/ Influxdb would be my next choice: https://grafana.com/docs/features/datasources/influxdb/ Linking just in case you haven't seen https://opentelemetry.io/about/ & https://opencensus.io/exporters/. Notice that Datadog accepts both metrics & traces. Maybe that would be a low-friction alternative. On Mon, Oct 28, 2019 at 12:04 PM Frank Muller wrote: > Hi guys > > While deploying one of our Erlang products in a highly secure environment > (private bank), this partner required from us to export the VM metrics to > Grafana. > > I know that a Prometheus Erlang client do exist, but our partner refused > to deploy Prometheus (didn?t pass their pentesting phase). > > Any other alternative will be welcomed. > > /Frank > -------------- next part -------------- An HTML attachment was scrubbed... URL: From klg.amit@REDACTED Mon Oct 28 13:23:42 2019 From: klg.amit@REDACTED (Amit K) Date: Mon, 28 Oct 2019 14:23:42 +0200 Subject: Zeroization of sensitive data in Erlang In-Reply-To: References: <774ufvduq149fiqk9dl27c3h.1572160279411@email.android.com> Message-ID: Hi Daniel, Thanks very much for your thoughts and insights. Regarding your following comment about refc binaries: > What you should worry about are things like shared, reference-counted binaries. They may be shared between sensitive and non-sensitive processes, so do they have to be zeroed or not? What about the memory allocated by NIF-s in sensitive processes? Like the openssl data structures where crypto will copy the keys? I have a feeling that blindly zeroing every freed up memory is inefficient, but doable, while extending the sensitive concept to cover every aspect of the BEAM VM may not be feasible at all. I read up on this a little bit here: http://erlang.org/doc/efficiency_guide/binaryhandling.html and it seems that even when the binary is not shared it can reside on the global binary heap - either because it is larger than 64 bytes, or because it was referenced from another term, even if it's in the same process. The latter seems to be easily avoidable if you carefully write your sensitive data handling code, and the same goes for shared binaries - if you're really writing code that handles sensitive data you need to be careful about which other processes you share it with anyway :) The size limitation of 64 bytes however is a bit more painful, although actually, for symmetric crypto keys and passwords 64 bytes is very reasonable because AES-256 keys for example are only 32 bytes long. Most passwords are also much shorter than 64 bytes. However of course for asymmetric crypto the key sizes are much larger than 64 bytes and this becomes a real issue. I have two ideas on how to address this: 1. Add a "sensitive" flag for the entire Erlang node as well, which at least to begin with, will only tell the GC to also zero the binary heap when it frees it. More relevant actions can be piled on top of that semantic in the future. 2. The second would be to allow larger binaries to reside on the process heap of sensitive processes, possibly even make this configurable. This sounds to me a bit more complex than 1. One final point about NIFs - I think that's outside the scope of what is possible to address. The use case I really want to address is the handling of sensitive data in Erlang code and the passing of it to and from the "crypto" module. Granted, OpenSSL is a C program and hence a NIF (hope I got that right) but if OpenSSL fails to zero its internal sensitive buffers, that's a bug that should be handled in the OpenSSL context :) (BTW: after a bit of searching I found that OpenSSL uses "openssl_cleanse" whenever it needs to zero a sensitive buffer - https://github.com/openssl/openssl/search?q=openssl_cleanse&unscoped_q=openssl_cleanse) Thanks, Amit From frank.muller.erl@REDACTED Mon Oct 28 13:35:52 2019 From: frank.muller.erl@REDACTED (Frank Muller) Date: Mon, 28 Oct 2019 13:35:52 +0100 Subject: Send VM Metrics directly to Grafana In-Reply-To: References: Message-ID: Thanks, I'll check them out. Le lun. 28 oct. 2019 ? 13:22, Gerhard Lazu a ?crit : > Hi, > > Question: is there a way to send the VM metrics directly to Grafana? > > > No. You need to use one of the supported data sources (Prometheus is just > one of them): https://grafana.com/docs/features/datasources/ > > Would Pushgateway work in your case? > https://prometheus.io/docs/practices/pushing/ > > Influxdb would be my next choice: > https://grafana.com/docs/features/datasources/influxdb/ > > Linking just in case you haven't seen https://opentelemetry.io/about/ & > https://opencensus.io/exporters/. Notice that Datadog accepts both > metrics & traces. Maybe that would be a low-friction alternative. > > On Mon, Oct 28, 2019 at 12:04 PM Frank Muller > wrote: > >> Hi guys >> >> While deploying one of our Erlang products in a highly secure environment >> (private bank), this partner required from us to export the VM metrics to >> Grafana. >> >> I know that a Prometheus Erlang client do exist, but our partner refused >> to deploy Prometheus (didn?t pass their pentesting phase). >> >> Any other alternative will be welcomed. >> >> /Frank >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ebegumisa@REDACTED Mon Oct 28 14:15:54 2019 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Mon, 28 Oct 2019 23:15:54 +1000 Subject: Send VM Metrics directly to Grafana In-Reply-To: References: Message-ID: There's a useful little MIT-licensed plugin which allows Grafana to directly query JSON data sources over HTTP(S): https://grafana.com/grafana/plugins/simpod-json-datasource To use, you'd implement the plugin's simple API on the Erlang side. https://github.com/simPod/grafana-json-datasource#api If for some reason that would fail pen-testing, you could use that code as an example of how to implement your own Grafana datasource. But that's probably overkill. - Edmond - On Mon, 28 Oct 2019 22:35:52 +1000, Frank Muller wrote: > Thanks, I'll check them out. > > Le lun. 28 oct. 2019 ? 13:22, Gerhard Lazu a ?crit : >> Hi, >> >>> Question: is there a way to send the VM metrics directly to Grafana? >> >> No. You need to use one of the supported data sources (Prometheus is >> just one of them): https://>>grafana.com/docs/features/datasources/ >> >> Would Pushgateway work in your case? >> https://prometheus.io/docs/practices/pushing/ >> >> Influxdb would be my next choice: >> https://grafana.com/docs/features/datasources/influxdb/ >> >> Linking just in case you haven't seen https://opentelemetry.io/about/ & >> https://opencensus.io/>>exporters/. Notice that Datadog accepts both >> metrics & traces. Maybe that would be a low-friction >>alternative. >> >> On Mon, Oct 28, 2019 at 12:04 PM Frank Muller >> wrote: >>> Hi guys >>> >>> While deploying one of our Erlang products in a highly secure >>> environment (private bank), this >>>partner required from us to export >>> the VM metrics to Grafana. >>> I know that a Prometheus Erlang client do exist, but our partner >>> refused to deploy Prometheus >>>(didn?t pass their pentesting phase). >>> >>> Any other alternative will be welcomed. >>> /Frank >>> >> > -- Using Opera's mail client: http://www.opera.com/mail/ From kenneth@REDACTED Mon Oct 28 14:18:05 2019 From: kenneth@REDACTED (Kenneth Lundin) Date: Mon, 28 Oct 2019 14:18:05 +0100 Subject: Zeroization of sensitive data in Erlang In-Reply-To: References: <774ufvduq149fiqk9dl27c3h.1572160279411@email.android.com> Message-ID: The OpenSSL engine concept can be used to store cryptographical keys in a safe manner. The engine is a plugin for the OpenSSL libcrypto and the API for loading and using engines is supported in the crypto application in OTP. See http://erlang.org/doc/apps/crypto/engine_keys.html /Kenneth, Erlang/OTP Ericsson On Mon, Oct 28, 2019 at 1:24 PM Amit K wrote: > Hi Daniel, > > Thanks very much for your thoughts and insights. > Regarding your following comment about refc binaries: > > > What you should worry about are things like shared, reference-counted > binaries. They may be shared between sensitive and non-sensitive processes, > so do they have to be zeroed or not? What about the memory allocated by > NIF-s in sensitive processes? Like the openssl data structures where crypto > will copy the keys? I have a feeling that blindly zeroing every freed up > memory is inefficient, but doable, while extending the sensitive concept to > cover every aspect of the BEAM VM may not be feasible at all. > > I read up on this a little bit here: > http://erlang.org/doc/efficiency_guide/binaryhandling.html > and it seems that even when the binary is not shared it can reside on > the global binary heap - either because it is larger than 64 bytes, or > because it was referenced from another term, even if it's in the same > process. The latter seems to be easily avoidable if you carefully > write your sensitive data handling code, and the same goes for shared > binaries - if you're really writing code that handles sensitive data > you need to be careful about which other processes you share it with > anyway :) > The size limitation of 64 bytes however is a bit more painful, > although actually, for symmetric crypto keys and passwords 64 bytes is > very reasonable because AES-256 keys for example are only 32 bytes > long. Most passwords are also much shorter than 64 bytes. However of > course for asymmetric crypto the key sizes are much larger than 64 > bytes and this becomes a real issue. > > I have two ideas on how to address this: > > 1. Add a "sensitive" flag for the entire Erlang node as well, which at > least to begin with, will only tell the GC to also zero the binary > heap when it frees it. More relevant actions can be piled on top of > that semantic in the future. > > 2. The second would be to allow larger binaries to reside on the > process heap of sensitive processes, possibly even make this > configurable. This sounds to me a bit more complex than 1. > > One final point about NIFs - I think that's outside the scope of what > is possible to address. The use case I really want to address is the > handling of sensitive data in Erlang code and the passing of it to and > from the "crypto" module. Granted, OpenSSL is a C program and hence a > NIF (hope I got that right) but if OpenSSL fails to zero its internal > sensitive buffers, that's a bug that should be handled in the OpenSSL > context :) > > (BTW: after a bit of searching I found that OpenSSL uses > "openssl_cleanse" whenever it > needs to zero a sensitive buffer - > > https://github.com/openssl/openssl/search?q=openssl_cleanse&unscoped_q=openssl_cleanse > ) > > Thanks, > Amit > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Mon Oct 28 15:22:06 2019 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Mon, 28 Oct 2019 15:22:06 +0100 Subject: [erlang-questions] OTP 22.1 socket.erl somehow breaks message delivery or scheduler In-Reply-To: References: Message-ID: Thanks for reporting this issue. We have investigated the issue and found that it is a bug in the compiler. https://bugs.erlang.org/browse/ERL-1076 https://github.com/erlang/otp/pull/2434 /Bj?rn On Wed, Oct 23, 2019 at 3:53 PM Andreas Schultz wrote: > > Hi, > > After converting an application to socket.erl in OTP 22.1, the test suites started to fail with random timeouts. It took me a while to figure out that gen_server:calls arrived multiple seconds late in the server. > > I have a demonstration at https://gist.github.com/RoadRunnr/311a7679fff6fbdf367c455b960f1ba8. It implements a simple UDP echo server with socket.erl. The client uses gen_udp to send messages and wait for the response. > The client also sends Erlang ping message to the server and expects to get a pong answer back. The socket.erl based server is supposed to not block (and as far as I can tell, it does not), it therefore should be able to answer the Erlang ping message all the time. > There are also some simple busy loop process running to get some load. Without them the problem is not reproducible. > > The sample is failing in about 20% off the test runs, when it does the output is something like: > > $ ~/stest.escript > Server Pid <0.78.0> > Server Addr #{addr => {127,0,0,1},family => inet,port => 38959} > ping timeout > round trip Clnt/Srvr Srvr/Clnt ProcPing > ... > 85: ******** ns, ******** ns, 57675 ns, 42332 ns > ... > > The failure happens because a 'ping' message is not see in time by the receive clause in the server process. It seems that either the process is not scheduled for some time (multiple seconds), or the scanning of the mailbox is missing the message. > > I have ruled out that the UDP messages are being dropped, otherwise the clients gen_udp:recv would never return. > > Does anyone have a clue what might cause this? Or point out where my sample is broken. > > Many thanks > Andreas > > -- > > Andreas Schultz > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From klg.amit@REDACTED Mon Oct 28 16:11:37 2019 From: klg.amit@REDACTED (Amit K) Date: Mon, 28 Oct 2019 17:11:37 +0200 Subject: Zeroization of sensitive data in Erlang In-Reply-To: References: <774ufvduq149fiqk9dl27c3h.1572160279411@email.android.com> Message-ID: Hi Kenneth, This seems like a great option for integrating with something like HSM (pkcs11 in particular) devices. However I want to emphasize again, at the risk of being repetitive, that since the OTP provides the crypto module as an Erlang native way to perform crypto operations, that naturally involve sensitive data such as keys and passwords, the requirement that the OTP will also be able to securely wipe such sensitive data from memory seems very important to me. Regards, Amit On Mon, Oct 28, 2019, 3:18 PM Kenneth Lundin wrote: > The OpenSSL engine concept can be used to store cryptographical keys in a > safe manner. > The engine is a plugin for the OpenSSL libcrypto and the API for loading > and using engines is supported in the crypto application in OTP. > > See http://erlang.org/doc/apps/crypto/engine_keys.html > > /Kenneth, Erlang/OTP Ericsson > > On Mon, Oct 28, 2019 at 1:24 PM Amit K wrote: > >> Hi Daniel, >> >> Thanks very much for your thoughts and insights. >> Regarding your following comment about refc binaries: >> >> > What you should worry about are things like shared, reference-counted >> binaries. They may be shared between sensitive and non-sensitive processes, >> so do they have to be zeroed or not? What about the memory allocated by >> NIF-s in sensitive processes? Like the openssl data structures where crypto >> will copy the keys? I have a feeling that blindly zeroing every freed up >> memory is inefficient, but doable, while extending the sensitive concept to >> cover every aspect of the BEAM VM may not be feasible at all. >> >> I read up on this a little bit here: >> http://erlang.org/doc/efficiency_guide/binaryhandling.html >> and it seems that even when the binary is not shared it can reside on >> the global binary heap - either because it is larger than 64 bytes, or >> because it was referenced from another term, even if it's in the same >> process. The latter seems to be easily avoidable if you carefully >> write your sensitive data handling code, and the same goes for shared >> binaries - if you're really writing code that handles sensitive data >> you need to be careful about which other processes you share it with >> anyway :) >> The size limitation of 64 bytes however is a bit more painful, >> although actually, for symmetric crypto keys and passwords 64 bytes is >> very reasonable because AES-256 keys for example are only 32 bytes >> long. Most passwords are also much shorter than 64 bytes. However of >> course for asymmetric crypto the key sizes are much larger than 64 >> bytes and this becomes a real issue. >> >> I have two ideas on how to address this: >> >> 1. Add a "sensitive" flag for the entire Erlang node as well, which at >> least to begin with, will only tell the GC to also zero the binary >> heap when it frees it. More relevant actions can be piled on top of >> that semantic in the future. >> >> 2. The second would be to allow larger binaries to reside on the >> process heap of sensitive processes, possibly even make this >> configurable. This sounds to me a bit more complex than 1. >> >> One final point about NIFs - I think that's outside the scope of what >> is possible to address. The use case I really want to address is the >> handling of sensitive data in Erlang code and the passing of it to and >> from the "crypto" module. Granted, OpenSSL is a C program and hence a >> NIF (hope I got that right) but if OpenSSL fails to zero its internal >> sensitive buffers, that's a bug that should be handled in the OpenSSL >> context :) >> >> (BTW: after a bit of searching I found that OpenSSL uses >> "openssl_cleanse" whenever it >> needs to zero a sensitive buffer - >> >> https://github.com/openssl/openssl/search?q=openssl_cleanse&unscoped_q=openssl_cleanse >> ) >> >> Thanks, >> Amit >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.schultz@REDACTED Mon Oct 28 16:39:13 2019 From: andreas.schultz@REDACTED (Andreas Schultz) Date: Mon, 28 Oct 2019 16:39:13 +0100 Subject: [erlang-questions] OTP 22.1 socket.erl somehow breaks message delivery or scheduler In-Reply-To: References: Message-ID: Am Mo., 28. Okt. 2019 um 15:22 Uhr schrieb Bj?rn Gustavsson < bjorn@REDACTED>: > Thanks for reporting this issue. > > We have investigated the issue and found that it is a bug in the compiler. > wow. I have to say I'm a bit scared of OTP 22.x at the moment because of all the "bugs in the compiler" changes lately. Did the rewrite of the compiler in 22.0 cause all this or was it present before and just went unnoticed? Anyway, many thanks for finding this, Andreas > https://bugs.erlang.org/browse/ERL-1076 > > https://github.com/erlang/otp/pull/2434 > > /Bj?rn > > On Wed, Oct 23, 2019 at 3:53 PM Andreas Schultz > wrote: > > > > Hi, > > > > After converting an application to socket.erl in OTP 22.1, the test > suites started to fail with random timeouts. It took me a while to figure > out that gen_server:calls arrived multiple seconds late in the server. > > > > I have a demonstration at > https://gist.github.com/RoadRunnr/311a7679fff6fbdf367c455b960f1ba8. It > implements a simple UDP echo server with socket.erl. The client uses > gen_udp to send messages and wait for the response. > > The client also sends Erlang ping message to the server and expects to > get a pong answer back. The socket.erl based server is supposed to not > block (and as far as I can tell, it does not), it therefore should be able > to answer the Erlang ping message all the time. > > There are also some simple busy loop process running to get some load. > Without them the problem is not reproducible. > > > > The sample is failing in about 20% off the test runs, when it does the > output is something like: > > > > $ ~/stest.escript > > Server Pid <0.78.0> > > Server Addr #{addr => {127,0,0,1},family => inet,port => 38959} > > ping timeout > > round trip Clnt/Srvr Srvr/Clnt ProcPing > > ... > > 85: ******** ns, ******** ns, 57675 ns, 42332 ns > > ... > > > > The failure happens because a 'ping' message is not see in time by the > receive clause in the server process. It seems that either the process is > not scheduled for some time (multiple seconds), or the scanning of the > mailbox is missing the message. > > > > I have ruled out that the UDP messages are being dropped, otherwise the > clients gen_udp:recv would never return. > > > > Does anyone have a clue what might cause this? Or point out where my > sample is broken. > > > > Many thanks > > Andreas > > > > -- > > > > Andreas Schultz > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB > -- Andreas Schultz -- Principal Engineer t: +49 391 819099-224 ------------------------------- enabling your networks ----------------------------- Travelping GmbH Roentgenstra?e 13 39108 Magdeburg Germany t: +49 391 819099-0 f: +49 391 819099-299 e: info@REDACTED w: https://www.travelping.com/ Company registration: Amtsgericht Stendal Reg. No.: HRB 10578 Geschaeftsfuehrer: Holger Winkelmann VAT ID: DE236673780 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.platt@REDACTED Mon Oct 28 18:31:06 2019 From: pablo.platt@REDACTED (pablo platt) Date: Mon, 28 Oct 2019 19:31:06 +0200 Subject: Socket module (NIF) UDP example? Message-ID: Hi, I have a gen_udp server in production and I want to test the performance of the new socket NIF. How can I receive a message asynchronously similar to {active, once} in gen_udp? Do I need to keep a queue of outgoing packets in case the send buffer is full and resend them? Can I just increase the send buffer and ignore this complexity? With gen_udp I open the socket with random port: {ok, Socket} = gen_udp:open(0, [binary, {active, once}, {recbuf, 4096}]) Send data to a remote socket: gen_udp:send(Socket, Addr, Port, Data), Receive data with {active, one} handle_info({udp, Socket, Addr, Port, Data}, State) -> % process data inet:setopts(Socket, [{active, once}]), {noreply, State}. Close the socket: gen_udp:close(Socket), -------------- next part -------------- An HTML attachment was scrubbed... URL: From jakabac@REDACTED Mon Oct 28 18:29:43 2019 From: jakabac@REDACTED (Jaka Bac) Date: Mon, 28 Oct 2019 18:29:43 +0100 Subject: [erlang-questions] OTP 22.1 socket.erl somehow breaks message delivery or scheduler In-Reply-To: References: Message-ID: Hi, There were big improvements in compiler that were released with OTP 22. If you are interested in details, see http://blog.erlang.org/OTP-22-Highlights/ Jaka On Mon, Oct 28, 2019, 16:39 Andreas Schultz wrote: > Am Mo., 28. Okt. 2019 um 15:22 Uhr schrieb Bj?rn Gustavsson < > bjorn@REDACTED>: > >> Thanks for reporting this issue. >> >> We have investigated the issue and found that it is a bug in the compiler. >> > > wow. I have to say I'm a bit scared of OTP 22.x at the moment because of > all the "bugs in the compiler" changes lately. > Did the rewrite of the compiler in 22.0 cause all this or was it present > before and just went unnoticed? > > Anyway, many thanks for finding this, > Andreas > > >> https://bugs.erlang.org/browse/ERL-1076 >> >> https://github.com/erlang/otp/pull/2434 >> >> /Bj?rn >> >> On Wed, Oct 23, 2019 at 3:53 PM Andreas Schultz >> wrote: >> > >> > Hi, >> > >> > After converting an application to socket.erl in OTP 22.1, the test >> suites started to fail with random timeouts. It took me a while to figure >> out that gen_server:calls arrived multiple seconds late in the server. >> > >> > I have a demonstration at >> https://gist.github.com/RoadRunnr/311a7679fff6fbdf367c455b960f1ba8. It >> implements a simple UDP echo server with socket.erl. The client uses >> gen_udp to send messages and wait for the response. >> > The client also sends Erlang ping message to the server and expects to >> get a pong answer back. The socket.erl based server is supposed to not >> block (and as far as I can tell, it does not), it therefore should be able >> to answer the Erlang ping message all the time. >> > There are also some simple busy loop process running to get some load. >> Without them the problem is not reproducible. >> > >> > The sample is failing in about 20% off the test runs, when it does the >> output is something like: >> > >> > $ ~/stest.escript >> > Server Pid <0.78.0> >> > Server Addr #{addr => {127,0,0,1},family => inet,port => 38959} >> > ping timeout >> > round trip Clnt/Srvr Srvr/Clnt ProcPing >> > ... >> > 85: ******** ns, ******** ns, 57675 ns, 42332 ns >> > ... >> > >> > The failure happens because a 'ping' message is not see in time by the >> receive clause in the server process. It seems that either the process is >> not scheduled for some time (multiple seconds), or the scanning of the >> mailbox is missing the message. >> > >> > I have ruled out that the UDP messages are being dropped, otherwise the >> clients gen_udp:recv would never return. >> > >> > Does anyone have a clue what might cause this? Or point out where my >> sample is broken. >> > >> > Many thanks >> > Andreas >> > >> > -- >> > >> > Andreas Schultz >> > >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> -- >> Bj?rn Gustavsson, Erlang/OTP, Ericsson AB >> > > > -- > > Andreas Schultz > > -- > > Principal Engineer > > t: +49 391 819099-224 > > ------------------------------- enabling your networks > ----------------------------- > > Travelping GmbH > > Roentgenstra?e 13 > > 39108 Magdeburg > > Germany > > t: +49 391 819099-0 > > f: +49 391 819099-299 > > e: info@REDACTED > > w: https://www.travelping.com/ > > Company registration: Amtsgericht Stendal Reg. No.: HRB 10578 > Geschaeftsfuehrer: Holger Winkelmann VAT ID: DE236673780 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.platt@REDACTED Mon Oct 28 20:30:24 2019 From: pablo.platt@REDACTED (pablo platt) Date: Mon, 28 Oct 2019 21:30:24 +0200 Subject: [erlang-questions] OTP 22 socket module In-Reply-To: References: Message-ID: Andreas, do you have a wrapper module or some code you can share? I'm trying to replace a gen_udp server with the socket module. On Fri, Mar 29, 2019 at 7:18 PM Ingela Andin wrote: > Hi! > > Den ons 27 mars 2019 kl 08:20 skrev Andreas Schultz < > andreas.schultz@REDACTED>: > >> Hi, >> >> Is a {active, N} like delivery of data planed for the new socket or will >> polling the recv methods be the only way the receive data? What about >> non-blocking connect and accept? >> >> > It is planned to have behaviours that implement the active behaviour but > not as part of the socket module. The socket module should be the > primitives. > > > >> My preference would be to get an active notification when data is >> available for receive and another one when when a non-blocking write is >> possible. Sending and reception should then be left to the caller. >> Same goes for non-block accept and connect. Call accept/connect with a >> async option and get some message when the procedures finishes. >> >> I do realize that all this could be build with a gen_server/gen_statem >> around socket, so I guess the real question is does it make sense to start >> build such a wrapper or are the plans to make async functions part of the >> socket module? >> >> > Maybe Michael will have some more comments. > > > >> Thanks >> Andreas >> -- >> -- >> Dipl.-Inform. Andreas Schultz >> >> ----------------------- enabling your networks ---------------------- >> Travelping GmbH Phone: +49-391-81 90 99 0 >> Roentgenstr. 13 Fax: +49-391-81 90 99 299 >> 39108 Magdeburg Email: info@REDACTED >> GERMANY Web: http://www.travelping.com >> >> Company Registration: Amtsgericht Stendal Reg No.: HRB 10578 >> Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780 >> --------------------------------------------------------------------- >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dszoboszlay@REDACTED Mon Oct 28 23:46:46 2019 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Mon, 28 Oct 2019 23:46:46 +0100 Subject: Zeroization of sensitive data in Erlang In-Reply-To: References: <774ufvduq149fiqk9dl27c3h.1572160279411@email.android.com> Message-ID: Hi Amit, Maybe instead of 1. and 2. it would work just to make the GC in sensitive processes to zero refcounted binaries when it removes the last reference to them. This means all binary data exclusive to a sensitive process would be zeroed, and a binary shared between a sensitive and nonsensitive process may or may not get zeroed, depending on which process stops using it last. But that's fine most of the cases: data shared with a nonsensitive process shouldn't be sensitive. There's one caveat with sub binaries: you may have a large binary in a sensitive process that contains some sensitive data and some nonsensitive data too (e.g. you read in a file that contains a private key and a certificate), in which case you have to be careful not to share the nonsensitive part as a sub binary with an other process. Otherwise the GC may fail to zero out the sensitive part too when run from the nonsensitive process. This could be avoided by using binary:copy/1 before sharing the nonsensitive part. Daniel On Mon, 28 Oct 2019 at 13:23, Amit K wrote: > Hi Daniel, > > Thanks very much for your thoughts and insights. > Regarding your following comment about refc binaries: > > > What you should worry about are things like shared, reference-counted > binaries. They may be shared between sensitive and non-sensitive processes, > so do they have to be zeroed or not? What about the memory allocated by > NIF-s in sensitive processes? Like the openssl data structures where crypto > will copy the keys? I have a feeling that blindly zeroing every freed up > memory is inefficient, but doable, while extending the sensitive concept to > cover every aspect of the BEAM VM may not be feasible at all. > > I read up on this a little bit here: > http://erlang.org/doc/efficiency_guide/binaryhandling.html > and it seems that even when the binary is not shared it can reside on > the global binary heap - either because it is larger than 64 bytes, or > because it was referenced from another term, even if it's in the same > process. The latter seems to be easily avoidable if you carefully > write your sensitive data handling code, and the same goes for shared > binaries - if you're really writing code that handles sensitive data > you need to be careful about which other processes you share it with > anyway :) > The size limitation of 64 bytes however is a bit more painful, > although actually, for symmetric crypto keys and passwords 64 bytes is > very reasonable because AES-256 keys for example are only 32 bytes > long. Most passwords are also much shorter than 64 bytes. However of > course for asymmetric crypto the key sizes are much larger than 64 > bytes and this becomes a real issue. > > I have two ideas on how to address this: > > 1. Add a "sensitive" flag for the entire Erlang node as well, which at > least to begin with, will only tell the GC to also zero the binary > heap when it frees it. More relevant actions can be piled on top of > that semantic in the future. > > 2. The second would be to allow larger binaries to reside on the > process heap of sensitive processes, possibly even make this > configurable. This sounds to me a bit more complex than 1. > > One final point about NIFs - I think that's outside the scope of what > is possible to address. The use case I really want to address is the > handling of sensitive data in Erlang code and the passing of it to and > from the "crypto" module. Granted, OpenSSL is a C program and hence a > NIF (hope I got that right) but if OpenSSL fails to zero its internal > sensitive buffers, that's a bug that should be handled in the OpenSSL > context :) > > (BTW: after a bit of searching I found that OpenSSL uses > "openssl_cleanse" whenever it > needs to zero a sensitive buffer - > > https://github.com/openssl/openssl/search?q=openssl_cleanse&unscoped_q=openssl_cleanse > ) > > Thanks, > Amit > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Tue Oct 29 07:07:35 2019 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Tue, 29 Oct 2019 07:07:35 +0100 Subject: [erlang-questions] OTP 22.1 socket.erl somehow breaks message delivery or scheduler In-Reply-To: References: Message-ID: On Mon, Oct 28, 2019 at 4:39 PM Andreas Schultz < andreas.schultz@REDACTED> wrote: > Am Mo., 28. Okt. 2019 um 15:22 Uhr schrieb Bj?rn Gustavsson < > bjorn@REDACTED>: > >> Thanks for reporting this issue. >> >> We have investigated the issue and found that it is a bug in the compiler. >> > > wow. I have to say I'm a bit scared of OTP 22.x at the moment because of > all the "bugs in the compiler" changes lately. > Did the rewrite of the compiler in 22.0 cause all this or was it present > before and just went unnoticed? > It is a new bug in OTP 22. This particular bug was in the new optimization pass for optimizing receives. There are more details in our blog, but the short story is that we had reached a dead end in the compiler and that it was very time consuming and error prone to add new optimizations to the compiler. Most of the reported compiler bugs in OTP 22 was of the kind that causes the compiler to crash. Those bugs are annoying but basically harmless in that they can't subtly change the meaning of your program. Unfortunately, a few bugs were of the kind that the compiler generated incorrect code. Moving forward, the new compiler architecture will give us a solid ground for further improvements of the compiler. /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.schultz@REDACTED Tue Oct 29 08:50:22 2019 From: andreas.schultz@REDACTED (Andreas Schultz) Date: Tue, 29 Oct 2019 08:50:22 +0100 Subject: [erlang-questions] OTP 22 socket module In-Reply-To: References: Message-ID: Hi Pablo, For demonstrating a problem I've recently written a small escript [1] that does asynchronous UDP with the socket module. I've also converted a fairly complex application with two different UDP socket endpoints from our own implementation to the socket module [2]. For asynchronous receive you need OTP 22.1 and you definitely want an OTP version that has PR #2434 applied [3]. Regards, Andreas [1] https://gist.github.com/RoadRunnr/311a7679fff6fbdf367c455b960f1ba8 [2] https://github.com/travelping/ergw/commit/df61b18362277129d946160439301b5c5df9fcf1 [3] https://github.com/erlang/otp/pull/2434 Am Mo., 28. Okt. 2019 um 20:30 Uhr schrieb pablo platt < pablo.platt@REDACTED>: > Andreas, do you have a wrapper module or some code you can share? > I'm trying to replace a gen_udp server with the socket module. > > On Fri, Mar 29, 2019 at 7:18 PM Ingela Andin > wrote: > >> Hi! >> >> Den ons 27 mars 2019 kl 08:20 skrev Andreas Schultz < >> andreas.schultz@REDACTED>: >> >>> Hi, >>> >>> Is a {active, N} like delivery of data planed for the new socket or will >>> polling the recv methods be the only way the receive data? What about >>> non-blocking connect and accept? >>> >>> >> It is planned to have behaviours that implement the active behaviour but >> not as part of the socket module. The socket module should be the >> primitives. >> >> >> >>> My preference would be to get an active notification when data is >>> available for receive and another one when when a non-blocking write is >>> possible. Sending and reception should then be left to the caller. >>> Same goes for non-block accept and connect. Call accept/connect with a >>> async option and get some message when the procedures finishes. >>> >>> I do realize that all this could be build with a gen_server/gen_statem >>> around socket, so I guess the real question is does it make sense to start >>> build such a wrapper or are the plans to make async functions part of the >>> socket module? >>> >>> >> Maybe Michael will have some more comments. >> >> >> >>> Thanks >>> Andreas >>> -- >>> -- >>> Dipl.-Inform. Andreas Schultz >>> >>> ----------------------- enabling your networks ---------------------- >>> Travelping GmbH Phone: +49-391-81 90 99 0 >>> Roentgenstr. 13 Fax: +49-391-81 90 99 299 >>> 39108 Magdeburg Email: info@REDACTED >>> GERMANY Web: http://www.travelping.com >>> >>> Company Registration: Amtsgericht Stendal Reg No.: HRB 10578 >>> Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780 >>> --------------------------------------------------------------------- >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -- Andreas Schultz -- Principal Engineer t: +49 391 819099-224 ------------------------------- enabling your networks ----------------------------- Travelping GmbH Roentgenstra?e 13 39108 Magdeburg Germany t: +49 391 819099-0 f: +49 391 819099-299 e: info@REDACTED w: https://www.travelping.com/ Company registration: Amtsgericht Stendal Reg. No.: HRB 10578 Geschaeftsfuehrer: Holger Winkelmann VAT ID: DE236673780 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.platt@REDACTED Tue Oct 29 09:02:09 2019 From: pablo.platt@REDACTED (pablo platt) Date: Tue, 29 Oct 2019 10:02:09 +0200 Subject: [erlang-questions] OTP 22 socket module In-Reply-To: References: Message-ID: Thank you. That's very helpful. On Tue, Oct 29, 2019 at 9:50 AM Andreas Schultz < andreas.schultz@REDACTED> wrote: > Hi Pablo, > > For demonstrating a problem I've recently written a small escript [1] that > does asynchronous UDP with the socket module. > I've also converted a fairly complex application with two different UDP > socket endpoints from our own implementation to the socket module [2]. > > For asynchronous receive you need OTP 22.1 and you definitely want an > OTP version that has PR #2434 applied [3]. > > Regards, > Andreas > > [1] https://gist.github.com/RoadRunnr/311a7679fff6fbdf367c455b960f1ba8 > [2] > https://github.com/travelping/ergw/commit/df61b18362277129d946160439301b5c5df9fcf1 > [3] https://github.com/erlang/otp/pull/2434 > > Am Mo., 28. Okt. 2019 um 20:30 Uhr schrieb pablo platt < > pablo.platt@REDACTED>: > >> Andreas, do you have a wrapper module or some code you can share? >> I'm trying to replace a gen_udp server with the socket module. >> >> On Fri, Mar 29, 2019 at 7:18 PM Ingela Andin >> wrote: >> >>> Hi! >>> >>> Den ons 27 mars 2019 kl 08:20 skrev Andreas Schultz < >>> andreas.schultz@REDACTED>: >>> >>>> Hi, >>>> >>>> Is a {active, N} like delivery of data planed for the new socket or >>>> will polling the recv methods be the only way the receive data? What about >>>> non-blocking connect and accept? >>>> >>>> >>> It is planned to have behaviours that implement the active behaviour >>> but not as part of the socket module. The socket module should be the >>> primitives. >>> >>> >>> >>>> My preference would be to get an active notification when data is >>>> available for receive and another one when when a non-blocking write is >>>> possible. Sending and reception should then be left to the caller. >>>> Same goes for non-block accept and connect. Call accept/connect with a >>>> async option and get some message when the procedures finishes. >>>> >>>> I do realize that all this could be build with a gen_server/gen_statem >>>> around socket, so I guess the real question is does it make sense to start >>>> build such a wrapper or are the plans to make async functions part of the >>>> socket module? >>>> >>>> >>> Maybe Michael will have some more comments. >>> >>> >>> >>>> Thanks >>>> Andreas >>>> -- >>>> -- >>>> Dipl.-Inform. Andreas Schultz >>>> >>>> ----------------------- enabling your networks ---------------------- >>>> Travelping GmbH Phone: +49-391-81 90 99 0 >>>> Roentgenstr. 13 Fax: +49-391-81 90 99 299 >>>> 39108 Magdeburg Email: info@REDACTED >>>> GERMANY Web: http://www.travelping.com >>>> >>>> Company Registration: Amtsgericht Stendal Reg No.: HRB 10578 >>>> Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780 >>>> --------------------------------------------------------------------- >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> > > -- > > Andreas Schultz > > -- > > Principal Engineer > > t: +49 391 819099-224 > > ------------------------------- enabling your networks > ----------------------------- > > Travelping GmbH > > Roentgenstra?e 13 > > 39108 Magdeburg > > Germany > > t: +49 391 819099-0 > > f: +49 391 819099-299 > > e: info@REDACTED > > w: https://www.travelping.com/ > > Company registration: Amtsgericht Stendal Reg. No.: HRB 10578 > Geschaeftsfuehrer: Holger Winkelmann VAT ID: DE236673780 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.schultz@REDACTED Tue Oct 29 09:14:29 2019 From: andreas.schultz@REDACTED (Andreas Schultz) Date: Tue, 29 Oct 2019 09:14:29 +0100 Subject: [erlang-questions] OTP 22.1 socket.erl somehow breaks message delivery or scheduler In-Reply-To: References: Message-ID: Am Di., 29. Okt. 2019 um 07:07 Uhr schrieb Bj?rn Gustavsson < bjorn@REDACTED>: > On Mon, Oct 28, 2019 at 4:39 PM Andreas Schultz < > andreas.schultz@REDACTED> wrote: > >> Am Mo., 28. Okt. 2019 um 15:22 Uhr schrieb Bj?rn Gustavsson < >> bjorn@REDACTED>: >> >>> Thanks for reporting this issue. >>> >>> We have investigated the issue and found that it is a bug in the >>> compiler. >>> >> >> wow. I have to say I'm a bit scared of OTP 22.x at the moment because of >> all the "bugs in the compiler" changes lately. >> Did the rewrite of the compiler in 22.0 cause all this or was it present >> before and just went unnoticed? >> > > It is a new bug in OTP 22. This particular > bug was in the new optimization pass for > optimizing receives. > > There are more details in our blog, but the short > story is that we had reached a dead end in the > compiler and that it was very time consuming > and error prone to add new optimizations to > the compiler. > I'm sure that over time all the benefits of the new compiler architecture are well worth the price. But for OTP 22.x it has led to a few, but highly frustrating problems. Even the few incorrect code generation bugs have led to very hard to understand problems for the users. I'm not sure that scope and impact of this bug here is even fully understood. My demonstration code used a bare receive, but the code that actually triggered it used a plain gen_server. The result of the bug was that gen_sever:calls seemed to arrive extremely late (multiple seconds). It would therefore seem that the incorrect code was present in main receive loop of gen_server (and probably also gen_statem and gen_event). This would mean almost all Erlang applications on OTP 22.x could be affected. The effects might go unnoticed in many tests cases in other projects, until it causes unexplainable failures. Since all this was introduced in OTP 22, the sensible suggestion for everyone seem to be test OTP 22.x as well as they can, but to stay away from it for production use. Most of the reported compiler bugs in OTP 22 > was of the kind that causes the compiler to crash. > Those bugs are annoying but basically harmless in > that they can't subtly change the meaning of your > program. Unfortunately, a few bugs were of the kind > that the compiler generated incorrect code. > > Moving forward, the new compiler architecture > will give us a solid ground for further improvements > of the compiler. > I'm sure it will and I'm sure the current pain is well worth it. Regards, Andreas > /Bj?rn > > > -- > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Tue Oct 29 10:54:19 2019 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Tue, 29 Oct 2019 10:54:19 +0100 Subject: [erlang-questions] Matching IP address in socket module In-Reply-To: <64c9307c-f8ec-3599-3d21-f86fb09c9770@ericsson.com> References: <20191024100444.GA28388@tpccl.tty1.net> <20191024153351.GB28388@tpccl.tty1.net> <64c9307c-f8ec-3599-3d21-f86fb09c9770@ericsson.com> Message-ID: <497d87b2-f66c-6e45-5dfe-304c9eb27171@ninenines.eu> Hello, I would like to do a similar experiment with RabbitMQ. Would you mind sharing the code that makes 'socket' work similar to 'gen_tcp'? That may give me a head start, even if the code is not complete. Thanks, On 25/10/2019 12:39, Micael Karlberg wrote: > Hi, > > Its early days still, but the goal is definitely that it should > be faster. > > Here is some figures from a (time) test tool which is part of the > test suite (its basically a ping-pong case), both server and client > running on the same host (but in different VMs): > > Two transports: > ??????? gen:? gen_tcp > ??????? sock: socket (tcp) > > The socket active mode is a simulation of gen_tcp active mode > (no active = N). > > The tables are the exchange from the client side. > > With server side using gen (active = false): > > ??????? Transport?????? Active????????? Data > ??????? gen???????????? false?????????? 10192 byte/ms, 154 msgs/ms > ??????? gen???????????? true??????????? 10383 byte/ms, 157 msgs/ms > ??????? gen???????????? once??????????? 6003 byte/ms,? 90? msgs/ms > ??????? sock??????????? false?????????? 14050 byte/ms, 212 msgs/ms > ??????? sock??????????? true??????????? 14772 byte/ms, 223 msgs/ms > ??????? sock??????????? once??????????? 14050 byte/ms, 210 msgs/ms > > > With server side using gen (active = true): > > ??????? Transport?????? Active????????? Data > ??????? gen???????????? false?????????? 9447 byte/ms,? 143 msgs/ms > ??????? gen???????????? true??????????? 22345 byte/ms, 338 msgs/ms > ??????? gen???????????? once??????????? 5532 byte/ms,? 83? msgs/ms > ??????? sock??????????? false?????????? 15316 byte/ms, 232 msgs/ms > ??????? sock??????????? true??????????? 23693 byte/ms, 358 msgs/ms > ??????? sock??????????? once??????????? 22068 byte/ms, 334 msgs/ms > > > With server side using sock (active = false, async = true): > > ??????? Transport?????? Active????????? Data > ??????? gen???????????? false?????????? 11260 byte/ms, 170 msgs/ms > ??????? gen???????????? true??????????? 22273 byte/ms, 337 msgs/ms > ??????? gen???????????? once??????????? 7703 byte/ms,? 116 msgs/ms > ??????? sock??????????? false?????????? 15211 byte/ms, 230 msgs/ms > ??????? sock??????????? true??????????? 24778 byte/ms, 375 msgs/ms > ??????? sock??????????? once??????????? 23086 byte/ms, 349 msgs/ms > > > With server side using sock (active = true, async = true): > > ??????? Transport?????? Active????????? Data > ??????? gen???????????? false?????????? 11351 byte/ms, 171 msgs/ms > ??????? gen???????????? true??????????? 22469 byte/ms, 340 msgs/ms > ??????? gen???????????? once??????????? 7407 byte/ms,? 112 msgs/ms > ??????? sock??????????? false?????????? 15484 byte/ms, 234 msgs/ms > ??????? sock??????????? true??????????? 24885 byte/ms, 377 msgs/ms > ??????? sock??????????? once??????????? 23570 byte/ms, 357 msgs/ms > > > There is of course a bit of overhead since the socket transport > is trying to emulate (part of) gen_tcp. > > This is run on a Dell Precision T1700 running SLES 12 SP2 (not that > that effects the relative performance). > Run with a snapshot from the maint branch. > > This is obviously not a real use case, but it can be a guideline. > Also no UDP test (at the moment). > > /BMK > > On 2019-10-24 18:48, Frank Muller wrote: >> Hi Thomas >> >> Is the socket module faster than gen_tcp/gen_udp in your case? If yes, >> can you please share some numbers. >> >> /Frank >> >> ???? > case socket:recvfrom(Sock, 0, nowait) of >> ???? >? ? ? ?{ok, {#{family := Domain, >> ???? >? ? ? ? ? ? ? ?port? ?:= Port, >> ???? >? ? ? ? ? ? ? ?addr? ?:= Addr}, Data}} -> >> ???? >? ? ? ?. >> >> ??? Hi Micael, Mark, >> >> ??? thanks for your replies. The snippet above helped me getting the >> record >> ??? matched. >> >> ??? I'm quite impressed with the socket module, it seems to be pretty >> ??? complete, at least for my application. >> >> ??? Thanks, >> ??? Thomas >> -- Lo?c Hoguin https://ninenines.eu From micael.karlberg@REDACTED Tue Oct 29 11:12:19 2019 From: micael.karlberg@REDACTED (Micael Karlberg) Date: Tue, 29 Oct 2019 11:12:19 +0100 Subject: [erlang-questions] Matching IP address in socket module In-Reply-To: <497d87b2-f66c-6e45-5dfe-304c9eb27171@ninenines.eu> References: <20191024100444.GA28388@tpccl.tty1.net> <20191024153351.GB28388@tpccl.tty1.net> <64c9307c-f8ec-3599-3d21-f86fb09c9770@ericsson.com> <497d87b2-f66c-6e45-5dfe-304c9eb27171@ninenines.eu> Message-ID: Hi, Its not a secret :) You will find it in the emulator test catalog. The only problem is that it was never intended for public "consumption" and is therefor not documented: socket_test_ttest_tcp_socket.erl (uses socket.erl) socket_test_ttest_tcp_gen.erl (uses gen_tcp.erl) The idea here is that they should have an "identical" interface (so the users should work with both transports). Obviously the "socket" is different (a gen_tcp socket is a port and a socket socket is a term()). The other socket_test_ttest_tcp* modules "can" be used to see how to use these... Also, there is work on making gen_tcp work with the new socket module. This is ongoing. There is a lot of special options that gen_tcp/inet manages (packaging for instance). Regards, /BMK On 2019-10-29 10:54, Lo?c Hoguin wrote: > Hello, > > I would like to do a similar experiment with RabbitMQ. Would you mind sharing the code that makes > 'socket' work similar to 'gen_tcp'? That may give me a head start, even if the code is not complete. > > Thanks, > > On 25/10/2019 12:39, Micael Karlberg wrote: >> Hi, >> >> Its early days still, but the goal is definitely that it should >> be faster. >> >> Here is some figures from a (time) test tool which is part of the >> test suite (its basically a ping-pong case), both server and client >> running on the same host (but in different VMs): >> >> Two transports: >> ???????? gen:? gen_tcp >> ???????? sock: socket (tcp) >> >> The socket active mode is a simulation of gen_tcp active mode >> (no active = N). >> >> The tables are the exchange from the client side. >> >> With server side using gen (active = false): >> >> ???????? Transport?????? Active????????? Data >> ???????? gen???????????? false?????????? 10192 byte/ms, 154 msgs/ms >> ???????? gen???????????? true??????????? 10383 byte/ms, 157 msgs/ms >> ???????? gen???????????? once??????????? 6003 byte/ms,? 90? msgs/ms >> ???????? sock??????????? false?????????? 14050 byte/ms, 212 msgs/ms >> ???????? sock??????????? true??????????? 14772 byte/ms, 223 msgs/ms >> ???????? sock??????????? once??????????? 14050 byte/ms, 210 msgs/ms >> >> >> With server side using gen (active = true): >> >> ???????? Transport?????? Active????????? Data >> ???????? gen???????????? false?????????? 9447 byte/ms,? 143 msgs/ms >> ???????? gen???????????? true??????????? 22345 byte/ms, 338 msgs/ms >> ???????? gen???????????? once??????????? 5532 byte/ms,? 83? msgs/ms >> ???????? sock??????????? false?????????? 15316 byte/ms, 232 msgs/ms >> ???????? sock??????????? true??????????? 23693 byte/ms, 358 msgs/ms >> ???????? sock??????????? once??????????? 22068 byte/ms, 334 msgs/ms >> >> >> With server side using sock (active = false, async = true): >> >> ???????? Transport?????? Active????????? Data >> ???????? gen???????????? false?????????? 11260 byte/ms, 170 msgs/ms >> ???????? gen???????????? true??????????? 22273 byte/ms, 337 msgs/ms >> ???????? gen???????????? once??????????? 7703 byte/ms,? 116 msgs/ms >> ???????? sock??????????? false?????????? 15211 byte/ms, 230 msgs/ms >> ???????? sock??????????? true??????????? 24778 byte/ms, 375 msgs/ms >> ???????? sock??????????? once??????????? 23086 byte/ms, 349 msgs/ms >> >> >> With server side using sock (active = true, async = true): >> >> ???????? Transport?????? Active????????? Data >> ???????? gen???????????? false?????????? 11351 byte/ms, 171 msgs/ms >> ???????? gen???????????? true??????????? 22469 byte/ms, 340 msgs/ms >> ???????? gen???????????? once??????????? 7407 byte/ms,? 112 msgs/ms >> ???????? sock??????????? false?????????? 15484 byte/ms, 234 msgs/ms >> ???????? sock??????????? true??????????? 24885 byte/ms, 377 msgs/ms >> ???????? sock??????????? once??????????? 23570 byte/ms, 357 msgs/ms >> >> >> There is of course a bit of overhead since the socket transport >> is trying to emulate (part of) gen_tcp. >> >> This is run on a Dell Precision T1700 running SLES 12 SP2 (not that >> that effects the relative performance). >> Run with a snapshot from the maint branch. >> >> This is obviously not a real use case, but it can be a guideline. >> Also no UDP test (at the moment). >> >> /BMK >> >> On 2019-10-24 18:48, Frank Muller wrote: >>> Hi Thomas >>> >>> Is the socket module faster than gen_tcp/gen_udp in your case? If yes, can you please share some >>> numbers. >>> >>> /Frank >>> >>> ???? > case socket:recvfrom(Sock, 0, nowait) of >>> ???? >? ? ? ?{ok, {#{family := Domain, >>> ???? >? ? ? ? ? ? ? ?port? ?:= Port, >>> ???? >? ? ? ? ? ? ? ?addr? ?:= Addr}, Data}} -> >>> ???? >? ? ? ?. >>> >>> ??? Hi Micael, Mark, >>> >>> ??? thanks for your replies. The snippet above helped me getting the record >>> ??? matched. >>> >>> ??? I'm quite impressed with the socket module, it seems to be pretty >>> ??? complete, at least for my application. >>> >>> ??? Thanks, >>> ??? Thomas >>> > From pablo.platt@REDACTED Tue Oct 29 11:33:28 2019 From: pablo.platt@REDACTED (pablo platt) Date: Tue, 29 Oct 2019 12:33:28 +0200 Subject: [erlang-questions] Matching IP address in socket module In-Reply-To: References: <20191024100444.GA28388@tpccl.tty1.net> <20191024153351.GB28388@tpccl.tty1.net> <64c9307c-f8ec-3599-3d21-f86fb09c9770@ericsson.com> <497d87b2-f66c-6e45-5dfe-304c9eb27171@ninenines.eu> Message-ID: Micael, are there similar tests for udp? On Tue, Oct 29, 2019 at 12:12 PM Micael Karlberg < micael.karlberg@REDACTED> wrote: > Hi, > > Its not a secret :) You will find it in the emulator test catalog. > The only problem is that it was never intended for public "consumption" > and is therefor not documented: > > socket_test_ttest_tcp_socket.erl (uses socket.erl) > socket_test_ttest_tcp_gen.erl (uses gen_tcp.erl) > > The idea here is that they should have an "identical" interface (so > the users should work with both transports). > Obviously the "socket" is different (a gen_tcp socket is a port and > a socket socket is a term()). > > The other socket_test_ttest_tcp* modules "can" be used to see how to > use these... > > Also, there is work on making gen_tcp work with the new socket module. > This is ongoing. There is a lot of special options that gen_tcp/inet > manages (packaging for instance). > > Regards, > /BMK > > On 2019-10-29 10:54, Lo?c Hoguin wrote: > > Hello, > > > > I would like to do a similar experiment with RabbitMQ. Would you mind > sharing the code that makes > > 'socket' work similar to 'gen_tcp'? That may give me a head start, even > if the code is not complete. > > > > Thanks, > > > > On 25/10/2019 12:39, Micael Karlberg wrote: > >> Hi, > >> > >> Its early days still, but the goal is definitely that it should > >> be faster. > >> > >> Here is some figures from a (time) test tool which is part of the > >> test suite (its basically a ping-pong case), both server and client > >> running on the same host (but in different VMs): > >> > >> Two transports: > >> gen: gen_tcp > >> sock: socket (tcp) > >> > >> The socket active mode is a simulation of gen_tcp active mode > >> (no active = N). > >> > >> The tables are the exchange from the client side. > >> > >> With server side using gen (active = false): > >> > >> Transport Active Data > >> gen false 10192 byte/ms, 154 msgs/ms > >> gen true 10383 byte/ms, 157 msgs/ms > >> gen once 6003 byte/ms, 90 msgs/ms > >> sock false 14050 byte/ms, 212 msgs/ms > >> sock true 14772 byte/ms, 223 msgs/ms > >> sock once 14050 byte/ms, 210 msgs/ms > >> > >> > >> With server side using gen (active = true): > >> > >> Transport Active Data > >> gen false 9447 byte/ms, 143 msgs/ms > >> gen true 22345 byte/ms, 338 msgs/ms > >> gen once 5532 byte/ms, 83 msgs/ms > >> sock false 15316 byte/ms, 232 msgs/ms > >> sock true 23693 byte/ms, 358 msgs/ms > >> sock once 22068 byte/ms, 334 msgs/ms > >> > >> > >> With server side using sock (active = false, async = true): > >> > >> Transport Active Data > >> gen false 11260 byte/ms, 170 msgs/ms > >> gen true 22273 byte/ms, 337 msgs/ms > >> gen once 7703 byte/ms, 116 msgs/ms > >> sock false 15211 byte/ms, 230 msgs/ms > >> sock true 24778 byte/ms, 375 msgs/ms > >> sock once 23086 byte/ms, 349 msgs/ms > >> > >> > >> With server side using sock (active = true, async = true): > >> > >> Transport Active Data > >> gen false 11351 byte/ms, 171 msgs/ms > >> gen true 22469 byte/ms, 340 msgs/ms > >> gen once 7407 byte/ms, 112 msgs/ms > >> sock false 15484 byte/ms, 234 msgs/ms > >> sock true 24885 byte/ms, 377 msgs/ms > >> sock once 23570 byte/ms, 357 msgs/ms > >> > >> > >> There is of course a bit of overhead since the socket transport > >> is trying to emulate (part of) gen_tcp. > >> > >> This is run on a Dell Precision T1700 running SLES 12 SP2 (not that > >> that effects the relative performance). > >> Run with a snapshot from the maint branch. > >> > >> This is obviously not a real use case, but it can be a guideline. > >> Also no UDP test (at the moment). > >> > >> /BMK > >> > >> On 2019-10-24 18:48, Frank Muller wrote: > >>> Hi Thomas > >>> > >>> Is the socket module faster than gen_tcp/gen_udp in your case? If yes, > can you please share some > >>> numbers. > >>> > >>> /Frank > >>> > >>> > case socket:recvfrom(Sock, 0, nowait) of > >>> > {ok, {#{family := Domain, > >>> > port := Port, > >>> > addr := Addr}, Data}} -> > >>> > . > >>> > >>> Hi Micael, Mark, > >>> > >>> thanks for your replies. The snippet above helped me getting the > record > >>> matched. > >>> > >>> I'm quite impressed with the socket module, it seems to be pretty > >>> complete, at least for my application. > >>> > >>> Thanks, > >>> Thomas > >>> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.safarov@REDACTED Tue Oct 29 11:47:12 2019 From: s.safarov@REDACTED (Sergey Safarov) Date: Tue, 29 Oct 2019 13:47:12 +0300 Subject: Memory usage Message-ID: Hello On server installed program compiled using erlang 20.3.4 Using ps utility I can see that this program eat server memory. Program use about 50G of memory. How I can investigate what is eat memory? [root@REDACTED ~]# ps -F --pid 3947 UID PID PPID C SZ RSS PSR STIME TTY TIME CMD root 3947 3638 0 22520660 56920600 6 Aug09 pts/0 17:19:16 /opt/kazoo/erts-8.3/bin/beam.smp -Bd -pc unicode -K true -A 100 -W w -- -root /opt/kazoo -progname usr/local [root@REDACTED ~]# -------------- next part -------------- An HTML attachment was scrubbed... URL: From micael.karlberg@REDACTED Tue Oct 29 12:09:02 2019 From: micael.karlberg@REDACTED (Micael Karlberg) Date: Tue, 29 Oct 2019 12:09:02 +0100 Subject: [erlang-questions] Matching IP address in socket module In-Reply-To: References: <20191024100444.GA28388@tpccl.tty1.net> <20191024153351.GB28388@tpccl.tty1.net> <64c9307c-f8ec-3599-3d21-f86fb09c9770@ericsson.com> <497d87b2-f66c-6e45-5dfe-304c9eb27171@ninenines.eu> Message-ID: <82bf8e1d-9eec-ea4d-c2f3-58b322724b92@ericsson.com> No, its on the todo list, there is actually a ticket for this. I would be good for completeness, if nothing else. /BMK On 2019-10-29 11:33, pablo platt wrote: > Micael, are there similar tests for udp? > > On Tue, Oct 29, 2019 at 12:12 PM Micael Karlberg > wrote: > > Hi, > > Its not a secret :) You will find it in the emulator test catalog. > The only problem is that it was never intended for public "consumption" > and is therefor not documented: > > ? ? ? ? socket_test_ttest_tcp_socket.erl (uses socket.erl) > ? ? ? ? socket_test_ttest_tcp_gen.erl? ? (uses gen_tcp.erl) > > The idea here is that they should have an "identical" interface (so > the users should work with both transports). > Obviously the "socket" is different (a gen_tcp socket is a port and > a socket socket is a term()). > > The other socket_test_ttest_tcp* modules "can" be used to see how to > use these... > > Also, there is work on making gen_tcp work with the new socket module. > This is ongoing. There is a lot of special options that gen_tcp/inet > manages (packaging for instance). > > Regards, > ? ? ? ? /BMK > > On 2019-10-29 10:54, Lo?c Hoguin wrote: > > Hello, > > > > I would like to do a similar experiment with RabbitMQ. Would you mind sharing the code that > makes > > 'socket' work similar to 'gen_tcp'? That may give me a head start, even if the code is not > complete. > > > > Thanks, > > > > On 25/10/2019 12:39, Micael Karlberg wrote: > >> Hi, > >> > >> Its early days still, but the goal is definitely that it should > >> be faster. > >> > >> Here is some figures from a (time) test tool which is part of the > >> test suite (its basically a ping-pong case), both server and client > >> running on the same host (but in different VMs): > >> > >> Two transports: > >> ???????? gen:? gen_tcp > >> ???????? sock: socket (tcp) > >> > >> The socket active mode is a simulation of gen_tcp active mode > >> (no active = N). > >> > >> The tables are the exchange from the client side. > >> > >> With server side using gen (active = false): > >> > >> ???????? Transport?????? Active????????? Data > >> ???????? gen???????????? false?????????? 10192 byte/ms, 154 msgs/ms > >> ???????? gen???????????? true??????????? 10383 byte/ms, 157 msgs/ms > >> ???????? gen???????????? once??????????? 6003 byte/ms,? 90? msgs/ms > >> ???????? sock??????????? false?????????? 14050 byte/ms, 212 msgs/ms > >> ???????? sock??????????? true??????????? 14772 byte/ms, 223 msgs/ms > >> ???????? sock??????????? once??????????? 14050 byte/ms, 210 msgs/ms > >> > >> > >> With server side using gen (active = true): > >> > >> ???????? Transport?????? Active????????? Data > >> ???????? gen???????????? false?????????? 9447 byte/ms,? 143 msgs/ms > >> ???????? gen???????????? true??????????? 22345 byte/ms, 338 msgs/ms > >> ???????? gen???????????? once??????????? 5532 byte/ms,? 83? msgs/ms > >> ???????? sock??????????? false?????????? 15316 byte/ms, 232 msgs/ms > >> ???????? sock??????????? true??????????? 23693 byte/ms, 358 msgs/ms > >> ???????? sock??????????? once??????????? 22068 byte/ms, 334 msgs/ms > >> > >> > >> With server side using sock (active = false, async = true): > >> > >> ???????? Transport?????? Active????????? Data > >> ???????? gen???????????? false?????????? 11260 byte/ms, 170 msgs/ms > >> ???????? gen???????????? true??????????? 22273 byte/ms, 337 msgs/ms > >> ???????? gen???????????? once??????????? 7703 byte/ms,? 116 msgs/ms > >> ???????? sock??????????? false?????????? 15211 byte/ms, 230 msgs/ms > >> ???????? sock??????????? true??????????? 24778 byte/ms, 375 msgs/ms > >> ???????? sock??????????? once??????????? 23086 byte/ms, 349 msgs/ms > >> > >> > >> With server side using sock (active = true, async = true): > >> > >> ???????? Transport?????? Active????????? Data > >> ???????? gen???????????? false?????????? 11351 byte/ms, 171 msgs/ms > >> ???????? gen???????????? true??????????? 22469 byte/ms, 340 msgs/ms > >> ???????? gen???????????? once??????????? 7407 byte/ms,? 112 msgs/ms > >> ???????? sock??????????? false?????????? 15484 byte/ms, 234 msgs/ms > >> ???????? sock??????????? true??????????? 24885 byte/ms, 377 msgs/ms > >> ???????? sock??????????? once??????????? 23570 byte/ms, 357 msgs/ms > >> > >> > >> There is of course a bit of overhead since the socket transport > >> is trying to emulate (part of) gen_tcp. > >> > >> This is run on a Dell Precision T1700 running SLES 12 SP2 (not that > >> that effects the relative performance). > >> Run with a snapshot from the maint branch. > >> > >> This is obviously not a real use case, but it can be a guideline. > >> Also no UDP test (at the moment). > >> > >> /BMK > >> > >> On 2019-10-24 18:48, Frank Muller wrote: > >>> Hi Thomas > >>> > >>> Is the socket module faster than gen_tcp/gen_udp in your case? If yes, can you please share > some > >>> numbers. > >>> > >>> /Frank > >>> > >>> ???? > case socket:recvfrom(Sock, 0, nowait) of > >>> ???? >? ? ? ?{ok, {#{family := Domain, > >>> ???? >? ? ? ? ? ? ? ?port? ?:= Port, > >>> ???? >? ? ? ? ? ? ? ?addr? ?:= Addr}, Data}} -> > >>> ???? >? ? ? ?. > >>> > >>> ??? Hi Micael, Mark, > >>> > >>> ??? thanks for your replies. The snippet above helped me getting the record > >>> ??? matched. > >>> > >>> ??? I'm quite impressed with the socket module, it seems to be pretty > >>> ??? complete, at least for my application. > >>> > >>> ??? Thanks, > >>> ??? Thomas > >>> > > > From gerhard@REDACTED Tue Oct 29 12:10:32 2019 From: gerhard@REDACTED (Gerhard Lazu) Date: Tue, 29 Oct 2019 11:10:32 +0000 Subject: Memory usage In-Reply-To: References: Message-ID: http://erlang.org/doc/apps/observer/observer_ug.html is a good starting point. I had a good experience with https://github.com/zhongwencool/observer_cli & http://ferd.github.io/recon/ Relevant reads: https://github.com/happi/theBeamBook/blob/master/chapters/memory.asciidoc & https://www.erlang-in-anger.com/ On Tue, Oct 29, 2019 at 10:47 AM Sergey Safarov wrote: > Hello > On server installed program compiled using erlang 20.3.4 > Using ps utility I can see that this program eat server memory. > Program use about 50G of memory. > > How I can investigate what is eat memory? > > [root@REDACTED ~]# ps -F --pid 3947 > UID PID PPID C SZ RSS PSR STIME TTY TIME CMD > root 3947 3638 0 22520660 56920600 6 Aug09 pts/0 17:19:16 > /opt/kazoo/erts-8.3/bin/beam.smp -Bd -pc unicode -K true -A 100 -W w -- > -root /opt/kazoo -progname usr/local > [root@REDACTED ~]# > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn@REDACTED Tue Oct 29 12:40:07 2019 From: bjorn@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Tue, 29 Oct 2019 12:40:07 +0100 Subject: [erlang-questions] OTP 22.1 socket.erl somehow breaks message delivery or scheduler In-Reply-To: References: Message-ID: On Tue, Oct 29, 2019 at 9:14 AM Andreas Schultz wrote: > > I'm sure that over time all the benefits of the new compiler architecture are well worth the price. > But for OTP 22.x it has led to a few, but highly frustrating problems. Even the few incorrect code generation bugs have led to very hard to understand problems for the users. > > I'm not sure that scope and impact of this bug here is even fully understood. > > My demonstration code used a bare receive, but the code that actually triggered it used a plain gen_server. The result of the bug was that gen_sever:calls seemed to arrive extremely late (multiple seconds). > It would therefore seem that the incorrect code was present in main receive loop of gen_server (and probably also gen_statem and gen_event). This would mean almost all Erlang applications on OTP 22.x could be affected. > > The effects might go unnoticed in many tests cases in other projects, until it causes unexplainable failures. > > Since all this was introduced in OTP 22, the sensible suggestion for everyone seem to be test OTP 22.x as well as they can, but to stay away from it for production use. > We have done some more investigating. The issues you saw are caused by TWO bugs, the compiler bug and a bug in the run-time system. The bug in the compiler caused a position in the message queue to be saved even when it was not guaranteed that a receive would be executed. The only module in OTP that was hit by this bug was the socket module, and in this case it would have been harmless without the other bug. The bug in the run-time system would cause the saved position to be used for a receive that was not supposed to use the saved position. This bug was introduced in OTP 21, but it is unlikely that it could ever be triggered by code emitted by the compiler in OTP 21. So what does that mean for production use of OTP 22.x? We know that OTP 22 is already used in production. The problems you saw were caused by the use of the socket module that was hit by the compiler bug. Unless one has another module that uses receive in a similar way to how socket uses it, using gen_server and the other modules is perfectly safe. Since the socket module was introduced in OTP 22 and is still experimental, it would be advisable to avoid it in production use for the moment. The fix for the compiler should be enough to fix the kind of problems you saw, but we will of course fix the bug in the run-time system as well. /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From andreas.schultz@REDACTED Tue Oct 29 13:06:54 2019 From: andreas.schultz@REDACTED (Andreas Schultz) Date: Tue, 29 Oct 2019 13:06:54 +0100 Subject: [erlang-questions] OTP 22.1 socket.erl somehow breaks message delivery or scheduler In-Reply-To: References: Message-ID: Hi Bj?rn, Thanks for the detailed and thorough explanation. Can you point me to the fix for the second bug? I'm currently chasing what looks very much like the same issue, only the message that is not arriving in time is originating in the inet driver (gen_tcp) this time. But it could also be something different or me being too stupid. A complete explanation is here http://erlang.org/pipermail/erlang-questions/2019-September/098419.html Many thanks, Andreas Am Di., 29. Okt. 2019 um 12:40 Uhr schrieb Bj?rn Gustavsson < bjorn@REDACTED>: > On Tue, Oct 29, 2019 at 9:14 AM Andreas Schultz > wrote: > > > > I'm sure that over time all the benefits of the new compiler > architecture are well worth the price. > > But for OTP 22.x it has led to a few, but highly frustrating problems. > Even the few incorrect code generation bugs have led to very hard to > understand problems for the users. > > > > I'm not sure that scope and impact of this bug here is even fully > understood. > > > > My demonstration code used a bare receive, but the code that actually > triggered it used a plain gen_server. The result of the bug was that > gen_sever:calls seemed to arrive extremely late (multiple seconds). > > It would therefore seem that the incorrect code was present in main > receive loop of gen_server (and probably also gen_statem and gen_event). > This would mean almost all Erlang applications on OTP 22.x could be > affected. > > > > The effects might go unnoticed in many tests cases in other projects, > until it causes unexplainable failures. > > > > Since all this was introduced in OTP 22, the sensible suggestion for > everyone seem to be test OTP 22.x as well as they can, but to stay away > from it for production use. > > > > We have done some more investigating. > > The issues you saw are caused by TWO bugs, > the compiler bug and a bug in the run-time > system. > > The bug in the compiler caused a position in > the message queue to be saved even when it > was not guaranteed that a receive would be > executed. The only module in OTP that was > hit by this bug was the socket module, and > in this case it would have been harmless > without the other bug. > > The bug in the run-time system would cause > the saved position to be used for a receive that > was not supposed to use the saved position. > This bug was introduced in OTP 21, but it is > unlikely that it could ever be triggered by > code emitted by the compiler in OTP 21. > > So what does that mean for production use > of OTP 22.x? > > We know that OTP 22 is already used in > production. The problems you saw were > caused by the use of the socket module that > was hit by the compiler bug. Unless one has > another module that uses receive in a similar > way to how socket uses it, using gen_server > and the other modules is perfectly safe. > > Since the socket module was introduced > in OTP 22 and is still experimental, it > would be advisable to avoid it in production > use for the moment. > > The fix for the compiler should be enough > to fix the kind of problems you saw, but we > will of course fix the bug in the run-time > system as well. > > /Bj?rn > > -- > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB > -- Andreas Schultz -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladimir.ralev@REDACTED Tue Oct 29 13:24:46 2019 From: vladimir.ralev@REDACTED (Vladimir Ralev) Date: Tue, 29 Oct 2019 14:24:46 +0200 Subject: Memory usage In-Reply-To: References: Message-ID: Hi, This is something I used that has worked for me without installing additional software, it's like poor mans recon: https://www.mail-archive.com/user@REDACTED/msg29365.html Replace message_queue_len with total_heap_size to check the heap as well. It lists all processes by memory consumption, sorts them and looks at the top offender. In kazoo the biggest memory hog I've seen is the media file cache. If you have a lot of TTS I guess it's possible that it will get to 50GB at some point, but that's really a lot. You would see it in the contents here https://github.com/2600hz/kazoo/blob/master/core/kazoo_media/src/kz_media_tts_cache.erl If you can't figure it out I would volunteer to check it out for you via remsh. On Tue, Oct 29, 2019 at 12:47 PM Sergey Safarov wrote: > > Hello > On server installed program compiled using erlang 20.3.4 > Using ps utility I can see that this program eat server memory. > Program use about 50G of memory. > > How I can investigate what is eat memory? > > [root@REDACTED ~]# ps -F --pid 3947 > UID PID PPID C SZ RSS PSR STIME TTY TIME CMD > root 3947 3638 0 22520660 56920600 6 Aug09 pts/0 17:19:16 /opt/kazoo/erts-8.3/bin/beam.smp -Bd -pc unicode -K true -A 100 -W w -- -root /opt/kazoo -progname usr/local > [root@REDACTED ~]# > > From lukas@REDACTED Tue Oct 29 13:25:48 2019 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 29 Oct 2019 13:25:48 +0100 Subject: [erlang-questions] OTP 22.1 socket.erl somehow breaks message delivery or scheduler In-Reply-To: References: Message-ID: On Tue, Oct 29, 2019 at 1:07 PM Andreas Schultz < andreas.schultz@REDACTED> wrote: > Hi Bj?rn, > > Thanks for the detailed and thorough explanation. > > Can you point me to the fix for the second bug? > You can get a temporary workwaround here: https://github.com/garazdawi/otp/tree/lukas/erts/fix-sigq-save-bug It is most likely not the fix we are going to merge as it is too conservative about when the message queue optimization can trigger, but it is correct. Lukas > > I'm currently chasing what looks very much like the same issue, only the > message that is not arriving in time is originating in the inet driver > (gen_tcp) this time. > But it could also be something different or me being too stupid. A > complete explanation is here > http://erlang.org/pipermail/erlang-questions/2019-September/098419.html > > Many thanks, > Andreas > > Am Di., 29. Okt. 2019 um 12:40 Uhr schrieb Bj?rn Gustavsson < > bjorn@REDACTED>: > >> On Tue, Oct 29, 2019 at 9:14 AM Andreas Schultz >> wrote: >> > >> > I'm sure that over time all the benefits of the new compiler >> architecture are well worth the price. >> > But for OTP 22.x it has led to a few, but highly frustrating problems. >> Even the few incorrect code generation bugs have led to very hard to >> understand problems for the users. >> > >> > I'm not sure that scope and impact of this bug here is even fully >> understood. >> > >> > My demonstration code used a bare receive, but the code that actually >> triggered it used a plain gen_server. The result of the bug was that >> gen_sever:calls seemed to arrive extremely late (multiple seconds). >> > It would therefore seem that the incorrect code was present in main >> receive loop of gen_server (and probably also gen_statem and gen_event). >> This would mean almost all Erlang applications on OTP 22.x could be >> affected. >> > >> > The effects might go unnoticed in many tests cases in other projects, >> until it causes unexplainable failures. >> > >> > Since all this was introduced in OTP 22, the sensible suggestion for >> everyone seem to be test OTP 22.x as well as they can, but to stay away >> from it for production use. >> > >> >> We have done some more investigating. >> >> The issues you saw are caused by TWO bugs, >> the compiler bug and a bug in the run-time >> system. >> >> The bug in the compiler caused a position in >> the message queue to be saved even when it >> was not guaranteed that a receive would be >> executed. The only module in OTP that was >> hit by this bug was the socket module, and >> in this case it would have been harmless >> without the other bug. >> >> The bug in the run-time system would cause >> the saved position to be used for a receive that >> was not supposed to use the saved position. >> This bug was introduced in OTP 21, but it is >> unlikely that it could ever be triggered by >> code emitted by the compiler in OTP 21. >> >> So what does that mean for production use >> of OTP 22.x? >> >> We know that OTP 22 is already used in >> production. The problems you saw were >> caused by the use of the socket module that >> was hit by the compiler bug. Unless one has >> another module that uses receive in a similar >> way to how socket uses it, using gen_server >> and the other modules is perfectly safe. >> >> Since the socket module was introduced >> in OTP 22 and is still experimental, it >> would be advisable to avoid it in production >> use for the moment. >> >> The fix for the compiler should be enough >> to fix the kind of problems you saw, but we >> will of course fix the bug in the run-time >> system as well. >> >> /Bj?rn >> >> -- >> Bj?rn Gustavsson, Erlang/OTP, Ericsson AB >> > > > -- > > Andreas Schultz > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtp11@REDACTED Wed Oct 30 00:02:40 2019 From: rtp11@REDACTED (rtp) Date: Wed, 30 Oct 2019 00:02:40 +0100 Subject: OT: erldoc isn't redirecting properly Message-ID: Hi, for some days now, I can not call the "Erlang/OTP Documentation" online documentation with the URL http://erlang.org/erldoc The request ends with the message: The page isn?t redirecting properly An error occurred during a connection to www.erlang.org. Does anybody know anything about that? Thanks in advance Ralf PS: Clearing cookies does not help. From fchschneider@REDACTED Wed Oct 30 07:20:21 2019 From: fchschneider@REDACTED (Schneider) Date: Wed, 30 Oct 2019 07:20:21 +0100 Subject: OT: erldoc isn't redirecting properly In-Reply-To: References: Message-ID: You?re probably using Firefox, which gives this problem after the recent update. Chrome works fine. Frans > Op 30 okt. 2019 om 05:54 heeft rtp het volgende geschreven: > > ?Hi, > > for some days now, I can not call the "Erlang/OTP Documentation" online > documentation with the URL > > http://erlang.org/erldoc > > The request ends with the message: > > The page isn?t redirecting properly > > An error occurred during a connection to www.erlang.org. > > Does anybody know anything about that? > > Thanks in advance > > Ralf > > PS: Clearing cookies does not help. From g.a.vinogradov@REDACTED Wed Oct 30 07:29:58 2019 From: g.a.vinogradov@REDACTED (Gleb Vinogradov) Date: Wed, 30 Oct 2019 13:29:58 +0700 Subject: OT: erldoc isn't redirecting properly In-Reply-To: References: Message-ID: Frans, it doesn't work in Chrome too. Even wget fails: wget http://www.erlang.org/erldoc --2019-10-30 06:28:21-- http://www.erlang.org/erldoc Resolving www.erlang.org (www.erlang.org)... 18.184.93.36, 35.157.223.92 Connecting to www.erlang.org (www.erlang.org)|18.184.93.36|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://www.erlang.org/erldoc [following] --2019-10-30 06:28:21-- https://www.erlang.org/erldoc Connecting to www.erlang.org (www.erlang.org)|18.184.93.36|:443... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: http://erlang.org/erldoc [following] .... --2019-10-30 06:28:22-- http://erlang.org/erldoc Connecting to erlang.org (erlang.org)|192.121.151.106|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: http://www.erlang.org/erldoc [following] 20 redirections exceeded. ??, 30 ???. 2019 ?. ? 13:20, Schneider : > You?re probably using Firefox, which gives this problem after the recent > update. Chrome works fine. > > Frans > > > Op 30 okt. 2019 om 05:54 heeft rtp het volgende > geschreven: > > > > ?Hi, > > > > for some days now, I can not call the "Erlang/OTP Documentation" online > > documentation with the URL > > > > http://erlang.org/erldoc > > > > The request ends with the message: > > > > The page isn?t redirecting properly > > > > An error occurred during a connection to www.erlang.org. > > > > Does anybody know anything about that? > > > > Thanks in advance > > > > Ralf > > > > PS: Clearing cookies does not help. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fchschneider@REDACTED Wed Oct 30 07:47:56 2019 From: fchschneider@REDACTED (Frans Schneider) Date: Wed, 30 Oct 2019 07:47:56 +0100 Subject: OT: erldoc isn't redirecting properly In-Reply-To: References: Message-ID: For me, Chrome Version 78.0.3904.70 does work but wget also fails here with the same error. Frans On 10/30/19 7:29 AM, Gleb Vinogradov wrote: > Frans, it doesn't work in Chrome too. > Even wget fails: > > wget http://www.erlang.org/erldoc > --2019-10-30 06:28:21-- http://www.erlang.org/erldoc > Resolving www.erlang.org (www.erlang.org > )... 18.184.93.36, 35.157.223.92 > Connecting to www.erlang.org (www.erlang.org > )|18.184.93.36|:80... connected. > HTTP request sent, awaiting response... 301 Moved Permanently > Location: https://www.erlang.org/erldoc [following] > --2019-10-30 06:28:21-- https://www.erlang.org/erldoc > Connecting to www.erlang.org (www.erlang.org > )|18.184.93.36|:443... connected. > HTTP request sent, awaiting response... 301 Moved Permanently > Location: http://erlang.org/erldoc [following] > .... > --2019-10-30 06:28:22-- http://erlang.org/erldoc > Connecting to erlang.org (erlang.org > )|192.121.151.106|:80... connected. > HTTP request sent, awaiting response... 301 Moved Permanently > Location: http://www.erlang.org/erldoc [following] > 20 redirections exceeded. > > ??, 30 ???. 2019 ?. ? 13:20, Schneider >: > > You?re probably using Firefox, which gives this problem after the > recent update. Chrome works fine. > > Frans > > > Op 30 okt. 2019 om 05:54 heeft rtp > het volgende geschreven: > > > > ?Hi, > > > > for some days now, I can not call the "Erlang/OTP Documentation" > online > > documentation with the URL > > > > http://erlang.org/erldoc > > > > The request ends with the message: > > > >? ? The page isn?t redirecting properly > > > >? ? An error occurred during a connection to www.erlang.org > . > > > > Does anybody know anything about that? > > > > Thanks in advance > > > > Ralf > > > > PS: Clearing cookies does not help. > From marc@REDACTED Wed Oct 30 08:04:55 2019 From: marc@REDACTED (Marc Worrell) Date: Wed, 30 Oct 2019 08:04:55 +0100 Subject: OT: erldoc isn't redirecting properly In-Reply-To: References: Message-ID: <8A5F7CA2-82DF-4D0E-8D50-E9EAC27351E6@worrell.nl> It is a configuration error. That some browsers still work is only because they cached a previous correct redirect. marc$ curl -i 'http://www.erlang.org/erldoc' HTTP/1.1 301 Moved Permanently Content-length: 0 Location: https://www.erlang.org/erldoc marc$ curl -i 'https://www.erlang.org/erldoc' HTTP/2 301 date: Wed, 30 Oct 2019 07:01:34 GMT content-type: text/html content-length: 162 location: http://erlang.org/erldoc marc$ curl -i http://erlang.org/erldoc HTTP/1.1 301 Moved Permanently Server: nginx Date: Wed, 30 Oct 2019 07:02:56 GMT Content-Type: text/html Content-Length: 178 Connection: keep-alive Location: http://www.erlang.org/erldoc > On 30 Oct 2019, at 07:47, Frans Schneider wrote: > > For me, Chrome Version 78.0.3904.70 does work but wget also fails here with the same error. > > Frans > > On 10/30/19 7:29 AM, Gleb Vinogradov wrote: >> Frans, it doesn't work in Chrome too. >> Even wget fails: >> wget http://www.erlang.org/erldoc >> --2019-10-30 06:28:21-- http://www.erlang.org/erldoc >> Resolving www.erlang.org (www.erlang.org )... 18.184.93.36, 35.157.223.92 >> Connecting to www.erlang.org (www.erlang.org )|18.184.93.36|:80... connected. >> HTTP request sent, awaiting response... 301 Moved Permanently >> Location: https://www.erlang.org/erldoc [following] >> --2019-10-30 06:28:21-- https://www.erlang.org/erldoc >> Connecting to www.erlang.org (www.erlang.org )|18.184.93.36|:443... connected. >> HTTP request sent, awaiting response... 301 Moved Permanently >> Location: http://erlang.org/erldoc [following] >> .... >> --2019-10-30 06:28:22-- http://erlang.org/erldoc >> Connecting to erlang.org (erlang.org )|192.121.151.106|:80... connected. >> HTTP request sent, awaiting response... 301 Moved Permanently >> Location: http://www.erlang.org/erldoc [following] >> 20 redirections exceeded. >> ??, 30 ???. 2019 ?. ? 13:20, Schneider >: >> You?re probably using Firefox, which gives this problem after the >> recent update. Chrome works fine. >> Frans >> > Op 30 okt. 2019 om 05:54 heeft rtp > > het volgende geschreven: >> > >> > ?Hi, >> > >> > for some days now, I can not call the "Erlang/OTP Documentation" >> online >> > documentation with the URL >> > >> > http://erlang.org/erldoc >> > >> > The request ends with the message: >> > >> > The page isn?t redirecting properly >> > >> > An error occurred during a connection to www.erlang.org >> . >> > >> > Does anybody know anything about that? >> > >> > Thanks in advance >> > >> > Ralf >> > >> > PS: Clearing cookies does not help. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Wed Oct 30 08:42:30 2019 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 30 Oct 2019 08:42:30 +0100 Subject: OT: erldoc isn't redirecting properly In-Reply-To: <8A5F7CA2-82DF-4D0E-8D50-E9EAC27351E6@worrell.nl> References: <8A5F7CA2-82DF-4D0E-8D50-E9EAC27351E6@worrell.nl> Message-ID: <20191030074230.GA3808@erix.ericsson.se> On Wed, Oct 30, 2019 at 08:04:55AM +0100, Marc Worrell wrote: > It is a configuration error. > That some browsers still work is only because they cached a previous correct redirect. Spot on! When transferring to the new server we forgot the old /erldoc redirect. The newer http://erlang.org/doc/search works, and the caching in my browser tricked me into believing that http://erlang.org/erldoc also still worked. But it is fixed now. Thank you first poster for pointing this out. / Raimo Niskanen > > marc$ curl -i 'http://www.erlang.org/erldoc' > HTTP/1.1 301 Moved Permanently > Content-length: 0 > Location: https://www.erlang.org/erldoc > > marc$ curl -i 'https://www.erlang.org/erldoc' > HTTP/2 301 > date: Wed, 30 Oct 2019 07:01:34 GMT > content-type: text/html > content-length: 162 > location: http://erlang.org/erldoc > > marc$ curl -i http://erlang.org/erldoc > HTTP/1.1 301 Moved Permanently > Server: nginx > Date: Wed, 30 Oct 2019 07:02:56 GMT > Content-Type: text/html > Content-Length: 178 > Connection: keep-alive > Location: http://www.erlang.org/erldoc > > > > > On 30 Oct 2019, at 07:47, Frans Schneider wrote: > > > > For me, Chrome Version 78.0.3904.70 does work but wget also fails here with the same error. > > > > Frans > > > > On 10/30/19 7:29 AM, Gleb Vinogradov wrote: > >> Frans, it doesn't work in Chrome too. > >> Even wget fails: > >> wget http://www.erlang.org/erldoc > >> --2019-10-30 06:28:21-- http://www.erlang.org/erldoc > >> Resolving www.erlang.org (www.erlang.org )... 18.184.93.36, 35.157.223.92 > >> Connecting to www.erlang.org (www.erlang.org )|18.184.93.36|:80... connected. > >> HTTP request sent, awaiting response... 301 Moved Permanently > >> Location: https://www.erlang.org/erldoc [following] > >> --2019-10-30 06:28:21-- https://www.erlang.org/erldoc > >> Connecting to www.erlang.org (www.erlang.org )|18.184.93.36|:443... connected. > >> HTTP request sent, awaiting response... 301 Moved Permanently > >> Location: http://erlang.org/erldoc [following] > >> .... > >> --2019-10-30 06:28:22-- http://erlang.org/erldoc > >> Connecting to erlang.org (erlang.org )|192.121.151.106|:80... connected. > >> HTTP request sent, awaiting response... 301 Moved Permanently > >> Location: http://www.erlang.org/erldoc [following] > >> 20 redirections exceeded. > >> ??, 30 ???. 2019 ?. ? 13:20, Schneider >: > >> You?re probably using Firefox, which gives this problem after the > >> recent update. Chrome works fine. > >> Frans > >> > Op 30 okt. 2019 om 05:54 heeft rtp >> > het volgende geschreven: > >> > > >> > ?Hi, > >> > > >> > for some days now, I can not call the "Erlang/OTP Documentation" > >> online > >> > documentation with the URL > >> > > >> > http://erlang.org/erldoc > >> > > >> > The request ends with the message: > >> > > >> > The page isn?t redirecting properly > >> > > >> > An error occurred during a connection to www.erlang.org > >> . > >> > > >> > Does anybody know anything about that? > >> > > >> > Thanks in advance > >> > > >> > Ralf > >> > > >> > PS: Clearing cookies does not help. > -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From essen@REDACTED Wed Oct 30 09:01:58 2019 From: essen@REDACTED (=?UTF-8?Q?Lo=c3=afc_Hoguin?=) Date: Wed, 30 Oct 2019 09:01:58 +0100 Subject: OT: erldoc isn't redirecting properly In-Reply-To: <20191030074230.GA3808@erix.ericsson.se> References: <8A5F7CA2-82DF-4D0E-8D50-E9EAC27351E6@worrell.nl> <20191030074230.GA3808@erix.ericsson.se> Message-ID: I mentioned it last week on IRC but I guess you're not on there. :-) On 30/10/2019 08:42, Raimo Niskanen wrote: > On Wed, Oct 30, 2019 at 08:04:55AM +0100, Marc Worrell wrote: >> It is a configuration error. >> That some browsers still work is only because they cached a previous correct redirect. > > Spot on! > > When transferring to the new server we forgot the old /erldoc redirect. > > The newer http://erlang.org/doc/search works, and the caching in my browser > tricked me into believing that http://erlang.org/erldoc also still worked. > > But it is fixed now. > > Thank you first poster for pointing this out. > / Raimo Niskanen > > >> >> marc$ curl -i 'http://www.erlang.org/erldoc' >> HTTP/1.1 301 Moved Permanently >> Content-length: 0 >> Location: https://www.erlang.org/erldoc >> >> marc$ curl -i 'https://www.erlang.org/erldoc' >> HTTP/2 301 >> date: Wed, 30 Oct 2019 07:01:34 GMT >> content-type: text/html >> content-length: 162 >> location: http://erlang.org/erldoc >> >> marc$ curl -i http://erlang.org/erldoc >> HTTP/1.1 301 Moved Permanently >> Server: nginx >> Date: Wed, 30 Oct 2019 07:02:56 GMT >> Content-Type: text/html >> Content-Length: 178 >> Connection: keep-alive >> Location: http://www.erlang.org/erldoc >> >> >> >>> On 30 Oct 2019, at 07:47, Frans Schneider wrote: >>> >>> For me, Chrome Version 78.0.3904.70 does work but wget also fails here with the same error. >>> >>> Frans >>> >>> On 10/30/19 7:29 AM, Gleb Vinogradov wrote: >>>> Frans, it doesn't work in Chrome too. >>>> Even wget fails: >>>> wget http://www.erlang.org/erldoc >>>> --2019-10-30 06:28:21-- http://www.erlang.org/erldoc >>>> Resolving www.erlang.org (www.erlang.org )... 18.184.93.36, 35.157.223.92 >>>> Connecting to www.erlang.org (www.erlang.org )|18.184.93.36|:80... connected. >>>> HTTP request sent, awaiting response... 301 Moved Permanently >>>> Location: https://www.erlang.org/erldoc [following] >>>> --2019-10-30 06:28:21-- https://www.erlang.org/erldoc >>>> Connecting to www.erlang.org (www.erlang.org )|18.184.93.36|:443... connected. >>>> HTTP request sent, awaiting response... 301 Moved Permanently >>>> Location: http://erlang.org/erldoc [following] >>>> .... >>>> --2019-10-30 06:28:22-- http://erlang.org/erldoc >>>> Connecting to erlang.org (erlang.org )|192.121.151.106|:80... connected. >>>> HTTP request sent, awaiting response... 301 Moved Permanently >>>> Location: http://www.erlang.org/erldoc [following] >>>> 20 redirections exceeded. >>>> ??, 30 ???. 2019 ?. ? 13:20, Schneider >: >>>> You?re probably using Firefox, which gives this problem after the >>>> recent update. Chrome works fine. >>>> Frans >>>> > Op 30 okt. 2019 om 05:54 heeft rtp >>> > het volgende geschreven: >>>> > >>>> > ?Hi, >>>> > >>>> > for some days now, I can not call the "Erlang/OTP Documentation" >>>> online >>>> > documentation with the URL >>>> > >>>> > http://erlang.org/erldoc >>>> > >>>> > The request ends with the message: >>>> > >>>> > The page isn?t redirecting properly >>>> > >>>> > An error occurred during a connection to www.erlang.org >>>> . >>>> > >>>> > Does anybody know anything about that? >>>> > >>>> > Thanks in advance >>>> > >>>> > Ralf >>>> > >>>> > PS: Clearing cookies does not help. >> > -- Lo?c Hoguin https://ninenines.eu From v@REDACTED Wed Oct 30 14:34:53 2019 From: v@REDACTED (Valentin Micic) Date: Wed, 30 Oct 2019 15:34:53 +0200 Subject: Binary, List and Tuple Inequalities (Paradox?) In-Reply-To: <180ae745ba2acccfadafebe278a45155a68ab461.camel@ericsson.com> References: <180ae745ba2acccfadafebe278a45155a68ab461.camel@ericsson.com> Message-ID: > On 28 Oct 2019, at 10:02, Raimo Niskanen wrote: > > I'd like to defend the current term order of lists, tuples and > binaries. > > Lists are compared a'la string comparison, which avoids having to > traverse the whole list when comparing. Binaries are compared as lists > since both are used for handling strings, and also to be compared like > memcmp does it. > > Tuples are used for general complex terms. They can be compared size > first whithout having to be fully traversed, as opposed to lists, which > is useful in other contexts. You can thereby choose your data > structure to select comparision. > -- > Raimo Niskanen > Thanks Raimo, Your argument seems to have one?well, misconception. I don?t think it would be correct to imply that Binaries are used exclusively for handling strings, but rather, one of their uses may be that. For example, here's a syntax that is not anything ?string-like?: (tsdb_1_1@REDACTED)3593> <<1:16>> < <<2:16>>. true (tsdb_1_1@REDACTED)3594> <<50:16>> < <<36:16>>. false As you could see, this pretty much behaves the way we would expect when we compare two integer values, e.g. 1 is less than 2, and so is <<1:16>> < <<2:16>> Technically, we are not talking about strings here, but, rather two 16-bit integers. I can also say that: (tsdb_1_1@REDACTED)3597> 1 < 000002. true When we write: (tsdb_1_1@REDACTED)3593> <<1:16>>. It would be wrong to pretend that we?re actually talking about a strings (e.g. in alphanumeric sense). This clearly means that the integer value of 1 stored using Big-Endian encoding (e.g. network byte order). Thus, when we write: <<2:16>> we get <<0,2>>. When we write: <<2:24>> we get <<0,0,2>>... these values are *not* intended to be strings, but integers. So, when we add leading zeroes, we do not change the integer value. So why is then: (tsdb_1_1@REDACTED)3600> <<1:16>> < <<2:24>>. false First, we?re clearly use integer syntax to encode integer values, then we have the first integer value encoded using 16-bits, and the second integer value encoded using 24-bits. It just happens so, that 16-bits is used to encode the value of 1, and 24-bits to encode the value of two. Thus, since 16-bits are less then 24-bits (in length), but also 1 is less than 2, one may expect this to yield TRUE. Yet somehow, two octets are NOT LESS than there, nor 1 is NOT LESS than 2! I think this cannot pass the "red-face test?, and thus does not deserve defending. Contrast this with the way tuples are handled: (tsdb_1_1@REDACTED)3666> {1,1} < {1,2}. true (tsdb_1_1@REDACTED)3667> {1,3} < {1,2}. false (tsdb_1_1@REDACTED)3668> {1,3} < {1,2,3}. true Considering that Binaries may be used to encode ANYTHING, shouldn?t they be handled the same way as tuples instead of: (tsdb_1_1@REDACTED)3624> <<1,1>> < <<1,2>>. true (tsdb_1_1@REDACTED)3625> <<1,3>> < <<1,2>>. false (tsdb_1_1@REDACTED)3626> <<1,3>> < <<1,2,3>>. false As I said in my previous email, I do not expect Erlang to change, and for my "own intents and purposes? I am still considering if: (tsdb_1_1@REDACTED)3626> <<1,3>> < <<1,2,3>>. false should be given more credence than, say TRUE... if nothing else, because two octets are less than three octets. In other words, if a three-element-tuple, regardless of its content, could never be less than any given two-elements-tuple, why shouldn't the same hold for Binaries? Kind regards V/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From icfp.publicity@REDACTED Wed Oct 30 14:49:13 2019 From: icfp.publicity@REDACTED (Sam Tobin-Hochstadt) Date: Wed, 30 Oct 2019 09:49:13 -0400 Subject: Call for Workshop Proposals: ICFP 2020 Message-ID: <5db994d9630d5_2a3d2af7799bc5bc28b2@homer.mail> CALL FOR WORKSHOP AND CO-LOCATED EVENT PROPOSALS ICFP 2020 25th ACM SIGPLAN International Conference on Functional Programming August 23 - 28, 2020 Jersey City, NJ, US https://icfp19.sigplan.org/ The 25th ACM SIGPLAN International Conference on Functional Programming will be held in Jersey City, New Jersey on August 23-28, 2020. ICFP provides a forum for researchers and developers to hear about the latest work on the design, implementations, principles, and uses of functional programming. Proposals are invited for workshops (and other co-located events, such as symposiums) to be affiliated with ICFP 2020 and sponsored by SIGPLAN. These events should be less formal and more focused than ICFP itself, include sessions that enable interaction among the attendees, and foster the exchange of new ideas. The preference is for one-day events, but other schedules can also be considered. The workshops are scheduled to occur on August 23rd (the day before ICFP) and 27-28th of August (the two days after ICFP). ---------------------------------------------------------------------- Submission details Deadline for submission: November 15, 2019 Notification of acceptance: December 13, 2019 Prospective organizers of workshops or other co-located events are invited to submit a completed workshop proposal form in plain text format to the ICFP 2020 workshop co-chairs (Jennifer Hackett and Leonidas Lampropoulos) via email to icfp-workshops-2020@REDACTED by November 15, 2019. (For proposals of co-located events other than workshops, please fill in the workshop proposal form and just leave blank any sections that do not apply.) Please note that this is a firm deadline. Organizers will be notified if their event proposal is accepted by December 13, 2019, and if successful, depending on the event, they will be asked to produce a final report after the event has taken place that is suitable for publication in SIGPLAN Notices. The proposal form is available at: http://www.icfpconference.org/icfp2020-files/icfp20-workshops-form.txt Further information about SIGPLAN sponsorship is available at: http://www.sigplan.org/Resources/Proposals/Sponsored/ ---------------------------------------------------------------------- Selection committee The proposals will be evaluated by a committee comprising the following members of the ICFP 2020 organizing committee, together with the members of the SIGPLAN executive committee. Workshop Co-Chair: Jennifer Hackett (University of Nottingham) Workshop Co-Chair: Leonidas Lampropoulos (University of Maryland) General Chair: Stephanie Weirich (University of Pennsylvania) Program Chair: Adam Chlipala (MIT) ---------------------------------------------------------------------- Further information Any queries should be addressed to the workshop co-chairs (Jennifer Hackett and Leonidas Lampropoulos), via email to icfp-workshops-2020@REDACTED From dszoboszlay@REDACTED Thu Oct 31 00:13:17 2019 From: dszoboszlay@REDACTED (=?UTF-8?Q?D=C3=A1niel_Szoboszlay?=) Date: Thu, 31 Oct 2019 00:13:17 +0100 Subject: Binary, List and Tuple Inequalities (Paradox?) In-Reply-To: References: <180ae745ba2acccfadafebe278a45155a68ab461.camel@ericsson.com> Message-ID: Hi Valentin, I think you should acknowledge that the ordering of terms as defined by the Erlang language does not aim to fit any semantic interpretation of the data you may have. The sole purpose of the ordering is to be well defined, so you can create e.g. a binary search tree that holds whatever data. For that purpose any crazy ordering like 1 < false < 0 < [1, 2, foobar] < <0.12.34> < 42 < [] < 30 < baz ... would work fine, as long as the place of every term in this sequence is well defined. There's only one constraint (apart from not violating the mathematical properties of a strict ordering relation), and that is to make comparison of terms reasonably efficient (my above example would fail this check). So you feel <<1:16>> should be less than <<2:24>>? That's because you interpret <<0,1>> and <<0,0,2>> as variable length encoding of integers. If you have this use case, than write a custom comparison function and sort your binaries with that. I may have a use case where the same two binaries mean operation #1 (with no arguments) and operation #0 (with a single argument 2), so I would have to write a different comparison function (but as lucky as I am, I may just use erlang:'<'/2 this time, phew!). But if you would like to know why I find the built-in ordering of terms logical, it is because both lists and binaries are usually used to represent variable sized data, while tuples are for fixed size data. I mean that {1,2} and {1,2,3} typically describe semantically different things, e.g. a 2D coordinate and a semver version number. Their relative ordering is as irrelevant for most applications as the ordering of {1,2} and <0.0.1>. Ordering by tuple size first makes some sense because it will keep types together: just like all integers are less than atoms, all 2D coordinates are less than version numbers. Binaries may represent fixed size data (e.g. a packet header), but than it doesn't matter whether you order lexicographically or size first, because all binaries in your domain will be the same size. Most often however binaries represent variable sized data, such as payload in a packet. And for variable sized data the human convention is lexicographical ordering. A printed dictionary may sort words by length first, and it would be a (mostly) usable dictionary, but that's just not how things are by convention. Cheers, D?niel P.S.: By the way, this is why I don't like passing fixed length lists as arguments for init/1 in gen_* callbacks (such as init([Some, Args, Here]) -> ...). These should be tuples instead, they are not variable length! On Wed, 30 Oct 2019 at 14:35, Valentin Micic wrote: > > On 28 Oct 2019, at 10:02, Raimo Niskanen > wrote: > > I'd like to defend the current term order of lists, tuples and > binaries. > > Lists are compared a'la string comparison, which avoids having to > traverse the whole list when comparing. Binaries are compared as lists > since both are used for handling strings, and also to be compared like > memcmp does it. > > Tuples are used for general complex terms. They can be compared size > first whithout having to be fully traversed, as opposed to lists, which > is useful in other contexts. You can thereby choose your data > structure to select comparision. > -- > Raimo Niskanen > > > Thanks Raimo, > > Your argument seems to have one?well, misconception. I don?t think it > would be correct to imply that Binaries are used exclusively for handling > strings, but rather, one of their uses may be that. > > For example, here's a syntax that is not anything ?string-like?: > > > (tsdb_1_1@REDACTED)3593> <<1:16>> < <<2:16>>. > true > (tsdb_1_1@REDACTED)3594> <<50:16>> < <<36:16>>. > false > > > As you could see, this pretty much behaves the way we would expect when we > compare two integer values, e.g. > > 1 is less than 2, and so is <<1:16>> < <<2:16>> Technically, we are not > talking about strings here, but, rather two 16-bit integers. > > I can also say that: > > (tsdb_1_1@REDACTED)3597> 1 < 000002. > true > > When we write: > > (tsdb_1_1@REDACTED)3593> <<1:16>>. > > It would be wrong to pretend that we?re actually talking about a strings > (e.g. in alphanumeric sense). This clearly means that the integer value of > 1 stored using Big-Endian encoding (e.g. network byte order). > > Thus, when we write: <<2:16>> we get <<0,2>>. When we write: <<2:24>> we > get <<0,0,2>>... these values are *not* intended to be strings, but > integers. > So, when we add leading zeroes, we do not change the integer value. > > So why is then: > > (tsdb_1_1@REDACTED)3600> <<1:16>> < <<2:24>>. > false > > First, we?re clearly use integer syntax to encode integer values, then we > have the first integer value encoded using 16-bits, and the second integer > value encoded using 24-bits. > It just happens so, that 16-bits is used to encode the value of 1, and > 24-bits to encode the value of two. > > Thus, since 16-bits are less then 24-bits (in length), but also 1 is less > than 2, one may expect this to yield TRUE. Yet somehow, two octets are NOT > LESS than there, nor 1 is NOT LESS than 2! > > I think this cannot pass the "red-face test?, and thus does not deserve > defending. > > Contrast this with the way tuples are handled: > > (tsdb_1_1@REDACTED)3666> {1,1} < {1,2}. > true > (tsdb_1_1@REDACTED)3667> {1,3} < {1,2}. > false > (tsdb_1_1@REDACTED)3668> {1,3} < {1,2,3}. > true > > Considering that Binaries may be used to encode ANYTHING, shouldn?t they > be handled the same way as tuples instead of: > > > (tsdb_1_1@REDACTED)3624> <<1,1>> < <<1,2>>. > true > (tsdb_1_1@REDACTED)3625> <<1,3>> < <<1,2>>. > false > (tsdb_1_1@REDACTED)3626> <<1,3>> < <<1,2,3>>. > false > > > As I said in my previous email, I do not expect Erlang to change, and for > my "own intents and purposes? I am still considering if: > > (tsdb_1_1@REDACTED)3626> <<1,3>> < <<1,2,3>>. > false > > should be given more credence than, say TRUE... if nothing else, because > two octets are less than three octets. > > In other words, if a three-element-tuple, regardless of its content, > could never be less than any given two-elements-tuple, why shouldn't the > same hold for Binaries? > > Kind regards > > V/ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elbrujohalcon@REDACTED Thu Oct 31 03:23:38 2019 From: elbrujohalcon@REDACTED (Brujo Benavides) Date: Wed, 30 Oct 2019 23:23:38 -0300 Subject: Off-topic from Binary, List and Tuple Inequalities (Paradox?) In-Reply-To: References: <180ae745ba2acccfadafebe278a45155a68ab461.camel@ericsson.com> Message-ID: *P.S.: By the way, this is why I don't like passing fixed length lists as arguments for init/1 in gen_* callbacks (such as init([Some, Args, Here]) -> ...). These should be tuples instead, they are not variable length!* YES!!! Yes yes!! N+1 times yes! I've said this many times: the only argument that init/1 has should be called Arg, not Args... And if you don't have anything to pass in to that function as an argument on start_link, just use 'undefined' or any other sensibly named atom... Even {} is better than []. Using lists is not only semantically incorrect, it's also often confused with mfa (i.e. module, function & arguments, like in spawn/3) and people then define init with multiple arguments ????? On Wed, 30 Oct 2019 at 20:13 D?niel Szoboszlay wrote: > Hi Valentin, > > I think you should acknowledge that the ordering of terms as defined by > the Erlang language does not aim to fit any semantic interpretation of the > data you may have. The sole purpose of the ordering is to be well defined, > so you can create e.g. a binary search tree that holds whatever data. For > that purpose any crazy ordering like 1 < false < 0 < [1, 2, foobar] < > <0.12.34> < 42 < [] < 30 < baz ... would work fine, as long as the place of > every term in this sequence is well defined. There's only one constraint > (apart from not violating the mathematical properties of a strict ordering > relation), and that is to make comparison of terms reasonably efficient (my > above example would fail this check). > > So you feel <<1:16>> should be less than <<2:24>>? That's because you > interpret <<0,1>> and <<0,0,2>> as variable length encoding of integers. If > you have this use case, than write a custom comparison function and sort > your binaries with that. I may have a use case where the same two binaries > mean operation #1 (with no arguments) and operation #0 (with a single > argument 2), so I would have to write a different comparison function (but > as lucky as I am, I may just use erlang:'<'/2 this time, phew!). > > But if you would like to know why I find the built-in ordering of terms > logical, it is because both lists and binaries are usually used to > represent variable sized data, while tuples are for fixed size data. I mean > that {1,2} and {1,2,3} typically describe semantically different things, > e.g. a 2D coordinate and a semver version number. Their relative ordering > is as irrelevant for most applications as the ordering of {1,2} and > <0.0.1>. Ordering by tuple size first makes some sense because it will keep > types together: just like all integers are less than atoms, all 2D > coordinates are less than version numbers. > > Binaries may represent fixed size data (e.g. a packet header), but than it > doesn't matter whether you order lexicographically or size first, because > all binaries in your domain will be the same size. Most often however > binaries represent variable sized data, such as payload in a packet. And > for variable sized data the human convention is lexicographical ordering. A > printed dictionary may sort words by length first, and it would be a > (mostly) usable dictionary, but that's just not how things are by > convention. > > Cheers, > D?niel > > P.S.: By the way, this is why I don't like passing fixed length lists as > arguments for init/1 in gen_* callbacks (such as init([Some, Args, Here]) > -> ...). These should be tuples instead, they are not variable length! > > On Wed, 30 Oct 2019 at 14:35, Valentin Micic wrote: > >> >> On 28 Oct 2019, at 10:02, Raimo Niskanen >> wrote: >> >> I'd like to defend the current term order of lists, tuples and >> binaries. >> >> Lists are compared a'la string comparison, which avoids having to >> traverse the whole list when comparing. Binaries are compared as lists >> since both are used for handling strings, and also to be compared like >> memcmp does it. >> >> Tuples are used for general complex terms. They can be compared size >> first whithout having to be fully traversed, as opposed to lists, which >> is useful in other contexts. You can thereby choose your data >> structure to select comparision. >> -- >> Raimo Niskanen >> >> >> Thanks Raimo, >> >> Your argument seems to have one?well, misconception. I don?t think it >> would be correct to imply that Binaries are used exclusively for handling >> strings, but rather, one of their uses may be that. >> >> For example, here's a syntax that is not anything ?string-like?: >> >> >> (tsdb_1_1@REDACTED)3593> <<1:16>> < <<2:16>>. >> true >> (tsdb_1_1@REDACTED)3594> <<50:16>> < <<36:16>>. >> false >> >> >> As you could see, this pretty much behaves the way we would expect when >> we compare two integer values, e.g. >> >> 1 is less than 2, and so is <<1:16>> < <<2:16>> Technically, we are not >> talking about strings here, but, rather two 16-bit integers. >> >> I can also say that: >> >> (tsdb_1_1@REDACTED)3597> 1 < 000002. >> true >> >> When we write: >> >> (tsdb_1_1@REDACTED)3593> <<1:16>>. >> >> It would be wrong to pretend that we?re actually talking about a strings >> (e.g. in alphanumeric sense). This clearly means that the integer value of >> 1 stored using Big-Endian encoding (e.g. network byte order). >> >> Thus, when we write: <<2:16>> we get <<0,2>>. When we write: <<2:24>> we >> get <<0,0,2>>... these values are *not* intended to be strings, but >> integers. >> So, when we add leading zeroes, we do not change the integer value. >> >> So why is then: >> >> (tsdb_1_1@REDACTED)3600> <<1:16>> < <<2:24>>. >> false >> >> First, we?re clearly use integer syntax to encode integer values, then we >> have the first integer value encoded using 16-bits, and the second integer >> value encoded using 24-bits. >> It just happens so, that 16-bits is used to encode the value of 1, and >> 24-bits to encode the value of two. >> >> Thus, since 16-bits are less then 24-bits (in length), but also 1 is less >> than 2, one may expect this to yield TRUE. Yet somehow, two octets are NOT >> LESS than there, nor 1 is NOT LESS than 2! >> >> I think this cannot pass the "red-face test?, and thus does not deserve >> defending. >> >> Contrast this with the way tuples are handled: >> >> (tsdb_1_1@REDACTED)3666> {1,1} < {1,2}. >> true >> (tsdb_1_1@REDACTED)3667> {1,3} < {1,2}. >> false >> (tsdb_1_1@REDACTED)3668> {1,3} < {1,2,3}. >> true >> >> Considering that Binaries may be used to encode ANYTHING, shouldn?t they >> be handled the same way as tuples instead of: >> >> >> (tsdb_1_1@REDACTED)3624> <<1,1>> < <<1,2>>. >> true >> (tsdb_1_1@REDACTED)3625> <<1,3>> < <<1,2>>. >> false >> (tsdb_1_1@REDACTED)3626> <<1,3>> < <<1,2,3>>. >> false >> >> >> As I said in my previous email, I do not expect Erlang to change, and for >> my "own intents and purposes? I am still considering if: >> >> (tsdb_1_1@REDACTED)3626> <<1,3>> < <<1,2,3>>. >> false >> >> should be given more credence than, say TRUE... if nothing else, because >> two octets are less than three octets. >> >> In other words, if a three-element-tuple, regardless of its content, >> could never be less than any given two-elements-tuple, why shouldn't the >> same hold for Binaries? >> >> Kind regards >> >> V/ >> >> >> -- Brujo Benavides about.me/elbrujohalcon -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo.niskanen@REDACTED Thu Oct 31 11:52:47 2019 From: raimo.niskanen@REDACTED (Raimo Niskanen) Date: Thu, 31 Oct 2019 10:52:47 +0000 Subject: Binary, List and Tuple Inequalities (Paradox?) In-Reply-To: References: <180ae745ba2acccfadafebe278a45155a68ab461.camel@ericsson.com> Message-ID: On Wed, 2019-10-30 at 15:34 +0200, Valentin Micic wrote: > > On 28 Oct 2019, at 10:02, Raimo Niskanen < > > raimo.niskanen@REDACTED> wrote: > > > > I'd like to defend the current term order of lists, tuples and > > binaries. > > > > Lists are compared a'la string comparison, which avoids having to > > traverse the whole list when comparing. Binaries are compared as > > lists > > since both are used for handling strings, and also to be compared > > like > > memcmp does it. > > > > Tuples are used for general complex terms. They can be compared > > size > > first whithout having to be fully traversed, as opposed to lists, > > which > > is useful in other contexts. You can thereby choose your data > > structure to select comparision. > > -- > > Raimo Niskanen > > > > Thanks Raimo, > > Your argument seems to have one?well, misconception. I don?t think it > would be correct to imply that Binaries are used exclusively for > handling strings, but rather, one of their uses may be that. I did not mean to imply that binaries are used exclusevily for handling strings. I merely said that both lists and binaries are used for handling strings, i.e that strings is one use case. Although I confess that I think it is an important use case, if not even a very important use case. When binaries were introduced it was already recognized that having strings as lists is a clumsy representation, especially with 64-bit then on the way with almost double the space waste. So one could foresee that binaries would be used for storing strings. And to not have strings as binaries sort like strings as lists would be a really bad idea for that use case. As for now, Elixir uses binaries as their default representation for strings, so having them sort as strings is vital for Elixir. (Unicode has shot a big hole in that argument, but still true for latin-1) > > For example, here's a syntax that is not anything ?string-like?: > > > (tsdb_1_1@REDACTED)3593> <<1:16>> < <<2:16>>. > true > (tsdb_1_1@REDACTED)3594> <<50:16>> < <<36:16>>. > false > : : > As I said in my previous email, I do not expect Erlang to change, and > for my "own intents and purposes? I am still considering if: > > (tsdb_1_1@REDACTED)3626> <<1,3>> < <<1,2,3>>. > false > > should be given more credence than, say TRUE... if nothing else, > because two octets are less than three octets. > > In other words, if a three-element-tuple, regardless of its content, > could never be less than any given two-elements-tuple, why shouldn't > the same hold for Binaries? If you want to compare terms as integers, then represent them as integers! Erlang bignums can be big enough for most purposes. Erlang does not have operator overloading or subclassing, and has a global term order. So exactly one sort order has to be chosen for a type, and then a guess has to be made of which one is "best" or "most useful". The one chosen for binaries was string sort order, because that is how C compares binaries, and strings seemed to be a very important use case for binaries, which Elixir kind of has proven after the fact. The tuple sort order could have been chosen for binaries, but as I said above that would be bad for the string use case. Also, regarding "useful"; tuple sort order for binaries can easily be implemented by first comparing sizes then comparing content, whilst should you have tulpe sort order and want to implement list sort order you would have to loop byte by byte. Therefore the list sort order can be regarded as more useful since it can be a building block for efficient tuple sort order... Best regards / Raimo Niskanen > > Kind regards > > V/ > > From raoknz@REDACTED Thu Oct 31 20:43:25 2019 From: raoknz@REDACTED (Richard O'Keefe) Date: Fri, 1 Nov 2019 08:43:25 +1300 Subject: Binary, List and Tuple Inequalities (Paradox?) In-Reply-To: References: <180ae745ba2acccfadafebe278a45155a68ab461.camel@ericsson.com> Message-ID: You can represent all sorts of information as binaries, but the binary doesn't know. It may be that you mean <<65,66,67>> to represent "ABC"; it may be that you mean to represent 4276803; it may be that you mean to represent 'play A below middle c for 66 milliseconds on instrument 67'. Your intentions are not part of the binary. So there is no way to have a sort order for binaries (or anything else) that perfectly suits all use cases. Even for strings, English dictionaries and telephone books use *different* orders and neither of them matches strcmp(). For Prolog, I managed in 1984 to come up with a justification for the ordering of terms of different kinds (coherence between @< and subtypes) which the ISO committee chose to ignore. The best we can hope for in a "global" term ordering is that it satisfy the axioms of a total order. That's a useful thing to have. Want something else? Program it. On Thu, 31 Oct 2019 at 23:53, Raimo Niskanen wrote: > > On Wed, 2019-10-30 at 15:34 +0200, Valentin Micic wrote: > > > On 28 Oct 2019, at 10:02, Raimo Niskanen < > > > raimo.niskanen@REDACTED> wrote: > > > > > > I'd like to defend the current term order of lists, tuples and > > > binaries. > > > > > > Lists are compared a'la string comparison, which avoids having to > > > traverse the whole list when comparing. Binaries are compared as > > > lists > > > since both are used for handling strings, and also to be compared > > > like > > > memcmp does it. > > > > > > Tuples are used for general complex terms. They can be compared > > > size > > > first whithout having to be fully traversed, as opposed to lists, > > > which > > > is useful in other contexts. You can thereby choose your data > > > structure to select comparision. > > > -- > > > Raimo Niskanen > > > > > > > Thanks Raimo, > > > > Your argument seems to have one?well, misconception. I don?t think it > > would be correct to imply that Binaries are used exclusively for > > handling strings, but rather, one of their uses may be that. > > I did not mean to imply that binaries are used exclusevily for handling > strings. I merely said that both lists and binaries are used for > handling strings, i.e that strings is one use case. > > Although I confess that I think it is an important use case, if not > even a very important use case. > > When binaries were introduced it was already recognized that having > strings as lists is a clumsy representation, especially with 64-bit > then on the way with almost double the space waste. So one could > foresee that binaries would be used for storing strings. > > And to not have strings as binaries sort like strings as lists would be > a really bad idea for that use case. > > As for now, Elixir uses binaries as their default representation for > strings, so having them sort as strings is vital for Elixir. > (Unicode has shot a big hole in that argument, but still true for > latin-1) > > > > > For example, here's a syntax that is not anything ?string-like?: > > > > > > (tsdb_1_1@REDACTED)3593> <<1:16>> < <<2:16>>. > > true > > (tsdb_1_1@REDACTED)3594> <<50:16>> < <<36:16>>. > > false > > > : > : > > As I said in my previous email, I do not expect Erlang to change, and > > for my "own intents and purposes? I am still considering if: > > > > (tsdb_1_1@REDACTED)3626> <<1,3>> < <<1,2,3>>. > > false > > > > should be given more credence than, say TRUE... if nothing else, > > because two octets are less than three octets. > > > > In other words, if a three-element-tuple, regardless of its content, > > could never be less than any given two-elements-tuple, why shouldn't > > the same hold for Binaries? > > If you want to compare terms as integers, then represent them as > integers! Erlang bignums can be big enough for most purposes. > > Erlang does not have operator overloading or subclassing, and has a > global term order. So exactly one sort order has to be chosen for a > type, and then a guess has to be made of which one is "best" or "most > useful". > > The one chosen for binaries was string sort order, because that is how > C compares binaries, and strings seemed to be a very important use case > for binaries, which Elixir kind of has proven after the fact. > > The tuple sort order could have been chosen for binaries, but as I said > above that would be bad for the string use case. > > Also, regarding "useful"; tuple sort order for binaries can easily be > implemented by first comparing sizes then comparing content, whilst > should you have tulpe sort order and want to implement list sort order > you would have to loop byte by byte. Therefore the list sort order can > be regarded as more useful since it can be a building block for > efficient tuple sort order... > > Best regards > / Raimo Niskanen > > > > > Kind regards > > > > V/ > > > > From v@REDACTED Thu Oct 31 22:03:40 2019 From: v@REDACTED (Valentin Micic) Date: Thu, 31 Oct 2019 23:03:40 +0200 Subject: Binary, List and Tuple Inequalities (Paradox?) In-Reply-To: References: <180ae745ba2acccfadafebe278a45155a68ab461.camel@ericsson.com> Message-ID: <63F2C658-515F-416E-822A-5445EFFB46DC@micic.co.za> > On 31 Oct 2019, at 21:43, Richard O'Keefe wrote: > > You can represent all sorts of information as binaries, > but the binary doesn't know. It may be that you mean > <<65,66,67>> to represent "ABC"; it may be that you > mean to represent 4276803; it may be that you mean > to represent 'play A below middle c for 66 milliseconds > on instrument 67'. Your intentions are not part of the > binary. So there is no way to have a sort order for > binaries (or anything else) that perfectly suits all use > cases. Even for strings, English dictionaries and > telephone books use *different* orders and neither of > them matches strcmp(). > > For Prolog, I managed in 1984 to come up with a > justification for the ordering of terms of different kinds > (coherence between @< and subtypes) which the ISO > committee chose to ignore. > > The best we can hope for in a "global" term ordering is > that it satisfy the axioms of a total order. That's a useful > thing to have. > > Want something else? Program it. Yes, indeed, and I?ve implied that much ? I do want to program it, but I also wanted to know what people think ? if one aspires to program for others, it seems reasonable to consider dominant logic on the issue. I started by thinking that less cannot be more, e.g. two octets cannot be less than three. Now I am increasingly convinced that one should never compare Binaries that are not of the same size. However, I am not so sure what to do if, or rather when, one has to (compare binaries of unequal sizes). So here?s the question: Presuming that one does not know anything about semantic given by data stored in a Binary, but one knows about the size of the binaries being compared, wouldn?t it make sense then to consider that two octets should be less than three? Wouldn?t this approach fit the most general case (independent of what Elixir people do). V/ > > On Thu, 31 Oct 2019 at 23:53, Raimo Niskanen > wrote: >> >> On Wed, 2019-10-30 at 15:34 +0200, Valentin Micic wrote: >>>> On 28 Oct 2019, at 10:02, Raimo Niskanen < >>>> raimo.niskanen@REDACTED> wrote: >>>> >>>> I'd like to defend the current term order of lists, tuples and >>>> binaries. >>>> >>>> Lists are compared a'la string comparison, which avoids having to >>>> traverse the whole list when comparing. Binaries are compared as >>>> lists >>>> since both are used for handling strings, and also to be compared >>>> like >>>> memcmp does it. >>>> >>>> Tuples are used for general complex terms. They can be compared >>>> size >>>> first whithout having to be fully traversed, as opposed to lists, >>>> which >>>> is useful in other contexts. You can thereby choose your data >>>> structure to select comparision. >>>> -- >>>> Raimo Niskanen >>>> >>> >>> Thanks Raimo, >>> >>> Your argument seems to have one?well, misconception. I don?t think it >>> would be correct to imply that Binaries are used exclusively for >>> handling strings, but rather, one of their uses may be that. >> >> I did not mean to imply that binaries are used exclusevily for handling >> strings. I merely said that both lists and binaries are used for >> handling strings, i.e that strings is one use case. >> >> Although I confess that I think it is an important use case, if not >> even a very important use case. >> >> When binaries were introduced it was already recognized that having >> strings as lists is a clumsy representation, especially with 64-bit >> then on the way with almost double the space waste. So one could >> foresee that binaries would be used for storing strings. >> >> And to not have strings as binaries sort like strings as lists would be >> a really bad idea for that use case. >> >> As for now, Elixir uses binaries as their default representation for >> strings, so having them sort as strings is vital for Elixir. >> (Unicode has shot a big hole in that argument, but still true for >> latin-1) >> >>> >>> For example, here's a syntax that is not anything ?string-like?: >>> >>> >>> (tsdb_1_1@REDACTED)3593> <<1:16>> < <<2:16>>. >>> true >>> (tsdb_1_1@REDACTED)3594> <<50:16>> < <<36:16>>. >>> false >>> >> : >> : >>> As I said in my previous email, I do not expect Erlang to change, and >>> for my "own intents and purposes? I am still considering if: >>> >>> (tsdb_1_1@REDACTED)3626> <<1,3>> < <<1,2,3>>. >>> false >>> >>> should be given more credence than, say TRUE... if nothing else, >>> because two octets are less than three octets. >>> >>> In other words, if a three-element-tuple, regardless of its content, >>> could never be less than any given two-elements-tuple, why shouldn't >>> the same hold for Binaries? >> >> If you want to compare terms as integers, then represent them as >> integers! Erlang bignums can be big enough for most purposes. >> >> Erlang does not have operator overloading or subclassing, and has a >> global term order. So exactly one sort order has to be chosen for a >> type, and then a guess has to be made of which one is "best" or "most >> useful". >> >> The one chosen for binaries was string sort order, because that is how >> C compares binaries, and strings seemed to be a very important use case >> for binaries, which Elixir kind of has proven after the fact. >> >> The tuple sort order could have been chosen for binaries, but as I said >> above that would be bad for the string use case. >> >> Also, regarding "useful"; tuple sort order for binaries can easily be >> implemented by first comparing sizes then comparing content, whilst >> should you have tulpe sort order and want to implement list sort order >> you would have to loop byte by byte. Therefore the list sort order can >> be regarded as more useful since it can be a building block for >> efficient tuple sort order... >> >> Best regards >> / Raimo Niskanen >> >>> >>> Kind regards >>> >>> V/ >>> >>>