From Ingela.Anderton.Andin@REDACTED Mon Jun 1 15:32:12 2015 From: Ingela.Anderton.Andin@REDACTED (Ingela Anderton Andin) Date: Mon, 1 Jun 2015 15:32:12 +0200 Subject: [erlang-patches] DTLS patch In-Reply-To: <1093525126.101068.1433085768186.JavaMail.zimbra@tpip.net> References: <1093525126.101068.1433085768186.JavaMail.zimbra@tpip.net> Message-ID: <556C5EDC.2040307@ericsson.com> Hi! See inlined comments, to both Andreas and Haiyangs mails. On 05/31/2015 05:22 PM, Andreas Schultz wrote: > Hi Haiyang, > > I'm sure Ingela will have plenty to say, starting with "Unit Tests". Yes you are absolutely correct we want tests, a good place to start could be a dtls_to_openssl_SUITE corresponding to ssl_to_openssl_SUITE. > A few comments from me: > > 1.) Have you tested your changes with packet loss and reordering, especially > reordering around the the Change Cipher Spec message? > > Keeping the correct cipher context for re-sending old flights and decoding > incoming re-sends across CCS messages is not trivial and from reading the > code I can't convince myself that it would work. > > 2.) I'm not happy with the way new clients are handled. You seem to > insert them into an ETS tables without waiting for the answer to an > HelloVerifyRequest. This opens a way for resource exhaustion and denial > of service attacks. The DTLS RFC is quite explicit that the verify > cookie should be generated stateless and that the server should expend > any memory resource until the Hello was verified successfully. > > 3.) TLS is almost exclusively used in a pure form over TCP, the situation > is different with DTLS where some interesting deviations exist. The TCP > like socket handling is not help full with those. > > For example CAPWAP (RFC 5415) multiplexes DTLS and non DTLS data over the > same UDP socket. Some mechanism to hook between the raw socket and the > DTLS logic is required for that. > > 4.) You have certainly added lots of code, but you have still reused some > ideas and code from my earlier versions. A small attribution wouldn't hurt. I am sure these are all valid comments to consider. Andreas has made many valuable contributions to the ssl application in the past and has a good knowledge in the subject. >> We have a server application needs DTLS protocol support. But the current OTP >> release still has an incomplete DTLS implementation. So we create this patch to >> include DTLS implementation based on current well-designed ssl architecture >> (which we don't consider this as a new feature, just a patch). What we have >> added are: >> >> 1. DTLS transport layer on top of gen_udp >> 2. DTLS flight retransmission and timeout mechanism >> 3. DTLS record fragmentation/defragmentation handling >> >> It looks good that the patch can work with OpenSSL 1.0.2a release. If no one is >> working on dtls now, we would like to have this patch to be reviewed. >> >> Following is the repository contains the patch: >> >> https://github.com/haiyang-yin/otp >> >> To fetch the patch, refer to following git commands: >> >> git clone https://github.com/haiyang-yin/otp.git >> git checkout dtls_patch >> >> Here is code review location: >> >> https://github.com/haiyang-yin/otp/compare/maint...haiyang-yin:dtls_patch >> >> There are two demo programs to show how dtls client/server works in the >> attachments. >> >> Feel free to let me know if further information is needed. We would prefer if you could make it into a pull-request, or maybe several pull-requests. I think the first commit moving functions from tls_connection into ssl_connection could be one pull-request. That can be considered part of the refactoring work that I was doing in order to make our DTLS-implementation (that I alas have not had time to finish due to other things being prioritized). So we could include that as a first step. The other parts I and the OTP team will need some more time to think about your implementation suggestion. But what ever the conclusion will be, it is a way to give this issue some more focus, and hopefully move it forward towards a functioning DLTS implementation. Regards Ingela Erlang/OTP Team Ericssson AB From haiyang.yin@REDACTED Tue Jun 2 03:31:15 2015 From: haiyang.yin@REDACTED (Haiyang Yin) Date: Mon, 1 Jun 2015 21:31:15 -0400 Subject: [erlang-patches] DTLS patch In-Reply-To: <556C5EDC.2040307@ericsson.com> References: <1093525126.101068.1433085768186.JavaMail.zimbra@tpip.net> <556C5EDC.2040307@ericsson.com> Message-ID: Hi Andreas/Ingela, Thanks you very much for your comments and time. I very appreciate it. I have some inline comments marked with [YHY]: 1.) Have you tested your changes with packet loss and reordering, especially reordering around the the Change Cipher Spec message? Keeping the correct cipher context for re-sending old flights and decoding incoming re-sends across CCS messages is not trivial and from reading the code I can't convince myself that it would work. [YHY] No, the change we made was based on existing dtls code and minor changes, e.g. fragmented handshake messages, retransmission of old flights and etc. The existing code dealt with cases specified by RFC on the basis of record sequence number and handshake message sequence number. I agree reordering is much more complicated than current code as more information needs to be cached in order to reorder/reconstruct handshake messages. As we move on, certainly that part in dtls_handshake.erl would be enhanced. Very good point to bring this up. 2.) I'm not happy with the way new clients are handled. You seem to insert them into an ETS tables without waiting for the answer to an HelloVerifyRequest. This opens a way for resource exhaustion and denial of service attacks. The DTLS RFC is quite explicit that the verify cookie should be generated stateless and that the server should expend any memory resource until the Hello was verified successfully. [YHY] Our purpose of designing dtls_transport module is to have a thin protocol-agnostic layer to just pass the datagram packets. Cookie mechanism makes it hard for spoofing DoS attack but no defense from valid IP addresses. You are right the resource leak in ETS table in this case, to overcome this, we can start a timer to wait for CLIENT HELLO and close the socket (remove it from ETS table) for both DoS cases. This provides another way of invalidating the Cookies if the attacker tries to collect a number of cookies from different addresses (See discussion in DTLS RFC). Any comments ? 3.) TLS is almost exclusively used in a pure form over TCP, the situation is different with DTLS where some interesting deviations exist. The TCP like socket handling is not help full with those. For example CAPWAP (RFC 5415) multiplexes DTLS and non DTLS data over the same UDP socket. Some mechanism to hook between the raw socket and the DTLS logic is required for that. [YHY] Sorry I am not familiar with CAPWAP, is it the service discovery (clear text) and later DTLS session established to convey encrypted data ? To me, this might be resolved in dtls_connection module. When raw datagram packets passed to handle_info callback, it can check data based on session state on which the data is encrypted or not. It is flexible but sacrifice the performance in raw socket level. This is very important because it will impact how application developers use the interface. I hope further discussion required for this topic. 4.) You have certainly added lots of code, but you have still reused some ideas and code from my earlier versions. A small attribution wouldn't hurt. [YHY] The existing DTLS code is well-written. We guess it just you didn't have enough time to polish it further. We actually didn't change too much and it works well in sunny cases. Having said that our change is a patch but not a new feature. Regarding the test cases, we will follow ssl_to_openssl_SUITE to add dtls_to_openssl_SUITE later, it is always a good idea to have test cases. This is a start point toward complete DTLS implementation as Ingela mentioned. We are sure more comments and ideas would come out when we dig deep further. We will continue follow up on this, Cheers, /Haiyang On Mon, Jun 1, 2015 at 9:32 AM, Ingela Anderton Andin < Ingela.Anderton.Andin@REDACTED> wrote: > Hi! > > See inlined comments, to both Andreas and Haiyangs mails. > > On 05/31/2015 05:22 PM, Andreas Schultz wrote: > >> Hi Haiyang, >> >> I'm sure Ingela will have plenty to say, starting with "Unit Tests". >> > > Yes you are absolutely correct we want tests, a good place to start > could be a dtls_to_openssl_SUITE corresponding to ssl_to_openssl_SUITE. > > A few comments from me: >> >> 1.) Have you tested your changes with packet loss and reordering, >> especially >> reordering around the the Change Cipher Spec message? >> >> Keeping the correct cipher context for re-sending old flights and decoding >> incoming re-sends across CCS messages is not trivial and from reading the >> code I can't convince myself that it would work. >> >> 2.) I'm not happy with the way new clients are handled. You seem to >> insert them into an ETS tables without waiting for the answer to an >> HelloVerifyRequest. This opens a way for resource exhaustion and denial >> of service attacks. The DTLS RFC is quite explicit that the verify >> cookie should be generated stateless and that the server should expend >> any memory resource until the Hello was verified successfully. >> >> 3.) TLS is almost exclusively used in a pure form over TCP, the situation >> is different with DTLS where some interesting deviations exist. The TCP >> like socket handling is not help full with those. >> >> For example CAPWAP (RFC 5415) multiplexes DTLS and non DTLS data over the >> same UDP socket. Some mechanism to hook between the raw socket and the >> DTLS logic is required for that. >> >> 4.) You have certainly added lots of code, but you have still reused some >> ideas and code from my earlier versions. A small attribution wouldn't >> hurt. >> > > I am sure these are all valid comments to consider. Andreas has made many > valuable contributions to the ssl application in the past and has a good > knowledge in the subject. > > We have a server application needs DTLS protocol support. But the current >>> OTP >>> release still has an incomplete DTLS implementation. So we create this >>> patch to >>> include DTLS implementation based on current well-designed ssl >>> architecture >>> (which we don't consider this as a new feature, just a patch). What we >>> have >>> added are: >>> >>> 1. DTLS transport layer on top of gen_udp >>> 2. DTLS flight retransmission and timeout mechanism >>> 3. DTLS record fragmentation/defragmentation handling >>> >>> It looks good that the patch can work with OpenSSL 1.0.2a release. If no >>> one is >>> working on dtls now, we would like to have this patch to be reviewed. >>> >>> Following is the repository contains the patch: >>> >>> https://github.com/haiyang-yin/otp >>> >>> To fetch the patch, refer to following git commands: >>> >>> git clone https://github.com/haiyang-yin/otp.git >>> git checkout dtls_patch >>> >> >> > >> Here is code review location: >>> >>> https://github.com/haiyang-yin/otp/compare/maint...haiyang-yin:dtls_patch >>> >>> There are two demo programs to show how dtls client/server works in the >>> attachments. >>> >>> Feel free to let me know if further information is needed. >>> >> > We would prefer if you could make it into a pull-request, or maybe several > pull-requests. I think the first commit moving functions from > tls_connection into ssl_connection could be one pull-request. That can be > considered part of the refactoring work that I was doing in order to make > our DTLS-implementation (that I alas have not had time to finish due to > other things being prioritized). So we could include that as a first step. > > The other parts I and the OTP team will need some more time to think about > your implementation suggestion. But what ever the conclusion will > be, it is a way to give this issue some more focus, and hopefully move it > forward towards a functioning DLTS implementation. > > Regards Ingela Erlang/OTP Team Ericssson AB > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger@REDACTED Tue Jun 2 17:22:00 2015 From: holger@REDACTED (Holger =?iso-8859-1?Q?Wei=DF?=) Date: Tue, 2 Jun 2015 17:22:00 +0200 Subject: [erlang-patches] Make host name lookups case-insensitive Message-ID: <20150602152200.GA559060@zedat.fu-berlin.de> Don't let inet_res:getbyname and inet_res:gethostbyname calls return {error, nxdomain} if the host name capitalization in the DNS response differs from the request, like in this example: 1> inet_res:gethostbyname("erlang.org"). {ok,{hostent,"erlang.org",[],inet,4,[{192,121,151,106}]}} 2> inet_res:gethostbyname("Erlang.ORG"). {error,nxdomain} The following PR makes sure these lookups are performed in a case-insensitive manner: git fetch git://github.com/weiss/otp.git case-insensitive-lookups Links: https://github.com/weiss/otp/compare/erlang:maint...case-insensitive-lookups https://github.com/weiss/otp/compare/erlang:maint...case-insensitive-lookups.patch https://github.com/erlang/otp/pull/763 From tuncer.ayaz@REDACTED Thu Jun 4 20:18:07 2015 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Thu, 4 Jun 2015 20:18:07 +0200 Subject: [erlang-patches] Proposal: reinstate pre-built PLT for Dialyzer In-Reply-To: References: Message-ID: On Thu, Jan 23, 2014 at 5:34 PM, Tuncer Ayaz wrote: > On Thu, Jan 23, 2014 at 5:19 PM, Sean Cribbs wrote: >> On Thu, Jan 23, 2014 at 9:56 AM, Tuncer Ayaz wrote: >>> >>> BTW, assuming this gets off the ground, would you suggest >>> distributing the PLT with the tarball or in the git tree? I think >>> the src tarball and maybe a separate tarball make more sense. For >>> reference, my local R16B03 PLT is 4MB. This would also be in line >>> with other generated files like configure scripts (what's generated >>> vs what's edited). In the git tree it would only make sense to be >>> included in a tagged tree, but that may be confusing, so release >>> tarballs make the most sense to me. >> >> >> Neither. If you build OTP from source, it would be built during the >> normal build/install process. If you download a prebuilt binary >> package (Windows or something from ESL or a distribution >> maintainer), the PLT should be included in that package. > > That sounds good to me. bump? From deadzen@REDACTED Fri Jun 5 21:32:22 2015 From: deadzen@REDACTED (DeadZen) Date: Fri, 5 Jun 2015 15:32:22 -0400 Subject: [erlang-patches] Add net_kernel and epmd support for multiple simultaneous distributed transport protocols In-Reply-To: References: Message-ID: What is the status of this? thanks, Pedram On Wed, Jan 15, 2014 at 4:58 AM, Kenneth Lundin wrote: > Hi Serge, > > See embedded comments at the end. > > > On Thu, Jan 9, 2014 at 9:26 PM, Serge Aleynikov wrote: >> >> Kenneth, >> >> Thank you for the thorough response! Your reasoning is appealing and I'd >> like to engage in the discussion on how we can move on to include this >> functionality in the distribution, as I believe the community will benefit >> greatly from being able to run an Erlang node listening on multiple >> transports concurrently. >> >> The changes in the patch (https://github.com/erlang/otp/pull/121) cover >> the following areas: >> >> 1. Extension of the EPMD distributed protocol (here we can safely add new >> set of commands while being fully backward compatible) >> erts/doc/src/erl_dist_protocol.xml >> >> 2. Modification of the EPMD daemon to support the new protocol version and >> to keep track of the multiple protocols/ports per node (this is also done in >> a backward compatible way. Nodes supporting new protocol version communicate >> using new spec, and old nodes communicate using old spec) >> erts/epmd/src/epmd.h >> erts/epmd/src/epmd.c >> >> 3. Modification of the EPMD command-line tool to talk to the local EPMD >> daemon using the new protocol version (the command-line tool uses new >> protocol version. This is reasonable, since it's installed together with the >> version of VM supporting the new EPMD protocol features) >> erts/epmd/src/epmd_cli.c >> erts/epmd/src/epmd_cli.c >> erts/epmd/src/epmd_int.h >> erts/epmd/src/epmd_srv.c >> >> 4. Updates of the UDS distribution example code. I believe the example in >> the current release is outdated (I was unable to compile it), but I patched >> it anyway to illustrate the usage of the new protocol) >> lib/kernel/examples/uds_dist/src/dist_selector.erl >> lib/kernel/examples/uds_dist/src/uds_dist.erl >> lib/kernel/examples/uds_dist/src/uds_server.erl >> >> 5. Change of the net_kernel transport pluggable architecture to allow >> support of the new EPMD protocol. >> 5.1 Net_kernel support for the new protocol. >> 5.2 TCPv4 support of the new protocol >> 5.3 TCPv6 support of the new protocol >> 5.4 TLS support of the new protocol >> lib/kernel/src/net_kernel.erl >> lib/kernel/src/dist_util.erl >> lib/kernel/src/inet6_tcp_dist.erl >> lib/kernel/src/inet_tcp_dist.erl >> lib/kernel/src/net_kernel.erl >> lib/ssl/src/inet_tls_dist.erl >> lib/ssl/src/ssl_tls_dist_proxy.erl >> lib/kernel/doc/src/net_kernel.xml >> >> It seems to me that we could roll it out in two phases: >> >> 1. Implement 1,2,3 in the first phase. This would allow to deploy a >> version with EPMD supporting old and new protocols. >> 2. Implement changes to net_kernel and transports (4,5) to be able to talk >> to new EPMDs >> >> If agreed, I can break up my patch in two, corresponding to this plan. >> >> What do you think? > > > I think it is a good idea to roll out this in 2 or more phases but I think > we need think a bit more about this before we can give a more feedback on > exactly what to include in the different phases. > > Will be back with more comments/suggestions soon. > > Regards Kenneth >> >> >> Serge >> >> >> >> On Thu, Jan 9, 2014 at 10:53 AM, Kenneth Lundin >> wrote: >>> >>> On Fri, Jan 3, 2014 at 4:30 PM, Serge Aleynikov >>> wrote: >>>> >>>> I'd like to inquire the status of the following pull request: >>>> https://github.com/erlang/otp/pull/121 >>>> >>>> Has it been reviewed? >>>> >>>> Thanks! >>>> >>>> Serge >>>> >>> Hi Serge, >>> >>> We have discussed your contribution at the OTP technical board and here >>> are the results: >>> >>> Inclusion in OTP >>> ----------------------- >>> >>> Your patch touches many source files which are central for the Erlang >>> distribution and we feel it would require quite a lot of work from our side >>> as well to assure that nothing in the current distribution gets broken or >>> incompatible. It would also require quite deep involvement from us giving >>> more detailed feedback regarding interfaces, semantics etc. We can not >>> prioritize this at the moment so we will not include your contribution now >>> (which means REJECT for now). >>> >>> But we really want to add this kind of functionality and want to >>> encourage you (and others) to continue the good work in this area for >>> possible inclusion in OTP at a later stage. See more info and thoughts >>> below. >>> >>> >>> General thoughts >>> ----------------------- >>> >>> Enhancing/extending the Erlang distribution to support heterogenous >>> communication protocols is something we have discussed many times and it is >>> something that we really would like to have. >>> >>> 1) We would like a plugin architecture where different transport >>> protocols can be easily plugged in without having to change the basic Erlang >>> distribution or the VM. >>> >>> 2) It should be possible to run different protocols towards different >>> nodes >>> >>> 3) should be possible to have different timeouts (NET_TICK_TIME) per >>> connection >>> >>> 4) interesting protocols would be UDS, TCP, TLS over TCP, SCTP, ... >>> >>> 5) it should be possible to look up nodes in different ways and maybe not >>> only via contacting epmd on the host name which is part of the node name. >>> I.e having the hostname as part of the node name is not always what you >>> want, depending on protocol to use. >>> >>> .... >>> >>> Your specific contribution >>> ----------------------------------- >>> In your pull request you have implemented fully or partly 1,2, 4, and we >>> see it as a good start. >>> >>> A general comment is that the diffs are showing differences on many >>> places in the code where you actually haven't changed anything except tabs, >>> line breaks and spaces. This makes it more difficult to sort out what you >>> have changed and added. We would like a clean diff without unnecessary >>> touches lines which has no relevance for the new function. If a formatting >>> change is needed or suggested it should be a separate commit/pull request. >>> >>> Regarding details in your code we have not had the time to take a deeper >>> look but from a quick glance it looks good. The questionmarks are more about >>> how to introduce this new functionality in a safe way and what more is >>> needed that you have not addressed so far. >>> >>> >>> Suggested way forward >>> ------------------------------ >>> >>> I think that one approach that would make it easier to include this kind >>> of functionality soon would be to make minimal changes to the existing >>> distribution code and only add or change so that it is possible to support >>> heterogenous protocols with code that initially can live as a separate >>> extension that don't need to be part of the OTP distro. >>> In this way the functionality can be further developed and used together >>> with an ordinary OTP distro (unpatched) and can easily be included later >>> (hopefully within a year). >>> >>> Maybe you think that you already have done those minimal changes, but I >>> suggest you take another look and see if it is possible to find a way to >>> make minimal or now changes to net_kernel and epmd etc. which we could >>> include and then have the rest as a separate "application"/extension project >>> at github that can be included later. >>> >>> Just some thoughts: >>> Maybe let the old distribution work as before and let the user choose >>> "new_distro" in some way and will only then get to run the new code which >>> allows heterogenous protocols another new EPMD or an EPMD written in Erlang >>> or allow for plugins of different EPMD implementations that can run in >>> parallell. The communication towards EPMD maybe needs to be secure as well >>> and then it might be much easier to implement in Erlang. >>> >>> I (we) are more than willing to discuss more about how to find this >>> minimal patch which we think we can "safely" include and which makes it >>> possible to develop and use the rest as pure extensions to OTP in a first >>> step. >>> >>> Regards Kenneth, Erlang/OTP Ericsson >> >> > > > _______________________________________________ > erlang-patches mailing list > erlang-patches@REDACTED > http://erlang.org/mailman/listinfo/erlang-patches > From max.lapshin@REDACTED Sat Jun 6 14:13:47 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 6 Jun 2015 15:13:47 +0300 Subject: [erlang-patches] DTLS patch In-Reply-To: References: <1093525126.101068.1433085768186.JavaMail.zimbra@tpip.net> <556C5EDC.2040307@ericsson.com> Message-ID: I want to connect webrtc to erlang. Will this implementation help me to launch udp listener? -------------- next part -------------- An HTML attachment was scrubbed... URL: From haiyang.yin@REDACTED Tue Jun 9 03:39:04 2015 From: haiyang.yin@REDACTED (Haiyang Yin) Date: Mon, 8 Jun 2015 21:39:04 -0400 Subject: [erlang-patches] DTLS patch In-Reply-To: References: <1093525126.101068.1433085768186.JavaMail.zimbra@tpip.net> <556C5EDC.2040307@ericsson.com> Message-ID: Hi Max, Do you mean to launch udp listener for webrtc, which will use DTLS for data transfer ? or something else ? /Haiyang On Sat, Jun 6, 2015 at 8:13 AM, Max Lapshin wrote: > I want to connect webrtc to erlang. Will this implementation help me to > launch udp listener? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue Jun 9 07:26:11 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 9 Jun 2015 08:26:11 +0300 Subject: [erlang-patches] DTLS patch In-Reply-To: References: <1093525126.101068.1433085768186.JavaMail.zimbra@tpip.net> <556C5EDC.2040307@ericsson.com> Message-ID: I think, yes. Or maybe convert udp socket to a DTLS listener. -------------- next part -------------- An HTML attachment was scrubbed... URL: From haiyang.yin@REDACTED Fri Jun 12 00:41:40 2015 From: haiyang.yin@REDACTED (Haiyang Yin) Date: Thu, 11 Jun 2015 18:41:40 -0400 Subject: [erlang-patches] DTLS patch In-Reply-To: References: <1093525126.101068.1433085768186.JavaMail.zimbra@tpip.net> <556C5EDC.2040307@ericsson.com> Message-ID: The patch I provided could do what you want. However, that needs to be discussed by OTP team regarding the design according to Ingela. Since I saw not only you ask this kind of questions, I hope the DTLS support for Erlang could get attention sooner or later :). No matter what change can be made eventually. On Tue, Jun 9, 2015 at 1:26 AM, Max Lapshin wrote: > I think, yes. Or maybe convert udp socket to a DTLS listener. > -------------- next part -------------- An HTML attachment was scrubbed... URL: