From swvist@REDACTED Sun Nov 8 16:01:54 2015 From: swvist@REDACTED (Vipin Nair) Date: Sun, 8 Nov 2015 20:31:54 +0530 Subject: [erlang-patches] EEP Light: Option to override private key lookup when using Erlang ssh client Message-ID: EEP Light: Option to override private key lookup when using Erlang ssh client * Why do we need this new feature? Currently, ssh:connect/3,4 gets the user's private key from either the default directory (~/.ssh) of the user or from the directory specified in the 'user_dir' ssh connect option. The files names are always assumed to be openssh default file names, for example, `id_rsa` or `id_dsa`. This makes the API inflexible as we are required to create a directory and write the key data to the file system when we are connecting to a SSH server as a third party. An API to specify the private key is immensely useful when the private keys are stored in, say, a database. * How did you solve it? - Implementation description To address the above problem, new ssh connect options `rsa_priv_key` and `dsa_priv_key` are proposed. The connect API will look like: ssh:connect("localhost", 22, [{user, "[USERNAME]"}, {rsa_priv_key, <<"[BINARY_KEY_DATA]">>}]). If `rsa_priv_key` is specified and RSA is used for ssh negotiation, instead of reading the key from the `id_rsa` file, we use the key specified above. (This key could be protected by a passphrase and it is the system's responsibility to decrypt it.) `dsa_priv_key` will work similarly. See below for handling of ECDSA keys and known_hosts file. A pull request with a partial implementation of this proposal along with tests is on github[1]. - Why you choose to implement it the way you did? The above solution is in the similar vein as some of the existing API's. For example, when password authentication is used to connect to a server, the ssh connect option `password` is used to specify the password. Similarly, for encrypted private keys, the passphrase is specified using ssh connect options `rsa_pass_phrase` and `dsa_pass_phrase`. - Other possible implementations considered Binary(iodata) key data can be passed onto file:open with `ram` mode specified which lets us treat an object in memory as a file. We can build a solution around this feature. SSH connect option only lets specify the user directory using the `user_dir` option and *not* the individual files. File names are assumed to be openssh defaults. We could change user dir to also accept a proplist of individual file names/iodata. Current definition -> user_dir() :: string() Proposed definition -> user_dir() :: string() | [{key_file() :: string() | iodata()}] Where key_file() :: id_rsa | id_dsa | id_ecdsa | known_hosts Considering we can pass either the file name or iodata with key contents, we'll have to limit file API usage to the ones that work with both of them. For example, ssh_file:read_ssh_file/1 uses file:read_file/1 which is not compatible with iodata() input. I haven't given enough thought to this solution so it is possible that I could be missing something obvious. In my opinion, the first solution proposed above is cleaner way to solve this problem. * Risks or uncertain artifacts? - Describe known incompatibilities Introducing the new connect options does not introduce an incompatibility, is fully backward compatible and does not affect consumers of the existing API in any way. - Describe uncertainties that are not fully understood or unclear. ECDSA keys can be handled by introducing similar options above. The proposed options names are `ecdsa_sha2_nistp256_priv_key`, `ecdsa_sha2_nistp384_priv_key` and `ecdsa_sha2_nistp521_priv_key`. A new option `known_hosts` can be introduced for specifying known_hosts as an API. The value corresponding to this key would be a list of known server public keys. Some background: I am implementing a ssh multiplexer[2] and as a consumer of this API, I wasn't very happy. I thought about what a better API (IMO) could look like and took a stab at implementing it. [1]: https://github.com/erlang/otp/pull/884 [2]: https://github.com/swvist/hss -- Regards, Vipin From hans.r.nilsson@REDACTED Mon Nov 9 18:10:14 2015 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Mon, 9 Nov 2015 18:10:14 +0100 Subject: [erlang-patches] EEP Light: Option to override private key lookup when using Erlang ssh client In-Reply-To: References: Message-ID: <5640D376.9010409@ericsson.com> On 11/08/2015 04:01 PM, Vipin Nair wrote: > EEP Light: Option to override private key lookup when using Erlang ssh client > > > * Why do we need this new feature? > > Currently, ssh:connect/3,4 gets the user's private key from either the > default directory (~/.ssh) of the user or from the directory specified in the > 'user_dir' ssh connect option. The files names are always assumed to be > openssh default file names, for example, `id_rsa` or `id_dsa`. This makes the > API inflexible as we are required to create a directory and write the key > data to the file system when we are connecting to a SSH server as a third > party. An API to specify the private key is immensely useful when the > private keys are stored in, say, a database. You seem unaware of that both the client and the server get the keys from callbacks defined by the ssh_client_key_api and the ssh_server_key_api. The option key_cb defines the module to use for ssh:connect or ssh:daemon. The default callback module is ssh_file, which implements reading and writing of keys to and from files compatible with OpenSSH. You can write plugins to fetch keys from a database, fetch from files with names of your choice or even hard code them using those behaviours. See http://www.erlang.org/doc/man/ssh_client_key_api.html http://www.erlang.org/doc/man/ssh_server_key_api.html -Hans > > > * How did you solve it? > > - Implementation description > > To address the above problem, new ssh connect options `rsa_priv_key` and > `dsa_priv_key` are proposed. The connect API will look like: > > ssh:connect("localhost", 22, [{user, "[USERNAME]"}, > {rsa_priv_key, > <<"[BINARY_KEY_DATA]">>}]). > > If `rsa_priv_key` is specified and RSA is used for ssh negotiation, > instead of reading the key from the `id_rsa` file, we use the key > specified above. (This key could be protected by a passphrase and it is > the system's responsibility to decrypt it.) `dsa_priv_key` will work > similarly. See below for handling of ECDSA keys and known_hosts file. > > A pull request with a partial implementation of this proposal along with > tests is on github[1]. > > - Why you choose to implement it the way you did? > > The above solution is in the similar vein as some of the existing API's. > For example, when password authentication is used to connect to a server, > the ssh connect option `password` is used to specify the password. > Similarly, for encrypted private keys, the passphrase is specified using > ssh connect options `rsa_pass_phrase` and `dsa_pass_phrase`. > > - Other possible implementations considered > > Binary(iodata) key data can be passed onto file:open with `ram` mode > specified which lets us treat an object in memory as a file. We can build > a solution around this feature. > > SSH connect option only lets specify the user directory using the > `user_dir` option and *not* the individual files. File names are assumed > to be openssh defaults. We could change user dir to also accept a > proplist of individual file names/iodata. > > Current definition -> user_dir() :: string() > > Proposed definition -> user_dir() :: string() | > [{key_file() :: string() | iodata()}] > > Where > > key_file() :: id_rsa | id_dsa | id_ecdsa | > known_hosts > > Considering we can pass either the file name or iodata with key contents, > we'll have to limit file API usage to the ones that work with both of > them. For example, ssh_file:read_ssh_file/1 uses file:read_file/1 which > is not compatible with iodata() input. > > I haven't given enough thought to this solution so it is possible that I > could be missing something obvious. In my opinion, the first solution > proposed above is cleaner way to solve this problem. > > > * Risks or uncertain artifacts? > > - Describe known incompatibilities > > Introducing the new connect options does not introduce an > incompatibility, is fully backward compatible and does not affect > consumers of the existing API in any way. > > - Describe uncertainties that are not fully understood or unclear. > > ECDSA keys can be handled by introducing similar options above. The > proposed options names are `ecdsa_sha2_nistp256_priv_key`, > `ecdsa_sha2_nistp384_priv_key` and `ecdsa_sha2_nistp521_priv_key`. > > A new option `known_hosts` can be introduced for specifying known_hosts > as an API. The value corresponding to this key would be a list of known > server public keys. > > > > Some background: I am implementing a ssh multiplexer[2] and as a consumer of > this API, I wasn't very happy. I thought about what a better API (IMO) could > look like and took a stab at implementing it. > > [1]: https://github.com/erlang/otp/pull/884 > [2]: https://github.com/swvist/hss > > From nico.kruber@REDACTED Mon Nov 9 20:50:25 2015 From: nico.kruber@REDACTED (Nico Kruber) Date: Mon, 09 Nov 2015 20:50:25 +0100 Subject: [erlang-patches] [jinterface] fix regression of 4390e43558 writing small compressed values Message-ID: <30791541.3NBd8K3vbM@csr-pc40> Hi, we came across a regression in the JInterface code which slipped in for Erlang 17.5 iirc. It prevents the Java code to compress e.g. the string "{}" the way, erlang:term_to_binary("{}", [compressed]) does. The patch is rather simple and reverts the changes of 4390e43558 to the OtpOutputStream class and adds appropriate test cases to prevent further breakages. https://github.com/erlang/otp/compare/maint...NicoK:maint https://github.com/erlang/otp/compare/maint...NicoK:maint.patch Nico -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: This is a digitally signed message part. URL: From vladdu55@REDACTED Mon Nov 9 20:57:46 2015 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Mon, 9 Nov 2015 20:57:46 +0100 Subject: [erlang-patches] [jinterface] fix regression of 4390e43558 writing small compressed values In-Reply-To: <30791541.3NBd8K3vbM@csr-pc40> References: <30791541.3NBd8K3vbM@csr-pc40> Message-ID: Hi Nico, Yes, indeed, that slipped through. I'm sorry if it caused you any inconvenience. Please open a pull request with that change. regards, Vlad On Mon, Nov 9, 2015 at 8:50 PM, Nico Kruber wrote: > Hi, > we came across a regression in the JInterface code which slipped in for > Erlang > 17.5 iirc. It prevents the Java code to compress e.g. the string "{}" the > way, > erlang:term_to_binary("{}", [compressed]) does. The patch is rather simple > and > reverts the changes of 4390e43558 to the OtpOutputStream class and adds > appropriate test cases to prevent further breakages. > > > https://github.com/erlang/otp/compare/maint...NicoK:maint > https://github.com/erlang/otp/compare/maint...NicoK:maint.patch > > > > > Nico > _______________________________________________ > erlang-patches mailing list > erlang-patches@REDACTED > http://erlang.org/mailman/listinfo/erlang-patches > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From swvist@REDACTED Mon Nov 9 19:58:58 2015 From: swvist@REDACTED (Vipin Nair) Date: Tue, 10 Nov 2015 00:28:58 +0530 Subject: [erlang-patches] EEP Light: Option to override private key lookup when using Erlang ssh client In-Reply-To: <5640D376.9010409@ericsson.com> References: <5640D376.9010409@ericsson.com> Message-ID: Hello Hans, > You seem unaware of that both the client and the server get the keys > from callbacks defined by the ssh_client_key_api and the > ssh_server_key_api. The option key_cb defines the module to use for > ssh:connect or ssh:daemon. > [.....] > > See > http://www.erlang.org/doc/man/ssh_client_key_api.html > http://www.erlang.org/doc/man/ssh_server_key_api.html I am aware of this but it is very limited and will not fit my usecase. Let me give you an example of my usecase and demonstrate the limitation of the above API: I have to connect to hosts `host1` and `host2` as user `foo`. I have the private key corresponding to each user/host pair in a database. For simplicity sake, let's assume I can call a named process (PRIV_KEY_SERVER) that will return the private key data for a particular user/host pair. Something like: {ok, PrivKey} = gen_server:call(PRIV_KEY_SERVER, {get_priv_key, User, Host}). Let's say we have a module(KeyCB below) that implements the ssh_client_key_api. The definition could look like: KeyCB:user_key(Algorithm, ConnectOptions) -> {ok, PrivKey} = gen_server:call(PRIV_KEY_SERVER, {get_priv_key, User, Host}). This function cannot be implemented because I have no way to get the value of `Host`. The the ssh:connect/3,4 options that are passed as `ConnectOptions` to user_key/2 do have that value. Now: * I do not have the host name available and since host names need not be unique across machines, I have no way to get the private key corresponding to a user *and* a particular host from, say, a database. * ssh:connect/3,4 validates the connect options so we cannot just add an extra {host, "[HOSTNAME]"} to the connect options and get around the above limitation. * In my opinion, calling a database (or a named process like in the example above) from a module implementing ssh_client_key_api creates a high level of coupling that I would want to avoid when possible. Pragmatic way to do this would be: USER_INPUT -> FETCH_RELEVANT_DATA -> CREATE_SSH_CONN -> POST_CONNECT_STUFF This flows helps with error handling better. Let's say the named process dies for some reason, I do not wish to handle those kinds of error in my module implementing ssh_client_key_api. I want to keep external (Non SSH) IO elsewhere. -- Regards, Vipin From hans.r.nilsson@REDACTED Thu Nov 12 17:36:29 2015 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Thu, 12 Nov 2015 17:36:29 +0100 Subject: [erlang-patches] EEP Light: Option to override private key lookup when using Erlang ssh client In-Reply-To: References: <5640D376.9010409@ericsson.com> Message-ID: <5644C00D.4020407@ericsson.com> Hi Vipin, As I indicated when I closed your pull request, those options forces the callback modules to handle options that might be strange for a certain callback module. For example, the purpose of ssh_file is only to handle keys in the way of OpenSSH. You have correctly pointed out that to be able to write a specialized callback module there is a need to have options passed to callbacks. A more general way to pass options to the key callback module would be to introduce a second form of the key_cb option: - keep todays {key_cb, Module::atom()} to keep compatibility - introduce the variant {key_cb, {Module::atom(), ModuleOptions}} ModuleOptions has the usual list of {Tag,Value} format. That list is passed in the Options argument of the callback functions as {key_cb_private,ModuleOptions} The old {key_cb, Module::atom()} could be transformed to {key_cb, {Module,[]}} already in ssh:handle_options to make it cleaner inside. In that way it is possible to write a callback module with whatever is needed as private options. -Hans -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4101 bytes Desc: S/MIME Cryptographic Signature URL: From swvist@REDACTED Thu Nov 12 18:31:12 2015 From: swvist@REDACTED (Vipin Nair) Date: Thu, 12 Nov 2015 23:01:12 +0530 Subject: [erlang-patches] EEP Light: Option to override private key lookup when using Erlang ssh client In-Reply-To: <5644C00D.4020407@ericsson.com> References: <5640D376.9010409@ericsson.com> <5644C00D.4020407@ericsson.com> Message-ID: Hello Hans, > A more general way to pass options to the key callback module would be > to introduce a second form of the key_cb option: > - keep todays {key_cb, Module::atom()} to keep compatibility > - introduce the variant {key_cb, {Module::atom(), ModuleOptions}} > [...] > In that way it is possible to write a callback module with whatever is > needed as private options. This will work! In retrospect, I should have done this instead. This is much simpler and solves my problem. I'll send a new pull request with the suggested implementation. Thanks! -- Regards, Vipin From hans.r.nilsson@REDACTED Thu Nov 12 19:19:57 2015 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Thu, 12 Nov 2015 19:19:57 +0100 Subject: [erlang-patches] EEP Light: Option to override private key lookup when using Erlang ssh client In-Reply-To: References: <5640D376.9010409@ericsson.com> <5644C00D.4020407@ericsson.com> Message-ID: <5644D84D.1060101@ericsson.com> Great! If there also is documentation and a test case the likelihood of acceptance is higher :) -Hans On 11/12/2015 06:31 PM, Vipin Nair wrote: > Hello Hans, > >> A more general way to pass options to the key callback module would be >> to introduce a second form of the key_cb option: >> - keep todays {key_cb, Module::atom()} to keep compatibility >> - introduce the variant {key_cb, {Module::atom(), ModuleOptions}} >> [...] >> In that way it is possible to write a callback module with whatever is >> needed as private options. > > This will work! In retrospect, I should have done this instead. This is much > simpler and solves my problem. I'll send a new pull request with the suggested > implementation. Thanks! > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4101 bytes Desc: S/MIME Cryptographic Signature URL: From swvist@REDACTED Thu Nov 12 19:52:19 2015 From: swvist@REDACTED (Vipin Nair) Date: Fri, 13 Nov 2015 00:22:19 +0530 Subject: [erlang-patches] EEP Light: Option to override private key lookup when using Erlang ssh client In-Reply-To: <5644D84D.1060101@ericsson.com> References: <5640D376.9010409@ericsson.com> <5644C00D.4020407@ericsson.com> <5644D84D.1060101@ericsson.com> Message-ID: > Great! If there also is documentation and a test case the likelihood of > acceptance is higher :) Noted :) -- Regards, Vipin From hans.r.nilsson@REDACTED Thu Nov 12 20:56:19 2015 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Thu, 12 Nov 2015 20:56:19 +0100 Subject: [erlang-patches] EEP Light: Option to override private key lookup when using Erlang ssh client In-Reply-To: References: <5640D376.9010409@ericsson.com> <5644C00D.4020407@ericsson.com> <5644D84D.1060101@ericsson.com> Message-ID: <5644EEE3.2060901@ericsson.com> I would recommend the list erlang-questions@REDACTED for your question, see http://erlang.org/mailman/listinfo/erlang-questions -Hans On 11/12/2015 08:01 PM, Irina Guberman wrote: > Hello dear Erlang developers, > > I would tremendously appreciate any help or if anyone can point me to > someone who can help with linking Erlang/OTP. I created an > experimental BIF in my own fork of erlang/otp, but can't get past > linking stage (on Mac: Undefined symbols for architecture x86_64, but > same thing happens on ubuntu 14.04) even though I checked everywhere > and my BIF seems to be everywhere an existing BIF from the same module > is (with same naming conventions and all). I don't know where to post > this question either (Erlang Central didn't work which is > understandable, it's not like people are creating BIFs everyday), I'm so > desperate I'm about to post it on Twitter ;) > > Thanks a million for reading this message and even more so for pointing > me to where I can get help with this! > > Irina. > > > > > On Thu, Nov 12, 2015 at 12:19 PM, Hans Nilsson R > > wrote: > > Great! If there also is documentation and a test case the likelihood of > acceptance is higher :) > -Hans > > On 11/12/2015 06:31 PM, Vipin Nair wrote: > > Hello Hans, > > > >> A more general way to pass options to the key callback module > would be > >> to introduce a second form of the key_cb option: > >> - keep todays {key_cb, Module::atom()} to keep compatibility > >> - introduce the variant {key_cb, {Module::atom(), ModuleOptions}} > >> [...] > >> In that way it is possible to write a callback module with > whatever is > >> needed as private options. > > > > This will work! In retrospect, I should have done this instead. > This is much > > simpler and solves my problem. I'll send a new pull request with > the suggested > > implementation. Thanks! > > > > > _______________________________________________ > erlang-patches mailing list > erlang-patches@REDACTED > http://erlang.org/mailman/listinfo/erlang-patches > > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4101 bytes Desc: S/MIME Cryptographic Signature URL: From irina.guberman@REDACTED Thu Nov 12 20:01:32 2015 From: irina.guberman@REDACTED (Irina Guberman) Date: Thu, 12 Nov 2015 13:01:32 -0600 Subject: [erlang-patches] EEP Light: Option to override private key lookup when using Erlang ssh client In-Reply-To: <5644D84D.1060101@ericsson.com> References: <5640D376.9010409@ericsson.com> <5644C00D.4020407@ericsson.com> <5644D84D.1060101@ericsson.com> Message-ID: Hello dear Erlang developers, I would tremendously appreciate any help or if anyone can point me to someone who can help with linking Erlang/OTP. I created an experimental BIF in my own fork of erlang/otp, but can't get past linking stage (on Mac: Undefined symbols for architecture x86_64, but same thing happens on ubuntu 14.04) even though I checked everywhere and my BIF seems to be everywhere an existing BIF from the same module is (with same naming conventions and all). I don't know where to post this question either (Erlang Central didn't work which is understandable, it's not like people are creating BIFs everyday), I'm so desperate I'm about to post it on Twitter ;) Thanks a million for reading this message and even more so for pointing me to where I can get help with this! Irina. On Thu, Nov 12, 2015 at 12:19 PM, Hans Nilsson R < hans.r.nilsson@REDACTED> wrote: > Great! If there also is documentation and a test case the likelihood of > acceptance is higher :) > -Hans > > On 11/12/2015 06:31 PM, Vipin Nair wrote: > > Hello Hans, > > > >> A more general way to pass options to the key callback module would be > >> to introduce a second form of the key_cb option: > >> - keep todays {key_cb, Module::atom()} to keep compatibility > >> - introduce the variant {key_cb, {Module::atom(), ModuleOptions}} > >> [...] > >> In that way it is possible to write a callback module with whatever is > >> needed as private options. > > > > This will work! In retrospect, I should have done this instead. This is > much > > simpler and solves my problem. I'll send a new pull request with the > suggested > > implementation. Thanks! > > > > > _______________________________________________ > erlang-patches mailing list > erlang-patches@REDACTED > http://erlang.org/mailman/listinfo/erlang-patches > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From swvist@REDACTED Fri Nov 13 21:33:50 2015 From: swvist@REDACTED (Vipin Nair) Date: Sat, 14 Nov 2015 02:03:50 +0530 Subject: [erlang-patches] EEP Light: Option to override private key lookup when using Erlang ssh client In-Reply-To: <5644D84D.1060101@ericsson.com> References: <5640D376.9010409@ericsson.com> <5644C00D.4020407@ericsson.com> <5644D84D.1060101@ericsson.com> Message-ID: > Great! If there also is documentation and a test case the likelihood of > acceptance is higher :) Hans, I have made the suggested changes and sent a pull request[1]. Surprisingly, there were no tests for key callback. I have added tests for callback module as well as callback module options. Please review. [1]: https://github.com/erlang/otp/pull/892 -- Regards, Vipin From isvilen@REDACTED Mon Nov 23 10:57:51 2015 From: isvilen@REDACTED (Svilen Ivanov) Date: Mon, 23 Nov 2015 11:57:51 +0200 Subject: [erlang-patches] asn1 empty named bit string values encoding failure Message-ID: <1448272671.15638.5.camel@applicata.bg> Hi, currently code generated by asn1 library correctly decodes BER encoded empty named bit string values to '[]', but fails to encode them, provided patch (PR #900) adds function clause to asn1rtt_ber:encode_named_bit_string/4 to fix that. git fetch git://github.com/isvilen/otp.git empty_named_bitstrings https://github.com/erlang/otp/compare/maint...isvilen:empty_named_bitstrings https://github.com/erlang/otp/compare/maint...isvilen:empty_named_bitstrings.patch Svilen