From mario.santoro@REDACTED Mon Dec 12 18:01:03 2016 From: mario.santoro@REDACTED (Mario Santoro) Date: Mon, 12 Dec 2016 17:01:03 +0000 Subject: [erlang-bugs] Erlang SSH client hangs after some thousand requests Message-ID: Hi all, I wrote an Erlang ssh client module for running commands on a remote host over an ssh connection. The main calls to Erlang ssh library functions to setup the connection are the following: ssh:connect/4 ssh_connection:session_channel/2 ssh_connection:ptty_alloc/3 ssh_connection:shell/2 afterwards I'm issuing a command to remote host via: ssh_connection:send/3 and receiving related reply in a msg loop, with the following format: {ssh_cm, ConnectionRef, {data, ChannelId, Type, Data}} Everything works fine and I'm able to issue commands and receive responses correctly. What I'm experiencing is a strange behavior on long lasting ssh sessions: when I start a long loop of ssh send-receive requests, it seems like the client connection hangs after a certain number or positive command requests (several thousands), waiting indefinitely for a reply, wherefrom it exits with timeout. The only way to recover from this status is to disconnect and reconnect to the server. I tried with both Erlang OTP/17.x and OTP/18.x and towards remote hosts running Linux and/or Solaris OS, but the result is the same. This behavior seems to be deterministic: given a command, it happens after the same number of issued requests (e.g. for a trivial "pwd" command, the client always hangs after 2510 tries!!). This make me think as there's a limited receive buffer size somewhere into the client ssh library or something similar (maybe something that I should set at connection setup?); since I'm quite newbie to Erlang, I'd very much appreciate if anyone could help me to find out the reason for this behaviour. Thanks in advance for your help! Regards, /Mario [Ericsson] MARIO SANTORO Service Project Manager Region Mediterranean RMED ITC CU SCM SL AIB DMO Ericsson Via Anagnina, 203 00118, ROME, ITALY Mobile +39 335 699 5215 Office +39 06 7258 8404 mario.santoro@REDACTED www.ericsson.com [http://www.ericsson.com/current_campaign] ---------- Legal entity: Ericsson Telecomunicazioni SpA, registered office in Rome. This Communication is Confidential. We only send and receive email on the basis of the terms set out at www.ericsson.com/email_disclaimer -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.gif Type: image/gif Size: 2367 bytes Desc: image003.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.gif Type: image/gif Size: 32254 bytes Desc: image004.gif URL: From hans.r.nilsson@REDACTED Mon Dec 12 18:22:30 2016 From: hans.r.nilsson@REDACTED (Hans Nilsson R) Date: Mon, 12 Dec 2016 18:22:30 +0100 Subject: [erlang-bugs] Erlang SSH client hangs after some thousand requests In-Reply-To: References: Message-ID: Do you call ssh_connection:adjust_window/3? -Hans On 12/12/2016 06:01 PM, Mario Santoro wrote: > Hi all, > > I wrote an Erlang ssh client module for running commands on a remote > host over an ssh connection. > > The main calls to Erlang ssh library functions to setup the connection > are the following: > > ssh:connect/4 > > ssh_connection:session_channel/2 > > ssh_connection:ptty_alloc/3 > > ssh_connection:shell/2 > > > > afterwards I?m issuing a command to remote host via: > > ssh_connection:send/3 > > > > and receiving related reply in a msg loop, with the following format: > > {ssh_cm, ConnectionRef, {data, ChannelId, Type, Data}} > > > > Everything works fine and I?m able to issue commands and receive > responses correctly. > > > > What I?m experiencing is a strange behavior on long lasting ssh > sessions: when I start a long loop of > > ssh send-receive requests, it seems like the client connection hangs > after a certain number or positive command requests (several thousands), > > waiting indefinitely for a reply, wherefrom it exits with timeout. > > The only way to recover from this status is to disconnect and reconnect > to the server. > > > > I tried with both Erlang OTP/17.x and OTP/18.x and towards remote hosts > running Linux and/or Solaris OS, but the result is the same. > > > > This behavior seems to be deterministic: given a command, it happens > after the same number of issued requests (e.g. for a trivial ?pwd? command, > > the client always hangs after 2510 tries!!). > > This make me think as there?s a limited receive buffer size somewhere > into the client ssh library or something similar (maybe something that I > should set at connection setup?); > > since I?m quite newbie to Erlang, I?d very much appreciate if anyone > could help me to find out the reason for this behaviour. > > > > Thanks in advance for your help! > > > > Regards, > > /Mario > > > > > > > > Ericsson > > *MARIO SANTORO * > Service Project Manager > Region Mediterranean > RMED ITC CU SCM SL AIB DMO > > > *Ericsson* > Via Anagnina, 203 > 00118, ROME, ITALY > Mobile +39 335 699 5215 > Office +39 06 7258 8404 > mario.santoro@REDACTED > www.ericsson.com > > > > http://www.ericsson.com/current_campaign > > > > > ---------- > > Legal entity: Ericsson Telecomunicazioni SpA, registered office in Rome. > This Communication is Confidential. We only send and receive email on > the basis of the terms set out at www.ericsson.com/email_disclaimer > > > > > _______________________________________________ > erlang-bugs mailing list > erlang-bugs@REDACTED > http://erlang.org/mailman/listinfo/erlang-bugs > From mario.santoro@REDACTED Mon Dec 12 18:25:17 2016 From: mario.santoro@REDACTED (Mario Santoro) Date: Mon, 12 Dec 2016 17:25:17 +0000 Subject: [erlang-bugs] Erlang SSH client hangs after some thousand requests In-Reply-To: References: Message-ID: No, I don't. Should I? -Mario -----Original Message----- From: Hans Nilsson R Sent: 12 December 2016 18:23 To: Mario Santoro; erlang-bugs@REDACTED Subject: Re: [erlang-bugs] Erlang SSH client hangs after some thousand requests Do you call ssh_connection:adjust_window/3? -Hans On 12/12/2016 06:01 PM, Mario Santoro wrote: > Hi all, > > I wrote an Erlang ssh client module for running commands on a remote > host over an ssh connection. > > The main calls to Erlang ssh library functions to setup the connection > are the following: > > ssh:connect/4 > > ssh_connection:session_channel/2 > > ssh_connection:ptty_alloc/3 > > ssh_connection:shell/2 > > > > afterwards I'm issuing a command to remote host via: > > ssh_connection:send/3 > > > > and receiving related reply in a msg loop, with the following format: > > {ssh_cm, ConnectionRef, {data, ChannelId, Type, Data}} > > > > Everything works fine and I'm able to issue commands and receive > responses correctly. > > > > What I'm experiencing is a strange behavior on long lasting ssh > sessions: when I start a long loop of > > ssh send-receive requests, it seems like the client connection hangs > after a certain number or positive command requests (several > thousands), > > waiting indefinitely for a reply, wherefrom it exits with timeout. > > The only way to recover from this status is to disconnect and > reconnect to the server. > > > > I tried with both Erlang OTP/17.x and OTP/18.x and towards remote > hosts running Linux and/or Solaris OS, but the result is the same. > > > > This behavior seems to be deterministic: given a command, it happens > after the same number of issued requests (e.g. for a trivial "pwd" > command, > > the client always hangs after 2510 tries!!). > > This make me think as there's a limited receive buffer size somewhere > into the client ssh library or something similar (maybe something that > I should set at connection setup?); > > since I'm quite newbie to Erlang, I'd very much appreciate if anyone > could help me to find out the reason for this behaviour. > > > > Thanks in advance for your help! > > > > Regards, > > /Mario > > > > > > > > Ericsson > > *MARIO SANTORO * > Service Project Manager > Region Mediterranean > RMED ITC CU SCM SL AIB DMO > > > *Ericsson* > Via Anagnina, 203 > 00118, ROME, ITALY > Mobile +39 335 699 5215 > Office +39 06 7258 8404 > mario.santoro@REDACTED > www.ericsson.com > > > > http://www.ericsson.com/current_campaign > > > > > ---------- > > Legal entity: Ericsson Telecomunicazioni SpA, registered office in Rome. > This Communication is Confidential. We only send and receive email on > the basis of the terms set out at www.ericsson.com/email_disclaimer > > > > > _______________________________________________ > erlang-bugs mailing list > erlang-bugs@REDACTED > http://erlang.org/mailman/listinfo/erlang-bugs > From mario.santoro@REDACTED Tue Dec 13 13:46:33 2016 From: mario.santoro@REDACTED (Mario Santoro) Date: Tue, 13 Dec 2016 12:46:33 +0000 Subject: [erlang-bugs] Erlang SSH client hangs after some thousand requests In-Reply-To: References: Message-ID: Hi Hans, calling ssh_connection:adjust_window/3 with a large enough bytesize seems to overtake the problem. Thanks a lot for your useful hint! :) -Mario -----Original Message----- From: erlang-bugs-bounces@REDACTED [mailto:erlang-bugs-bounces@REDACTED] On Behalf Of Mario Santoro Sent: 12 December 2016 18:25 To: Hans Nilsson R; erlang-bugs@REDACTED Subject: Re: [erlang-bugs] Erlang SSH client hangs after some thousand requests No, I don't. Should I? -Mario -----Original Message----- From: Hans Nilsson R Sent: 12 December 2016 18:23 To: Mario Santoro; erlang-bugs@REDACTED Subject: Re: [erlang-bugs] Erlang SSH client hangs after some thousand requests Do you call ssh_connection:adjust_window/3? -Hans On 12/12/2016 06:01 PM, Mario Santoro wrote: > Hi all, > > I wrote an Erlang ssh client module for running commands on a remote > host over an ssh connection. > > The main calls to Erlang ssh library functions to setup the connection > are the following: > > ssh:connect/4 > > ssh_connection:session_channel/2 > > ssh_connection:ptty_alloc/3 > > ssh_connection:shell/2 > > > > afterwards I'm issuing a command to remote host via: > > ssh_connection:send/3 > > > > and receiving related reply in a msg loop, with the following format: > > {ssh_cm, ConnectionRef, {data, ChannelId, Type, Data}} > > > > Everything works fine and I'm able to issue commands and receive > responses correctly. > > > > What I'm experiencing is a strange behavior on long lasting ssh > sessions: when I start a long loop of > > ssh send-receive requests, it seems like the client connection hangs > after a certain number or positive command requests (several > thousands), > > waiting indefinitely for a reply, wherefrom it exits with timeout. > > The only way to recover from this status is to disconnect and > reconnect to the server. > > > > I tried with both Erlang OTP/17.x and OTP/18.x and towards remote > hosts running Linux and/or Solaris OS, but the result is the same. > > > > This behavior seems to be deterministic: given a command, it happens > after the same number of issued requests (e.g. for a trivial "pwd" > command, > > the client always hangs after 2510 tries!!). > > This make me think as there's a limited receive buffer size somewhere > into the client ssh library or something similar (maybe something that > I should set at connection setup?); > > since I'm quite newbie to Erlang, I'd very much appreciate if anyone > could help me to find out the reason for this behaviour. > > > > Thanks in advance for your help! > > > > Regards, > > /Mario > > > > > > > > Ericsson > > *MARIO SANTORO * > Service Project Manager > Region Mediterranean > RMED ITC CU SCM SL AIB DMO > > > *Ericsson* > Via Anagnina, 203 > 00118, ROME, ITALY > Mobile +39 335 699 5215 > Office +39 06 7258 8404 > mario.santoro@REDACTED > www.ericsson.com > > > > http://www.ericsson.com/current_campaign > > > > > ---------- > > Legal entity: Ericsson Telecomunicazioni SpA, registered office in Rome. > This Communication is Confidential. We only send and receive email on > the basis of the terms set out at www.ericsson.com/email_disclaimer > > > > > _______________________________________________ > erlang-bugs mailing list > erlang-bugs@REDACTED > http://erlang.org/mailman/listinfo/erlang-bugs > _______________________________________________ erlang-bugs mailing list erlang-bugs@REDACTED http://erlang.org/mailman/listinfo/erlang-bugs From kimhrac@REDACTED Wed Dec 14 10:20:45 2016 From: kimhrac@REDACTED (Kim Hansen) Date: Wed, 14 Dec 2016 10:20:45 +0100 Subject: [erlang-bugs] TLS "unexpected message" Message-ID: We're running RabbitMQ 3.6.6 on top of Erlang 19.1.5 and we are seeing these errors in the log: =ERROR REPORT==== 13-Dec-2016::11:48:04 === SSL: {connection,{alert,2,20,{"tls_record.erl",229},undefined}}: ssl_connection.erl:842:Fatal error: unexpected message =WARNING REPORT==== 13-Dec-2016::11:48:04 === STOMP connection 10.20.30.40:57788 -> 172.17.0.2:15679 terminated with reason {inet_error, {tls_alert, "unexpected message"}}, closing it =ERROR REPORT==== 13-Dec-2016::11:48:04 === ** Generic server <0.21658.12> terminating ** Last message in was {inet_async, {sslsocket, {gen_tcp,#Port<0.69369>,tls_connection, <0.701.0>}, <0.21655.12>}, #Ref<0.0.15466499.144103>, {error,{tls_alert,"unexpected message"}}} ** When Server state == {reader_state, {sslsocket, {gen_tcp,#Port<0.69369>,tls_connection,<0.701.0>}, <0.21655.12>}, "10.20.30.40:57788 -> 172.17.0.2:15679", {resume,#Fun}, {proc_state,"session-GSnfw7PxboMQVsaeobF8ug", <0.22191.12>,<0.22169.12>, {dict,2,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, {{[[<<"T_0">>| {subscription, "/exchange/h2s.rsp/eur1.USERNAME",client, false,"id='0'"}]], [],[],[],[],[],[],[],[],[],[],[],[], [[<<"T_1">>| {subscription, "/exchange/s2h.req/eur1.USERNAME.json.*.*", client,false,"id='1'"}]], [],[]}}}, "1.2",undefined, {0,nil}, {stomp_configuration,<<"guest">>,<<"guest">>,false, false,false}, {set,0,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}}}, {dict,0,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}}}, #Fun, {amqp_adapter_info, {0,0,0,0,0,65535,44049,2}, 15679, {0,0,0,0,0,65535,22167,24922}, 57788,<<"10.20.30.40:57788 -> 172.17.0.2:15679 ">>, {'STOMP',0}, [{channels,1}, {channel_max,1}, {frame_max,0}, {client_properties, [{<<"product">>,longstr,<<"STOMP client">>}]}, {ssl,true}, {ssl_protocol,'tlsv1.2'}, {ssl_key_exchange,rsa}, {ssl_cipher,aes_128_cbc}, {ssl_hash,sha256}]}, #Fun,none, {0,0,0,0,0,65535,44049,2}, false}, running,false,true, {state,fine,5000,#Ref<0.0.15466499.142857>}, undefined,<0.22169.12>,<0.21657.12>, {<0.22196.12>,<0.22197.12>}} ** Reason for termination == ** {inet_error,{tls_alert,"unexpected message"}} and it looks like it might cause a memory leak because we also get a lot of these: =WARNING REPORT==== 13-Dec-2016::11:48:05 === STOMP detected missed client heartbeat(s) on connection 10.20.30.40:51836 -> 172.17.0.2:15679, closing it We've asked Pivotal about the root cause of this and they tell us that it is a TLS alert (from Erlang) that ceases all TCP traffic. I'd like to know if we can provide you with more information about this? tcpdump traces or the like? If yes, can you provide us with a tcpdump command that include a pseudo source IP address or something and we are more than happy to dig into this and provide some logs. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: