[erlang-bugs] ssh2_msg_channel_failure

Fyodor Ustinov ufm@REDACTED
Wed Apr 25 19:10:29 CEST 2012


Hi!

putty say "Server send disconnect message type 11 (by application): 
"Internal error"

Erlang say:

=ERROR REPORT==== 25-Apr-2012::20:03:09 ===
** Generic server <0.156.0> terminating
** Last message in was {ssh_msg,<0.157.0>,
<<98,0,0,0,0,0,0,0,34,119,105,110,97,100,106,
                                   
64,112,117,116,116,121,46,112,114,111,106,
                                   
101,99,116,115,46,116,97,114,116,97,114,117,
                                   115,46,111,114,103,1>>}
** When Server state == {state,server,undefined,undefined,<0.157.0>,
                             {connection,[],315439,[],1,
                                 {ssh_cli,[#Fun<sshd.0.34972026>]},
                                 {10,128,0,10},
                                 2222,
                                 [{role,server},
                                  {pwdfun,#Fun<sshd.pwcheck.2>},
                                  {shell,#Fun<sshd.0.34972026>}],
                                 undefined,<0.153.0>,undefined},
                             0,
                             [{address,{10,128,0,10}},
                              {port,2222},
                              {role,server},
                              {socket_opts,[inet,{ip,{10,128,0,10}}]},
                              {ssh_opts,
                                  [{role,server},
                                   {pwdfun,#Fun<sshd.pwcheck.2>},
                                   {shell,#Fun<sshd.0.34972026>}]}],
                             undefined,true}
** Reason for termination ==
** {{badmatch,
         {disconnect,
             {11,"Internal error"},
             {{replies,[{channel_data,<0.159.0>,{closed,0}}]},
              {connection,[],315439,[],1,
                  {ssh_cli,[#Fun<sshd.0.34972026>]},
                  {10,128,0,10},
                  2222,
                  [{role,server},
                   {pwdfun,#Fun<sshd.pwcheck.2>},
                   {shell,#Fun<sshd.0.34972026>}],
                  undefined,<0.153.0>,undefined}}}},
     [{ssh_connection_manager,handle_call,3,
          [{file,"../src/ssh_connection_manager.erl"},{line,271}]},
      {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,588}]},
      {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

=ERROR REPORT==== 25-Apr-2012::20:03:09 ===
Erlang ssh connection handler failed with reason: {{{badmatch,
                                                      {disconnect,
                                                       {11,"Internal 
error"},
                                                       {{replies,
                                                         [{channel_data,
<0.159.0>,
                                                           {closed,0}}]},
                                                        
{connection,[],315439,
                                                         [],1,
                                                         {ssh_cli,
                                                          
[#Fun<sshd.0.34972026>]},
                                                         {10,128,0,10},
                                                         2222,
                                                         [{role,server},
                                                          {pwdfun,
                                                           
#Fun<sshd.pwcheck.2>},
                                                          {shell,
                                                           
#Fun<sshd.0.34972026>}],
                                                         
undefined,<0.153.0>,
                                                         undefined}}}},
                                                     
[{ssh_connection_manager,
                                                       handle_call,3,
                                                       [{file,
                                                         
"../src/ssh_connection_manager.erl"},
                                                        {line,271}]},
                                                      
{gen_server,handle_msg,
                                                       5,
                                                       
[{file,"gen_server.erl"},
                                                        {line,588}]},
                                                      {proc_lib,
                                                       init_p_do_apply,3,
                                                       
[{file,"proc_lib.erl"},
                                                        {line,227}]}]},
                                                    {gen_server,call,
                                                     [<0.156.0>,
                                                      {ssh_msg,<0.157.0>,
<<93,0,0,0,0,0,0,32,106>>},
                                                      infinity]}}
, Stacktace: [{gen_server,call,3,[{file,"gen_server.erl"},{line,188}]},
               {ssh_connection_manager,call,3,
                   [{file,"../src/ssh_connection_manager.erl"},{line,583}]},
               {ssh_connection_handler,generate_event,4,
                   [{file,"ssh_connection_handler.erl"},{line,804}]},
               {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,494}]},
               
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]
please report this to erlang-bugs@REDACTED
=ERROR REPORT==== 25-Apr-2012::20:03:09 ===
** State machine <0.157.0> terminating
** Last message in was {tcp,#Port<0.1845>,<<>>}
** When State == connected
**      Data  == {state,tcp,gen_tcp,tcp_closed,
                      {ssh,server,
                          {undefined,{{10,128,0,1},1076}},
                          {2,0},
                          {2,0},
                          "SSH-2.0-PuTTY_Release_0.62","SSH-2.0-Erlang",
<<20,231,36,36,12,195,57,51,47,209,229,164,97,204,116,
                            
116,90,0,0,0,154,100,105,102,102,105,101,45,104,101,
                            
108,108,109,97,110,45,103,114,111,117,112,45,101,
                            
120,99,104,97,110,103,101,45,115,104,97,50,53,54,44,
                            
100,105,102,102,105,101,45,104,101,108,108,109,97,
                            
110,45,103,114,111,117,112,45,101,120,99,104,97,110,
                            
103,101,45,115,104,97,49,44,100,105,102,102,105,101,
                            
45,104,101,108,108,109,97,110,45,103,114,111,117,
                            
112,49,52,45,115,104,97,49,44,100,105,102,102,105,
                            
101,45,104,101,108,108,109,97,110,45,103,114,111,
                            
117,112,49,45,115,104,97,49,44,114,115,97,50,48,52,
                            
56,45,115,104,97,50,53,54,44,114,115,97,49,48,50,52,
                            
45,115,104,97,49,0,0,0,15,115,115,104,45,114,115,97,
                            
44,115,115,104,45,100,115,115,0,0,0,159,97,101,115,
                            
50,53,54,45,99,116,114,44,97,101,115,50,53,54,45,99,
                            
98,99,44,114,105,106,110,100,97,101,108,45,99,98,99,
                            
64,108,121,115,97,116,111,114,46,108,105,117,46,115,
                            
101,44,97,101,115,49,57,50,45,99,116,114,44,97,101,
                            
115,49,57,50,45,99,98,99,44,97,101,115,49,50,56,45,
                            
99,116,114,44,97,101,115,49,50,56,45,99,98,99,44,98,
                            
108,111,119,102,105,115,104,45,99,116,114,44,98,108,
                            
111,119,102,105,115,104,45,99,98,99,44,51,100,101,
                            
115,45,99,116,114,44,51,100,101,115,45,99,98,99,44,
                            
97,114,99,102,111,117,114,50,53,54,44,97,114,99,102,
                            
111,117,114,49,50,56,0,0,0,159,97,101,115,50,53,54,
                            
45,99,116,114,44,97,101,115,50,53,54,45,99,98,99,44,
                            
114,105,106,110,100,97,101,108,45,99,98,99,64,108,
                            
121,115,97,116,111,114,46,108,105,117,46,115,101,44,
                            
97,101,115,49,57,50,45,99,116,114,44,97,101,115,49,
                            
57,50,45,99,98,99,44,97,101,115,49,50,56,45,99,116,
                            
114,44,97,101,115,49,50,56,45,99,98,99,44,98,108,
                            
111,119,102,105,115,104,45,99,116,114,44,98,108,111,
                            
119,102,105,115,104,45,99,98,99,44,51,100,101,115,
                            
45,99,116,114,44,51,100,101,115,45,99,98,99,44,97,
                            
114,99,102,111,117,114,50,53,54,44,97,114,99,102,
                            
111,117,114,49,50,56,0,0,0,31,104,109,97,99,45,115,
                            
104,97,49,44,104,109,97,99,45,115,104,97,49,45,57,
                            
54,44,104,109,97,99,45,109,100,53,0,0,0,31,104,109,
                            
97,99,45,115,104,97,49,44,104,109,97,99,45,115,104,
                            
97,49,45,57,54,44,104,109,97,99,45,109,100,53,0,0,0,
                            
9,110,111,110,101,44,122,108,105,98,0,0,0,9,110,111,
                            
110,101,44,122,108,105,98,0,0,0,0,0,0,0,0,0,0,0,0,0>>,
<<20,68,106,202,126,189,248,249,152,175,237,102,43,
                            
239,21,11,93,0,0,0,26,100,105,102,102,105,101,45,
                            
104,101,108,108,109,97,110,45,103,114,111,117,112,
                            
49,45,115,104,97,49,0,0,0,15,115,115,104,45,114,115,
                            
97,44,115,115,104,45,100,115,115,0,0,0,19,97,101,
                            
115,49,50,56,45,99,98,99,44,51,100,101,115,45,99,98,
                            
99,0,0,0,19,97,101,115,49,50,56,45,99,98,99,44,51,
                            
100,101,115,45,99,98,99,0,0,0,9,104,109,97,99,45,
                            
115,104,97,49,0,0,0,9,104,109,97,99,45,115,104,97,
                            
49,0,0,0,9,110,111,110,101,44,122,108,105,98,0,0,0,
                            
9,110,111,110,101,44,122,108,105,98,0,0,0,0,0,0,0,0,
                            0,0,0,0,0>>,
                          undefined,'diffie-hellman-group1-sha1','ssh-rsa',
                          ssh_file,ssh_io,'hmac-sha1',
<<92,89,171,169,113,136,6,188,156,238,58,144,38,214,
                            101,198,247,166,205,208>>,
                          20,'hmac-sha1',
<<24,206,173,99,5,252,206,70,166,228,30,17,101,89,36,
                            241,175,116,238,86>>,
                          20,'aes128-cbc',
<<45,208,196,236,205,82,244,222,24,132,67,151,22,62,
                            250,150>>,
                          16,
<<219,113,112,85,250,220,170,249,194,198,230,158,76,
                            122,39,155>>,
                          'aes128-cbc',
<<44,157,215,194,90,86,202,31,228,49,188,97,227,97,
                            233,144>>,
                          16,
<<196,110,206,199,93,111,201,100,74,194,231,242,179,
                            201,15,112>>,
                          none,undefined,none,undefined,none,none,true,
                          infinity,
                          
132151717501655075704740428602371821313660310182519774905429178494543592682738802201444120687073445805819575914576401425536104160589136387731343596760889931670361900328999551747697001054684538185039596790225261438983095509229017954492065022996307622720613396529860286665422597229362632923823896919144441708303,
<<163,238,219,114,10,240,95,30,207,178,78,138,146,58,
                            87,102,43,69,173,214>>,
<<163,238,219,114,10,240,95,30,207,178,78,138,146,58,
                            87,102,43,69,173,214>>,
                          [{address,{10,128,0,10}},
                           {port,2222},
                           {role,server},
                           {pwdfun,#Fun<sshd.pwcheck.2>},
                           {shell,#Fun<sshd.0.34972026>}],
                          114,49,
                          
{{49985839305237737688589908016292087537304670681285167039328910635484947391794443232164854289884214947929742793900078775832939310627327467626874637085242037758449972175725827907772272533385534014381436171929411436926043326068534386929776710296045274502307637113679098707841881585433086106245829282665980090815,
                            
8395950144135694221446766804944646771011612746974327140375413192360614631043395924085970461957169944119762874929925969231067866506208529309578676567265288403413833138477256106374779295193693244061192404399083583730265790053008130870886892278024924477046645863115470835406736499812828883753405586277633495326},
                           {2,
                            
179769313486231590770839156793787453197860296048756011706444423684197180216158519368947833795864925541502180565485980503646440548199239100050792877003355816639229553136239076508735759914822574862575007425302077447712589550957937778424442426617334727629299387668709205606050270810842907692932019128194467627007}},
                          undefined,undefined,"ssh-connection",undefined,
                          
"publickey,keyboard_interactive,password",undefined,
                          undefined,
                          ["ssh-rsa","ssh-dss"]},
                      #Port<0.1845>,<<>>,
<<107,0,185,229,147,59,116,109,92,16,81,210,0,243,152,220,
                        
153,112,133,2,156,113,5,213,202,236,150,200,184,190,61,
                        
227,37,31,157,1,53,98,136,245,199,18,165,166,214,36,210,
                        12,92,130,17,174>>,
                      undefined,
                      {ssh_msg_kexinit,
<<126,96,184,235,216,208,69,175,215,35,94,180,214,20,
                            246,141>>,
                          ["diffie-hellman-group1-sha1"],
                          ["ssh-rsa","ssh-dss"],
                          ["aes128-cbc","3des-cbc"],
                          ["aes128-cbc","3des-cbc"],
                          ["hmac-sha1"],
                          ["hmac-sha1"],
                          ["none","zlib"],
                          ["none","zlib"],
                          [],[],false,0},
                      false,<0.156.0>,undefined,undefined,undefined,
                      [{address,{10,128,0,10}},
                       {port,2222},
                       {role,server},
                       {pwdfun,#Fun<sshd.pwcheck.2>},
                       {shell,#Fun<sshd.0.34972026>}]}
** Reason for termination =
** {{{badmatch,
          {disconnect,
              {11,"Internal error"},
              {{replies,[{channel_data,<0.159.0>,{closed,0}}]},
               {connection,[],315439,[],1,
                   {ssh_cli,[#Fun<sshd.0.34972026>]},
                   {10,128,0,10},
                   2222,
                   [{role,server},
                    {pwdfun,#Fun<sshd.pwcheck.2>},
                    {shell,#Fun<sshd.0.34972026>}],
                   undefined,<0.153.0>,undefined}}}},
      [{ssh_connection_manager,handle_call,3,
           [{file,"../src/ssh_connection_manager.erl"},{line,271}]},
       {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,588}]},
       {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},
     {gen_server,call,
         
[<0.156.0>,{ssh_msg,<0.157.0>,<<93,0,0,0,0,0,0,32,106>>},infinity]}}



> Hi!
>
> Daniel Goertzen wrote:
>> The channel number on the putty side is 0x100, and on the Erlang side 
>> it is 0x0.  Erlang is responding to SSH_MSG_CHANNEL_REQUEST using 
>> 0x0, however RFC4254 says the responses should be...
>>
>>
>>       byte      SSH_MSG_CHANNEL_SUCCESS
>>       uint32    recipient channel
>>
>>
>>       byte      SSH_MSG_CHANNEL_FAILURE
>>       uint32    recipient channel
>>
>>
>> ...and I interpret "recipient channel" as being the Putty side, which 
>> is 0x100.  Putty seems to cope when Erlang returns SUCCESS, but blows 
>> up when Erlang returns a FAILURE.
>>
>
> Yes that is correct I remember correcting such a bug a long time ago, 
> checking the code however there seems to be one error handling case 
> that I missed.  Could you check if
> the following patch resolves your issue Fyodor?
>
> diff --git a/lib/ssh/src/ssh_connection.erl 
> b/lib/ssh/src/ssh_connection.erl
> index 46f0c7e..8574e08 100644
> --- a/lib/ssh/src/ssh_connection.erl
> +++ b/lib/ssh/src/ssh_connection.erl
> @@ -720,10 +720,12 @@ handle_msg(#ssh_msg_channel_request{request_type 
> = "env"},
>
> handle_msg(#ssh_msg_channel_request{recipient_channel = ChannelId,
>                                    request_type = _Other,
> -                                   want_reply = WantReply}, Connection,
> +                                   want_reply = WantReply},  
> #connection{channel_cache = Cache} = Connection,
>           ConnectionPid, _) ->
>     if WantReply == true ->
> -           FailMsg = channel_failure_msg(ChannelId),
> +           #channel{remote_id = RemoteId} = ChannelId =
> +               ssh_channel:cache_lookup(Cache, ChannelId),
> +           FailMsg = channel_failure_msg(RemoteId),
>            {{replies, [{connection_reply, ConnectionPid, FailMsg}]},
>             Connection};
>        true ->
>
>
>> I could imagine this passing automated tests when both ends number 
>> their channels starting from 0.  I bet randomizing the channel 
>> numbers would reveal this and maybe other issues.
>>
>> Dan.
>>
> I think you have a valid point however we have to change the 
> implementation in one way or an other to do that and that has not been 
> prioritized.
>
>
> Regards Ingela Erlang/OTP team -Ericsson AB
>
>>
>> On Tue, Apr 24, 2012 at 11:22 AM, Fyodor Ustinov <ufm@REDACTED 
>> <mailto:ufm@REDACTED>> wrote:
>>
>>     Hi!
>>
>>     I added this patch - did not help.
>>
>>     http://blog.ufm.su/putty.log - this putty log.
>>
>>
>>         Hi!
>>
>>         This might be a symptom of the "close race-condition"-bug
>>          reported by Daniel. The patch for that bug is
>>
>>         diff --git a/lib/ssh/src/ssh_channel.erl
>>         b/lib/ssh/src/ssh_channel.erl
>>         index 7b600ed..1938858 100644
>>         --- a/lib/ssh/src/ssh_channel.erl
>>         +++ b/lib/ssh/src/ssh_channel.erl
>>         @@ -1,7 +1,7 @@
>>         %%
>>         %% %CopyrightBegin%
>>         %%
>>         -%% Copyright Ericsson AB 2008-2011. All Rights Reserved.
>>         +%% Copyright Ericsson AB 2008-2012. All Rights Reserved.
>>         %%
>>         %% The contents of this file are subject to the Erlang Public
>>         License,
>>         %% Version 1.1, (the "License"); you may not use this file
>>         except in
>>         @@ -215,7 +215,7 @@ handle_info({ssh_cm, ConnectionManager,
>>         {closed, ChannelId}},
>>                          close_sent = false} = State) ->
>>            %% To be on the safe side, i.e. the manager has already
>>         been terminated.
>>            (catch ssh_connection:close(ConnectionManager, ChannelId)),
>>         -    {stop, normal, State};
>>         +    {stop, normal, State#state{close_sent = true}};
>>
>>         handle_info({ssh_cm, _, _} = Msg, #state{cm = ConnectionManager,
>>                               channel_cb = Module,
>>         diff --git a/lib/ssh/src/ssh_connection_manager.erl
>>         b/lib/ssh/src/ssh_connection_manager.erl
>>         index e993f59..8c1f88e 100644
>>         --- a/lib/ssh/src/ssh_connection_manager.erl
>>         +++ b/lib/ssh/src/ssh_connection_manager.erl
>>         @@ -384,9 +384,10 @@ handle_call({close, ChannelId}, _,
>>                   #state{connection = Pid, connection_state =
>>                          #connection{channel_cache = Cache}} = State) ->
>>            case ssh_channel:cache_lookup(Cache, ChannelId) of
>>         -       #channel{remote_id = Id} ->
>>         +       #channel{remote_id = Id} = Channel ->
>>                   send_msg({connection_reply, Pid,
>>                             ssh_connection:channel_close_msg(Id)}),
>>         +           ssh_channel:cache_update(Cache,
>>         Channel#channel{sent_close = true}),
>>                   {reply, ok, State};
>>               undefined ->
>>                   {reply, ok, State}
>>
>>
>>         Regards Ingela Erlang/OTP  team - Ericsson AB
>>
>>         Daniel Goertzen wrote:
>>
>>             Under Putty Session->Logging can you turn on "SSH Packets"
>>             and post the log showing the error?
>>
>>             Dan.
>>
>>             On Sun, Apr 22, 2012 at 3:58 AM, Fyodor Ustinov
>> <ufm@REDACTED <mailto:ufm@REDACTED> <mailto:ufm@REDACTED
>> <mailto:ufm@REDACTED>>> wrote:
>>
>>                Hi!
>>
>>                Windows ssh client PuTTY show message  "Disconnected:
>>             Received
>>                SSH2_MSG_CHANNEL_FAILURE for nonexistent channel 0"
>>             while working
>>                with erlang ssh daemon.
>>
>>                Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:8:8]
>>                [async-threads:0] [kernel-poll:false]
>>
>>                WBR,
>>                   Fyodor.
>>
>>                _______________________________________________
>>                erlang-bugs mailing list
>>                erlang-bugs@REDACTED <mailto:erlang-bugs@REDACTED>
>> <mailto:erlang-bugs@REDACTED
>> <mailto:erlang-bugs@REDACTED>>
>>                http://erlang.org/mailman/listinfo/erlang-bugs
>>
>>
>>
>>
>>             --             *Daniel Goertzen | Senior Software Engineer*
>>             Office: 828.610.4596 <tel:828.610.4596> | Fax:
>>             828.322.5294 <tel:828.322.5294> |
>>             dang@REDACTED
>> <mailto:dang@REDACTED>
>> <mailto:dang@REDACTED
>> <mailto:dang@REDACTED>>
>>             *Network Integrity Systems | We Bring Security To Light™*
>>
>>             1937 Tate Blvd. SE
>>
>>             Hickory, North Carolina, USA 28602
>>
>>             /Network Integrity Systems’ INTERCEPTOR™ Optical Network
>>             Security System is a Smart-PDS™ that ensures superior
>>             protection and cost effectiveness of classified networks.
>>              For more information, visit our website
>>             at:www.networkintegritysystems.com
>> <http://www.networkintegritysystems.com>
>> <http://www.networkintegritysystems.com/>./
>>
>>
>>             __________________________________________
>>             INTERCEPTOR™ Optical Network Security System is made in
>>             the USA for the USA. Although not an export controlled
>>             item, because of the role it plays in the assurance of the
>>             safety and integrity of National Security Information,
>>             Network Integrity Systems (NIS) is committed to compliance
>>             with the U.S. Export Administration Act. Accordingly, NIS
>>             will not ship INTERCEPTOR products to certain foreign
>>             government end users without U.S. government approval and
>>             will refuse transactions with individuals or entities that
>>             have been denied export privileges.
>>
>>
>>             
>> ------------------------------------------------------------------------
>>
>>             _______________________________________________
>>             erlang-bugs mailing list
>>             erlang-bugs@REDACTED <mailto:erlang-bugs@REDACTED>
>>             http://erlang.org/mailman/listinfo/erlang-bugs
>>
>>
>>
>>     _______________________________________________
>>     erlang-bugs mailing list
>>     erlang-bugs@REDACTED <mailto:erlang-bugs@REDACTED>
>>     http://erlang.org/mailman/listinfo/erlang-bugs
>>
>>
>>
>>
>> -- 
>> *Daniel Goertzen | Senior Software Engineer*
>> Office: 828.610.4596 | Fax: 828.322.5294 | 
>> dang@REDACTED 
>> <mailto:dang@REDACTED>
>> *Network Integrity Systems | We Bring Security To Light™*
>>
>> 1937 Tate Blvd. SE
>>
>> Hickory, North Carolina, USA 28602
>>
>> /Network Integrity Systems’ INTERCEPTOR™ Optical Network Security 
>> System is a Smart-PDS™ that ensures superior protection and cost 
>> effectiveness of classified networks.  For more information, visit 
>> our website at:www.networkintegritysystems.com 
>> <http://www.networkintegritysystems.com/>./
>>
>>
>> __________________________________________
>> INTERCEPTOR™ Optical Network Security System is made in the USA for 
>> the USA. Although not an export controlled item, because of the role 
>> it plays in the assurance of the safety and integrity of National 
>> Security Information, Network Integrity Systems (NIS) is committed to 
>> compliance with the U.S. Export Administration Act. Accordingly, NIS 
>> will not ship INTERCEPTOR products to certain foreign government end 
>> users without U.S. government approval and will refuse transactions 
>> with individuals or entities that have been denied export privileges.
>>
>>
>




More information about the erlang-bugs mailing list