[erlang-questions] Help: ct_telnet_client memory overflow

[Peter Andersson]

Hi,

I've looked into your problem now and it turns out to be a fairly
complex one. The memory usage behavior you see is the result of a few
separate problems:

When you do cat on a very large file over telnet, the result is that the
telnet client gets bombarded with loads of small TCP messages (will
depend on your TCP configuration). The CT telnet client is not designed
to handle this scenario (not well anyway), mainly because of these settings:


  gen_tcp:connect(Server, Port, [list,{packet,0}], Timeout)


It means connections are active, i.e. the client process receives the
messages as they come in without any means of controlling the flow.
Messages actually get queued in the mailbox faster than they are
processed. Also, the messages are received as lists, not as binaries.
The combination here is lethal.


Moreover, the client runs a search-and-copy operation on each message
received, and the result gets buffered with previous messages in
loopdata - a list. Every now and then, this buffer gets requested
(get_data), at which point the list is reversed and copied to the
requesting process, which in your case (according to the example you
sent me) prints the data to file by means of io:format/3. That actually
means sending a large list to the io server, which is yet another
expensive copy operation.


I'm also suspecting that the huge amount of small messages in the system
causes the memory to become badly fragmented.


Again, the nasty memory usage is caused by a combination of all these
things, so here's what needs to be done:


1. The CT telnet client should receive binary TCP packets instead of
lists so the data isn't copied, and so the loopdata list contains
references instead of lists.

2. The TCP connection must be passive in order to avoid massive buffering.

3. The search-and-copy operation (check_msg/3) should be changed into a
search-and-only-copy-if-necessary operation.

4. The get_data/1 function should return a binary rather than a list.

5. Write the data to file using the file server i/f (file) rather than
io server i/f (io).


To test this, I first modified ct_telnet_client according to #1 and #3
above. My first version would quickly run out of memory though because
of an increased rate of incoming (now binary) packets (and no flow
control), so I had to implement #2 as well to get it working.


The result was then that I could process a ~30 Mb text file using 20 Mb
RAM (the system total reported by observer). Linux top reported memory
usage of max 55 Mb.


The only possible workaround for you (that I can think of) that doesn't
require any modifications of ct_telnet_client, is that you run split
(also over telnet) to split up your large file first, then do cat on
each part, one at a time. A very primitive form of flow control! :-)  I
haven't tested if this works well in practice.


I will write a ticket on updating ct_telnet_client to be more efficient
dealing with large amounts of data, but can't say when that fix will be
officially released. In the meantime I can send you my experimental
module to play around with further.


Best regards,

Peter


Ericsson AB, Erlang/OTP


yasoho@REDACTED wrote:
>
> Hi, Peter
>
> Our performance analysis discover the cause of the problem is not tail
> recursion function loop(State, Sock, Acc)) to ct_telnet_client;
>
> Can you give me a advice How to modifiy it?   thanks!
>
>
> B.R
> ------------------------------------------------------------------------
> yasoho@REDACTED
>
>      
>     *From:* Peter Andersson <mailto:peppe@REDACTED>
>     *Date:* 2014-04-22 15:53
>     *To:* yasoho@REDACTED <mailto:yasoho@REDACTED>
>     *CC:* erlang-questions <mailto:erlang-questions@REDACTED>
>     *Subject:* Re: [erlang-questions] Help: ct_telnet_client memory
>     overflow
>      
>     Hi,
>      
>     Can you please tell me which Common Test version (or OTP version)
>     you're
>     running? (The CT telnet client got modified a bit from v1.7 to v1.8).
>      
>     Best regards,
>     Peter
>      
>     Ericsson AB, Erlang/OTP
>      
>     yasoho@REDACTED wrote:
>     >
>     > Hi, dear
>     >
>     >
>     > I use ct_telnet_client.erl, found that when a telnet command
>     returning
>     > message reaches 10MB, the Erlang process consumes memory up to
>     > 4~5GB, when the greater will memory overflow,.
>     >
>     >
>     > Can anybody help me, what is the cause? thanks a lot!
>     >
>     >
>     > B.R
>     >
>     ------------------------------------------------------------------------
>     > yasoho@REDACTED
>     >
>     ------------------------------------------------------------------------
>     >
>     > _______________________________________________
>     > erlang-questions mailing list
>     > erlang-questions@REDACTED
>     > http://erlang.org/mailman/listinfo/erlang-questions
>     >  
>      
>