[erlang-questions] os_mon memsup reporting memory consumption

[Michael McDaniel]

I have been using os_mon,memsup for monitoring a few servers since
last year.  I went back into the code I am using and found
my code comments ...


% Used with module autosys_logger, this module does the following:
%
% application:starts sasl, os_mon, autosys_logger, and then sends
% reports via email (using email.erl) every ?? minutes.
% Uses autosys_alarm.conf
%
% -record(alarm, {report_subject    ='REPORT delora' ,
%                 repeat_time       = 14400000 ,
%                 mem_percent_alarm = 85 , % alarm when this percent mem used
%                 emailList         = ["a_reporting_account@REDACTED", ...]
%                }).
%
% to force a certain amount of unused memory (Linux) do
%   # /sbin/sysctl -w vm.min_free_kbytes=60000
% otherwise "all" the memory may get used in buffers/cached and a single
% high_memory alarm will happen but never again (because when processes
% want memory they'll take it from buffer/cache and I'll never know
% if processes are going memory crazy).
%
% The 6000 above is nominal for a 1/2 GigByte system.  Adjust it as
% needed for amount of memory in system, and depending on what percent
% the alarm is set in alarm record.
%
%   # free -k (or -m) to watch free/buffers/cached change
%


Your mileage may vary regarding the vm.min_free_kbytes value and
how it affects your application.

A hack, of course, to make memsup useable in my (Linux) environment(s).

I would rather not see backwards compatibility breakage.

Perhaps os_mon,memsup could get a new system_memory_high_watermark_linux
threshold which would trigger a different memory calculation ?

Existing code using system_memory_high_watermark would be unaffected and
new (Linux) code could use the new _linux threshold.


~Michael




On Mon, Sep 25, 2006 at 02:29:13PM +0200, Gunilla Arendt wrote:
> Hi
> 
> I see the point, but it would be a non-backward compatible change
> and as you know we're very hesitant about those.
> 
> An alternative solution could be to add a new tag to the result of
> memsup:get_system_memory_data() for Linux systems, although that would
> not affect the setting/clearing of alarms.
> 
> If anybody else has thoughts about this, please feel free to share.
> 
> Regards,
> Gunilla, Erlang/OTP team
> 
> 
> 
> Serge Aleynikov wrote:
> > We had a few high memory watermark alarms in a production system running 
> > Linux reported by OS_MON whereas it seemed like the memory utilization 
> > on the server wasn't bad.  I looked at the implementation of the 
> > memsup.erl module in OS_MON, and I believe the calculation of allocated 
> > memory has a flow.
> > 
> > Memsup's code (sited at the end) reveals that allocated memory doesn't 
> > take in consideration cached memory.  Since cached memory can be quickly 
> > freed by the Linux memory manager if additional memory is needed by 
> > applications, it looks to me that it should be:
> > 
> > Allocated = MemTotal - (MemFree + Cached)
> > 
> > or even:
> > 
> > Allocated = MemTotal - (MemFree + Buffers + Cached)
> > 
> > This would be consistent with what "free" is showing on the "-/+ 
> > buffers/cache" line:
> > 
> > drp@REDACTED: ~$ uname -a
> > Linux drpdb02.corp.idt.net 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 EST 
> > 2005 i686 i686 i386 GNU/Linux
> > 
> > drp@REDACTED: ~$ free
> >              total      used       free     shared    buffers     cached
> > Mem:       4150324   3416788     733536          0     282412    2621788
> > -/+ buffers/cache:    512588    3637736
> > Swap:      2048248         0    2048248
> > 
> > drp@REDACTED: ~$ head -4 /proc/meminfo
> > MemTotal:      4150324 kB
> > MemFree:        733536 kB
> > Buffers:        282412 kB
> > Cached:        2621788 kB
> > 
> > 
> > 
> > %-------------------------
> > %% @spec get_memory_usage_linux() -> {Allocated, Total}
> > %-------------------------
> > get_memory_usage_linux() ->
> >     Res = os:cmd("cat /proc/meminfo"),
> >     case get_memory_usage_linux(Res, undef, undef) of
> >     {MemTotal, MemFree} ->
> >         {MemTotal-MemFree, MemTotal};
> >     error ->
> >         timer:sleep(1000),
> >         get_memory_usage_linux()
> >     end.
> > 
> > get_memory_usage_linux(_Str, Tot, Free) when is_integer(Tot),
> >                          is_integer(Free) ->
> >     {Tot, Free};
> > get_memory_usage_linux("MemTotal:"++T, _Tot0, Free0) ->
> >     {ok, [N], Rest} = io_lib:fread("~d", T),
> >     Tot = N*1024,
> >     get_memory_usage_linux(skip_to_eol(Rest), Tot, Free0);
> > get_memory_usage_linux("MemFree:"++T, Tot0, _Free0) ->
> >     {ok, [N], Rest} = io_lib:fread("~d", T),
> >     Free = N*1024,
> >     get_memory_usage_linux(skip_to_eol(Rest), Tot0, Free);
> > get_memory_usage_linux("", _Tot0, _Free0) ->
> >     error;
> > get_memory_usage_linux(Str, Tot0, Free0) ->
> >     get_memory_usage_linux(skip_to_eol(Str), Tot0, Free0).
> > %-------------------------
> > 
> > Regards,
> > 
> > Serge
> > 
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions

-- 
Michael McDaniel
Portland, Oregon, USA
http://autosys.us
+1 503 283 5284