[erlang-questions] Linux O_DIRECT flag

[Valentin Micic]

Well, part of my problem is that I did not do investigation myself, but was 
relaying on other people to do it for me. I have to admit, this is the best 
way to jump to conclusion. However,  I've been looking at tuning the cache 
during "first iteration" about few months ago (this was limited to kernel 
investigation, hence never consider bugging erlang list about that). I have 
found quite a few parameters (I cannot remember their names from the top of 
my head) that directs how often, and under which circumstances should data 
be committed to disk, however, none of them explicitly limit the amount of 
memory that may be used for caching. Committing data to disk does not imply 
de-allocating the memory.

The nature of a problem (if I may call it that, since the system has been in 
production with very good results for last six months without problems), is 
that I need a large amount of RAM to be always in RAM ;-) -- the system in 
question is a pre-paid voucher server that holds basic voucher data in RAM, 
and the rest on disk; business rules are such that any unused voucher has to 
be kept on a system (and hence in RAM) for a year before it may be 
discarded.

Now, even if I turn swappiness to 0, this would not guarantee that kernel is 
not going to swap the application out if "distress" value is high enough, 
which, in turn, mean that our application will end-up in un-interruptible 
sleep, causing whole bunch of timeouts and unpredictable behavior. So, in 
order to ensure that application is not swapped out, I need to keep 
"distress" values low enough, and, as I cannot keep my RAM out of RAM ;-), 
the best next thing was to get RAM back from the cache (or rather, to bypass 
it), hence O_DIRECT approach. Meanwhile, in my previous e-mail, responding 
to Frej, I've conceded that O_DIRECT is not needed if posix_fadvice does the 
trick of hinting to kernel that it may free particular cache timeously. But 
then, one may argue that kernel would do just that anyway, without me having 
to do a thing. But... can I trust it?

V.


----- Original Message ----- 
From: "Toby Thain" <toby@REDACTED>
To: "Per Hedeland" <per@REDACTED>
Cc: <valentin@REDACTED>; <erlang-questions@REDACTED>
Sent: Saturday, March 15, 2008 5:10 PM
Subject: Re: [erlang-questions] Linux O_DIRECT flag


>
> On 14-Mar-08, at 9:23 PM, Per Hedeland wrote:
>
>> ...
>> is why "some other" OSes implement some "push-back" mechanisms for the
>> cache, or in older times simply had a hard limit as percentage of RAM.
>>
>> I don't know off-hand whether there is a way to make Linux do  something
>> like that, but I would think so (at least the hard limit method, maybe
>> with a number calculated by the user) - but the default is AFAIK "give
>> the cache all it wants".
>
> Valentin, have you investigated tuning/limiting the cache. OTOH I  don't 
> know whether it's possible but it seems strange that the  capability would 
> have been entirely omitted.
>
> --Toby
>
>