[erlang-questions] disk_log and MD5 of big terms

[Hans Bolinder]

[Scott Lystig Fritchie:]
> Howdy.  I'm quite curious why disk_log insists on calculating the MD5
> checksum of terms larger than approx 64KB.  What's the reason behind it?

The reason is that the file driver has been optimized in order to
avoid copying data unnecessarily: before reading bytes from the file,
an area in RAM large enough to hold all of the requested bytes is
allocated. Therefore disk_log (and Dets, and any other application
storing objects tagged with size on file) must not call the file
driver unless the size is smaller than some harmless limit or it has
been verified that the size is indeed correct.

In this particular case (disk_log), the MD5 checksum was introduced
after a customer had experienced problems (read: core dumps) after a
power failure.

Best regards,

Hans Bolinder, Erlang OTP/team