[erlang-questions] DETS update or sync locking issue

[Art Beall]

Thanks for everybodys contributions to this email list! As Erlang newbies, the folks here have learned so much from reading this digest. Hopefully, we will be able to contribute as our experience grows.
 
We’ve been testing an erlang application that runs as a windows service that collects statistical data and stores timestamped records in a DETS table. The application takes data from a TCP message and stores historical data from the message into the table. Each record in the DETS table "grows" in size as the history is collected. The time span is truncated at 9 hours of data down to 6 hours. It continues to grow and truncate as time goes on. 

During the test, the data wasnt changing, so the actual size of the table was not growing. The memory usage was way higher than thought. It eventually crashed after a few hours exceeding 1.5 GB. Before it crashed, an erlang:garbage_collect() was issued from another Erlang node using remsh, but
this did not change anything. We’re using ramfile option with DETS, but sync’ing after every insert. 

A new process is created to parse each message and store the data. We are depending on the capability of the DETS table to allow con-current access through many processes. Crash dump viewer indicated there were thousands of processes waiting to update the dets table.

As a fix, the autosave option of dets was changed to 1000ms, and the call to dets:sync was removed from the callers process. This cleared up the memory
problem and the updates are very fast. Memory seems to be stable.

We're trying to understand why the dets update or sync hangs the processes.

Any help or explainations would be appreciated!
 
Art Beall
Virtual Hold Technology, LLC