[erlang-bugs] A dets big ?

[Tomas Abrahamsson]

>>>>> "Manuel" == Manuel Durán Aguete <manuel@REDACTED> writes:
>>
>> After upgrading a project from R14B03 to R16B01 I've found that
>> dets files are growing constanly after delete operations.
>>
>> I've uploaded a test case to github: http://kcy.me/oz1s
>
> After some research the problem seem to be related to a
> dets:traverse/2 after the dets file is opened.
>
> I've updated the code in github with new tests, now it calls dets
> directly.

Hi,

   I, too, have seen a(nother) strange issue with dets,
it might possibly be the same root cause.  I saw
bad_object_header errors.

A short summary of what I found is that when Erlang is
compiled with gcc 4.8.1, at least on a 32-bit linux, code in
efile_drv.c start acting unexpectedly, causing eg file:pread
with 2 positions to fail with {error,einval}.

I managed to boil my initial dets issues down to this test
program:

    -module(detstest).
    -export([start/0]).

    start() ->
        F = "detstest",
        file:delete(F),
        {error, enoent} = file:read_file_info(F),
        {ok, Db} = dets:open_file(d, [{file,F}, {keypos,2}]),
        %% need 2 entries for the error to show
        ok = dets:insert(Db, {a, 1}),
        ok = dets:insert(Db, {b, 2}),
        io:format("~p~n", [dets:close(Db)]).

If Erlang is compiled with gcc-4.8.1, it prints
{error,{bad_object_header,"detstest"}},
while I would have expected it to print just ok.

I traced the dets issue down to a call to file:pread, and
managed to boil it down a bit further to this test program:

    -module(pread).
    -export([start/0]).

    start() ->
        {ok,Fd} = file:open("pread-test", [write,raw,binary,read]),
        file:write(Fd, <<"abcdef\n">>),
        io:format("~p~n", [file:pread(Fd, [{0,1}])]),
        io:format("~p~n", [file:pread(Fd, [{1,1}])]),
        io:format("~p~n", [file:pread(Fd, [{0,1}, {1,1}])]),
        file:close(Fd).

When Erlang is compiled with gcc-4.8.1, it prints:

    {ok,[<<"a">>]}
    {ok,[<<"b">>]}
    {error,einval}

while with gcc-4.7.3, it prints what I would expect:

    {ok,[<<"a">>]}
    {ok,[<<"b">>]}
    {ok,[<<"a">>,<<"b">>]}

With the change in the diff below, pread, and also dets,
starts to behave as expected again, but this is not a
complete bugfix. I don't fully understand what's going on,
and similar code is at many places in efile_drv.c. That's
why I indented the diff---it is only an illustration---so
nobody can take it for a complete solution.

Even a seemingly unrelated change, eg inserting
fprintf(stdout,"sizeof(q)=%d",sizeof(q)); just before the if
statement, instead of applying the diff, will also make the
file:pread behave as expected. Perhaps the code is
runs into some kind of undefined behaviour in C?

Some info:
  - Erlang:  compiled OTP_R16B01 from git
  - System:  debian (unstable), core i5, 32-bits
  - Kernel:  3.10-1-686-pae
  - gcc:     gcc (Debian 4.8.1-8) 4.8.1
             with this version, pread fails with einval
  - gcc-4.7: gcc-4.7 (Debian 4.7.3-4) 4.7.3
             with this version, pread and dets behaves as expected.

I found I had the problem beacause debian's erlang
(r16.b.1-dfsg-4) was compiled with gcc-4.8, and I couldn't
recreate the issue when compiling from git, until I upgraded
gcc.

  diff --git a/erts/emulator/drivers/common/efile_drv.c b/erts/emulator/drivers/common/efile_drv.c
  index 595b048..65ce49a 100644
  --- a/erts/emulator/drivers/common/efile_drv.c
  +++ b/erts/emulator/drivers/common/efile_drv.c
  @@ -3746,9 +3746,14 @@ file_outputv(ErlDrvData e, ErlIOVec *ev) {
          for (i = 1; i < 1+n; i++) {
              Uint32 sizeH, sizeL;
              size_t size;
  -           if (   !EV_GET_UINT64(ev, &d->c.preadv.offsets[i-1], &p, &q)
  -               || !EV_GET_UINT32(ev, &sizeH, &p, &q)
  -               || !EV_GET_UINT32(ev, &sizeL, &p, &q)) {
  +           int gotOffsets;
  +           int gotSizeH;
  +           int gotSizeL;
  +
  +           gotOffsets = EV_GET_UINT64(ev, &d->c.preadv.offsets[i-1], &p, &q);
  +           gotSizeH   = !gotOffsets || EV_GET_UINT32(ev, &sizeH, &p, &q);
  +           gotSizeL   = !gotSizeH   || EV_GET_UINT32(ev, &sizeL, &p, &q);
  +           if (!gotOffsets || !gotSizeH || !gotSizeL) {
                  reply_posix_error(desc, EINVAL);
                  break;
              }


BRs
Tomas