Port driver memory free

Casper <>
Wed Mar 23 06:15:40 CET 2005


More updates....

Today when I checked the system, the Beam memory consumption was 253MB.

- Eranga



-----Original Message-----
From: 
[mailto:] On Behalf Of

Sent: Wednesday, March 23, 2005 8:29 AM
To: 
Cc: 
Subject: Re: Port driver memory free

Raimo,

As you instructed I ran 
erlang:memory(), erlang:memory(binary) - "erlang_memory.txt"
erlang:system_info(allocated_areas), erlang:system_info(garbage_collection)
-
 "erlang_system_info_allocated_areas.txt"
erlang:system_info(allocator) - "erlang_system_info_allocator.txt"

As I can see Erlang memory fluctuates around  a 10MB total memory and 100KB
binary memory 
marks. Those outputs are in the attached text files, respectively.

Also I have attached to "TOP" output in the file "mem_usage.txt". As you can
see, when the program 
started around 12:50pm, Beam was using 56MB. At 18:55pm, it has gone up to
153MB.

Looks like there's a memory leak somewhere. Is there anyway to check the C
port memory 
allocation? How much it has been allocated? Or can that memory being some
cache or something?

If you find time, please go through my dump files and see if you can
identify where those memory 
gone.

Thanks in advance!
- Eranga



Quoting Raimo Niskanen : > Quoting Raimo Niskanen :

> It is not certain you have a leak. It may be just fragmented memory.
> Try erlang:memory(), and/or also erlang:memory(binary) to see how
> the emulator\'s internal view of allocated memory varies over the
> time, and compare that to the crude figure from \'top\'.
> 
> It is probably so that after your application has started it takes
> some minutes / an hour for it to reach a quiescent state. And in
> that state there is a lot of binaries allocated, being allocated
> and being freed, so if erlang:memory(binary) asymtotically rises
> to a mean value, fluctuating around it, there is no problem. If
> on the other hand, erlang:memory(binary) rises linearily forever,
> you have a memory leak of binaries. 
> 
> The value from \'top\' can differ a lot from what erlang:memory()
> reports, due to memory fragmentation.
> 
> You can also read \"erl -man instrument\" for an advanced feature
> (I have never used) to analyze memory allocation in the emulator.
> 
> 
> 
>  (Casper) writes:
> 
> > Hi Raimo,
> > 
> > Thank you so much for enlighten me on the subject. I was very unclear
> about
> > the binary mode and list mode and I thought even on the Binary mode I
can
> > still use the already allocated memory in the **rbuf, if the total
output
> is
> > less than 64 bytes.
> > 
> > Today I loaded my Erlang work and used \"TOP\" unix command to monitor
the
> > Beam process and the CPU load. I noticed that Beam process memory size
> goes
> > up. Initially it was consuming about 36MB. But after about 4 hours
later,
> > i.e. after passing more than 800,000 messages between Erlang and C Port,
I
> > noticed that memory usage of Beam has gone up to 100MB.
> > 
> > As you instructed I don\'t use driver_free_binary. Could that be because
I
> > don\'t free it? I checked my C code thoroughly but cannot find any other
> > memory leak area.
> > 
> > Please advice!
> > 
> > Thanks!
> > - Eranga
> > 
> > 
> > 
> > 
> > 
> > -----Original Message-----
> > From: 
> > [mailto:] On Behalf Of Raimo Niskanen
> > Sent: Tuesday, March 22, 2005 3:06 PM
> > To: 
> > Subject: Re: Port driver memory free
> > 
> > Ok, to repeat once again. For port_control in binary mode, you must
> > _always_ return a binary. You can _not_ return the result in the
> > preallocated buffer *rbuf -it can only be used in non-binary
> > mode.
> > 
> > So, your loop that runs twice, once for size determination, and
> > once for encoding, should end in:
> > if (p)
> > break;
> > *rbuf = driver_alloc_binary(*rindex);
> > drv_binary = (ErlDrvBinary *) *rbuf;
> > p = drv_binary->orig_bytes;
> > *rindex = 0;
> > }
> > Now p always gets set for the second iteration, and *rbuf always
> > gets set to a binary. That ought to work.
> > 
> > 
> >  writes:
> > 
> > > Raimo,
> > > 
> > > Actually I allocate a binary. Here\'s what I do,
> > > 
> > > ----------------------- C Port ----------------------------- 
> > > int gen_reply(char **rbuf, int rlen, int *rindex, void *buffer, int
> > buflen) {
> > > ErlDrvBinary *drv_binary;
> > > char *p;
> > > 
> > > *rindex = 0;
> > > ........
> > > ........
> > > for (p = NULL;;) {
> > > if (ei_encode_version(p, rindex)
> > > || ei_encode_tuple_header(p, rindex, 2)
> > > || ei_encode_atom(p, rindex, \"ok\")
> > > || ei_encode_binary(p, rindex, buffer,
> > buflen)) {
> > > DBG(\"ei_encode failed\");
> > > return((int) ERL_DRV_ERROR_ERRNO);
> > > }
> > > if (p)
> > > break;
> > > //if (*rindex > rlen) {
> > > *rbuf = driver_alloc_binary(*rindex);
> > > //}
> > > drv_binary = (ErlDrvBinary *) *rbuf;
> > > p = drv_binary->orig_bytes;
> > > *rindex = 0;
> > > }
> > > 
> > > drv_binary->orig_size = *rindex;
> > > ........
> > > ----------------------------------------------------------
> > > 
> > > I had to comment out the \"(*rindex > rlen)\" part since when it was
> there,
> > what I received in the 
> > > emulator side was garbage after 8th char position.
> > > 
> > > ------------------------- Emulator -----------------------
> > > case catch binary_to_term(erlang:port_control(Port, Command,
> > 
> > > term_to_binary(Args))) of 
> > > {\'EXIT\', Reason} ->
> > > .......
> > > ----------------------------------------------------------
> > > 
> > > On the emulator side binary_to_term BIF didn\'t give any error. But
once
> > receive the term, the binary 
> > > part contain incorrect data after 8th char pos. 
> > > 
> > > Thanks!
> > > - Eranga
> > > 
> > > 
> > > 
> > > 
> > > -----Original Message-----
> > > From: 
> > > [mailto:] On Behalf Of Raimo Niskanen
> > > Sent: Friday, March 18, 2005 2:31 PM
> > > To: 
> > > Subject: Re: Port driver memory free
> > > 
> > > Yes, you do it wrong. The returned value _must_ be a binary, 
> > > (or NULL for R10B or later). The code that takes care of the value
> > > assumes it is a binary, so you have introduced something in your
> > > system that is assumed to be a binary, but is actually stack allocated
> > > plain memory. The emulator will probably crash much later because
> > > memory has been trashed. 
> > > 
> > > You will get partially correct behaviour because the receiving code 
> > > looks for the contents of the binary, and finds something there. 
> > > Unfortunately the header is not correct and it has not been allocated 
> > > with driver_alloc_binary(), but will later be freed with 
> > > driver_free_binary(), and both the contents and header will be
> > > overwritten with garbage as soon as the emulator C stack grows.
> > > 
> > > Do not do that - you will get hurt!
> > > 
> > > 
> > > 
> > >  (Casper) writes:
> > > 
> > > > Hi,
> > > > 
> > > > Today I noticed a funny thing with Port drivers \\\"control\\\"
call.
> > > > 
> > > > I set the mode to PORT_CONTROL_FLAG_BINARY and use **rbuf as it is
> > > (without
> > > > reinitializing), since the space I need to output the binary message
> is
> > > less
> > > > than 64 bytes.
> > > > 
> > > > When I decode the received binary message in the emulator side, up
to
> 7
> > > > bytes were exactly as I encoded in the C Port side. After that 8
> > > position
> > > > onward I get some other value.
> > > > 
> > > > Then I created a new binary using driver_alloc_binary, even though
the
> > > > output (encoded, total) is less than 64 bytes and tried, and viola,
> the
> > > > correct message came to emulator side.
> > > > 
> > > > If that a bug or something I do wrong?
> > > > 
> > > > Thanks!
> > > > - Eranga
> > > > 
> > > > 
> > > > 
> > > > -----Original Message-----
> > > > From: 
> > > > [mailto:] On Behalf Of Raimo
Niskanen
> > > > Sent: Thursday, March 17, 2005 2:17 PM
> > > > To: 
> > > > Subject: Re: Port driver memory free
> > > > 
> > > > The short answer is: do not free! 
> > > > 
> > > > To elaborate:
> > > > * In binary mode, you must return a binary:
> > > > ErlDrvBinary *binp = driver_alloc_binary(len);
> > > > /* Copy len bytes of data to to binp->orig_bytes */
> > > > *rbuf = (char *)binp;
> > > > return len;
> > > > The caller (the emulator) will increment the refcount and take care 
> > > > of deallocation when the binary is no longer used.
> > > > * In list mode, if the supplied result buffer is too small, you
> > > > should allocate a new result buffer using driver_alloc(size),
> > > > and the caller (the emulator) will free it it has fetched the
> > > > result from it. Otherwise just write the result in the supplied
> > > > result buffer. The emulator compares the returned buffer address
> > > > with the supplied buffer address to see if you return an allocated
> > > > buffer. So, if you set *rbuf, it _must_ be to something allocated 
> > > > with driver_alloc(size), because it _will_ be freed with 
> > > > driver_free(*rbuf). You can of course return an allocated buffer
> > > > for any size, also less than 64 bytes.
> > > > 
> > > > 
> > > > 
> > > >  (Casper) writes:
> > > > 
> > > > > Hi All,
> > > > > 
> > > > > There are 2 parameters, **rbuf and rlen in Port Driver entry
> > \\\"control.
> > > In
> > > > > PORT_CONTROL_FLAG_BINARY mode, I should pass a driver_binary in
> > > **rbuf.
> > > > > Usually rlen is 64, which mean **rbuf has 64 bytes, if I need to
use
> > > that
> > > > > without creating a new ErlDrvBinary. My questions are,
> > > > > 
> > > > > 1. If my final ErlDrvBinary size is < 64 bytes and I use **rbuf
> > > without
> > > > > allocating a new ErlDrvBinary, should I still call
> driver_free_binary
> > > or
> > > > > driver_free to free that **rbuf memory before \\\"control\\\"
function
> > > returns?
> > > > > 
> > > > > 2. If my final ErlDrvBinary size is > 64 bytes and I allocate a
new
> > > > > driver_binary and assign to **rbuf, what will happen to the
previous
> > > 64
> > > > > buffer which was in **rbuf? Will the port or erlang garbage
> collecter
> > > free
> > > > > that? Or do I need to free it inside the \\\"control\\\" function
by
> > calling
> > > > > driver_free_binary or driver_free?
> > > > > 
> > > > > 3. I (2) above, what will be the case if I use driver_realloc or
> > > > > driver_realloc_binary? Will the previous 64 byes get released?
> > > > > 
> > > > > 4. I (2) above, I still need to call driver_free_binary to free
the
> > > newly
> > > > > created driver_binary inside \\\"control\\\" function, correct?
> > > > > 
> > > > > 5. If I call \\\"set_port_control_flags(dd->port, 0)\\\" to set
the port
> > > output
> > > > to
> > > > > non-binary, do I need to free or clear the non-used space of
**rbuf?
> > > > > 
> > > > > 6. In above cased which free function, driver_free_binary or
> > > driver_free
> > > > and
> > > > > which allocation function, driver_alloc, driver_alloc_binary,
> > > > > driver_realloc, driver_realloc_binary should I use?
> > > > > 
> > > > > Thanks in advance!
> > > > > - Eranga
> > > > > 
> > > > > 
> > > > 
> > > > -- 
> > > > 
> > > > / Raimo Niskanen, Erlang/OTP, Ericsson AB
> > > > 
> > > 
> > > -- 
> > > 
> > > / Raimo Niskanen, Erlang/OTP, Ericsson AB
> > > 
> > > --------------This mail sent through OmniBIS.com--------------
> > > 
> > 
> > -- 
> > 
> > / Raimo Niskanen, Erlang/OTP, Ericsson AB
> > 
> 
> -- 
> 
> / Raimo Niskanen, Erlang/OTP, Ericsson AB
> 

--------------This mail sent through OmniBIS.com--------------




More information about the erlang-questions mailing list