Port driver memory free
Raimo Niskanen
raimo@REDACTED
Wed Mar 23 09:52:31 CET 2005
You should have a closer look at what the OS thinks about the Beam
process' memory allocation.
If you are running linux, firs get the pid of the Beam process,
e.g from os:getpid().
Then have a look at /proc/[BeamPid]/maps and /proc/[BeamPid]/statm,
see proc(5 or 4) man page, and watch how they change over time.
casper2000a@REDACTED writes:
> Raimo,
>
> As you instructed I ran
> erlang:memory(), erlang:memory(binary) - "erlang_memory.txt"
> erlang:system_info(allocated_areas), erlang:system_info(garbage_collection) -
> "erlang_system_info_allocated_areas.txt"
> erlang:system_info(allocator) - "erlang_system_info_allocator.txt"
>
> As I can see Erlang memory fluctuates around a 10MB total memory and 100KB binary memory
> marks. Those outputs are in the attached text files, respectively.
>
> Also I have attached to "TOP" output in the file "mem_usage.txt". As you can see, when the program
> started around 12:50pm, Beam was using 56MB. At 18:55pm, it has gone up to 153MB.
>
> Looks like there's a memory leak somewhere. Is there anyway to check the C port memory
> allocation? How much it has been allocated? Or can that memory being some cache or something?
>
> If you find time, please go through my dump files and see if you can identify where those memory
> gone.
>
> Thanks in advance!
> - Eranga
>
>
>
> Quoting Raimo Niskanen : > Quoting Raimo Niskanen :
>
> > It is not certain you have a leak. It may be just fragmented memory.
> > Try erlang:memory(), and/or also erlang:memory(binary) to see how
> > the emulator\'s internal view of allocated memory varies over the
> > time, and compare that to the crude figure from \'top\'.
> >
> > It is probably so that after your application has started it takes
> > some minutes / an hour for it to reach a quiescent state. And in
> > that state there is a lot of binaries allocated, being allocated
> > and being freed, so if erlang:memory(binary) asymtotically rises
> > to a mean value, fluctuating around it, there is no problem. If
> > on the other hand, erlang:memory(binary) rises linearily forever,
> > you have a memory leak of binaries.
> >
> > The value from \'top\' can differ a lot from what erlang:memory()
> > reports, due to memory fragmentation.
> >
> > You can also read \"erl -man instrument\" for an advanced feature
> > (I have never used) to analyze memory allocation in the emulator.
> >
> >
> >
> > casper2000a@REDACTED (Casper) writes:
> >
> > > Hi Raimo,
> > >
> > > Thank you so much for enlighten me on the subject. I was very unclear
> > about
> > > the binary mode and list mode and I thought even on the Binary mode I can
> > > still use the already allocated memory in the **rbuf, if the total output
> > is
> > > less than 64 bytes.
> > >
> > > Today I loaded my Erlang work and used \"TOP\" unix command to monitor the
> > > Beam process and the CPU load. I noticed that Beam process memory size
> > goes
> > > up. Initially it was consuming about 36MB. But after about 4 hours later,
> > > i.e. after passing more than 800,000 messages between Erlang and C Port, I
> > > noticed that memory usage of Beam has gone up to 100MB.
> > >
> > > As you instructed I don\'t use driver_free_binary. Could that be because I
> > > don\'t free it? I checked my C code thoroughly but cannot find any other
> > > memory leak area.
> > >
> > > Please advice!
> > >
> > > Thanks!
> > > - Eranga
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: owner-erlang-questions@REDACTED
> > > [mailto:owner-erlang-questions@REDACTED] On Behalf Of Raimo Niskanen
> > > Sent: Tuesday, March 22, 2005 3:06 PM
> > > To: erlang-questions@REDACTED
> > > Subject: Re: Port driver memory free
> > >
> > > Ok, to repeat once again. For port_control in binary mode, you must
> > > _always_ return a binary. You can _not_ return the result in the
> > > preallocated buffer *rbuf -it can only be used in non-binary
> > > mode.
> > >
> > > So, your loop that runs twice, once for size determination, and
> > > once for encoding, should end in:
> > > if (p)
> > > break;
> > > *rbuf = driver_alloc_binary(*rindex);
> > > drv_binary = (ErlDrvBinary *) *rbuf;
> > > p = drv_binary->orig_bytes;
> > > *rindex = 0;
> > > }
> > > Now p always gets set for the second iteration, and *rbuf always
> > > gets set to a binary. That ought to work.
> > >
> > >
> > > casper2000a@REDACTED writes:
> > >
> > > > Raimo,
> > > >
> > > > Actually I allocate a binary. Here\'s what I do,
> > > >
> > > > ----------------------- C Port -----------------------------
> > > > int gen_reply(char **rbuf, int rlen, int *rindex, void *buffer, int
> > > buflen) {
> > > > ErlDrvBinary *drv_binary;
> > > > char *p;
> > > >
> > > > *rindex = 0;
> > > > ........
> > > > ........
> > > > for (p = NULL;;) {
> > > > if (ei_encode_version(p, rindex)
> > > > || ei_encode_tuple_header(p, rindex, 2)
> > > > || ei_encode_atom(p, rindex, \"ok\")
> > > > || ei_encode_binary(p, rindex, buffer,
> > > buflen)) {
> > > > DBG(\"ei_encode failed\");
> > > > return((int) ERL_DRV_ERROR_ERRNO);
> > > > }
> > > > if (p)
> > > > break;
> > > > //if (*rindex > rlen) {
> > > > *rbuf = driver_alloc_binary(*rindex);
> > > > //}
> > > > drv_binary = (ErlDrvBinary *) *rbuf;
> > > > p = drv_binary->orig_bytes;
> > > > *rindex = 0;
> > > > }
> > > >
> > > > drv_binary->orig_size = *rindex;
> > > > ........
> > > > ----------------------------------------------------------
> > > >
> > > > I had to comment out the \"(*rindex > rlen)\" part since when it was
> > there,
> > > what I received in the
> > > > emulator side was garbage after 8th char position.
> > > >
> > > > ------------------------- Emulator -----------------------
> > > > case catch binary_to_term(erlang:port_control(Port, Command,
> > >
> > > > term_to_binary(Args))) of
> > > > {\'EXIT\', Reason} ->
> > > > .......
> > > > ----------------------------------------------------------
> > > >
> > > > On the emulator side binary_to_term BIF didn\'t give any error. But once
> > > receive the term, the binary
> > > > part contain incorrect data after 8th char pos.
> > > >
> > > > Thanks!
> > > > - Eranga
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: owner-erlang-questions@REDACTED
> > > > [mailto:owner-erlang-questions@REDACTED] On Behalf Of Raimo Niskanen
> > > > Sent: Friday, March 18, 2005 2:31 PM
> > > > To: erlang-questions@REDACTED
> > > > Subject: Re: Port driver memory free
> > > >
> > > > Yes, you do it wrong. The returned value _must_ be a binary,
> > > > (or NULL for R10B or later). The code that takes care of the value
> > > > assumes it is a binary, so you have introduced something in your
> > > > system that is assumed to be a binary, but is actually stack allocated
> > > > plain memory. The emulator will probably crash much later because
> > > > memory has been trashed.
> > > >
> > > > You will get partially correct behaviour because the receiving code
> > > > looks for the contents of the binary, and finds something there.
> > > > Unfortunately the header is not correct and it has not been allocated
> > > > with driver_alloc_binary(), but will later be freed with
> > > > driver_free_binary(), and both the contents and header will be
> > > > overwritten with garbage as soon as the emulator C stack grows.
> > > >
> > > > Do not do that - you will get hurt!
> > > >
> > > >
> > > >
> > > > casper2000a@REDACTED (Casper) writes:
> > > >
> > > > > Hi,
> > > > >
> > > > > Today I noticed a funny thing with Port drivers \\\"control\\\" call.
> > > > >
> > > > > I set the mode to PORT_CONTROL_FLAG_BINARY and use **rbuf as it is
> > > > (without
> > > > > reinitializing), since the space I need to output the binary message
> > is
> > > > less
> > > > > than 64 bytes.
> > > > >
> > > > > When I decode the received binary message in the emulator side, up to
> > 7
> > > > > bytes were exactly as I encoded in the C Port side. After that 8
> > > > position
> > > > > onward I get some other value.
> > > > >
> > > > > Then I created a new binary using driver_alloc_binary, even though the
> > > > > output (encoded, total) is less than 64 bytes and tried, and viola,
> > the
> > > > > correct message came to emulator side.
> > > > >
> > > > > If that a bug or something I do wrong?
> > > > >
> > > > > Thanks!
> > > > > - Eranga
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: owner-erlang-questions@REDACTED
> > > > > [mailto:owner-erlang-questions@REDACTED] On Behalf Of Raimo Niskanen
> > > > > Sent: Thursday, March 17, 2005 2:17 PM
> > > > > To: erlang-questions@REDACTED
> > > > > Subject: Re: Port driver memory free
> > > > >
> > > > > The short answer is: do not free!
> > > > >
> > > > > To elaborate:
> > > > > * In binary mode, you must return a binary:
> > > > > ErlDrvBinary *binp = driver_alloc_binary(len);
> > > > > /* Copy len bytes of data to to binp->orig_bytes */
> > > > > *rbuf = (char *)binp;
> > > > > return len;
> > > > > The caller (the emulator) will increment the refcount and take care
> > > > > of deallocation when the binary is no longer used.
> > > > > * In list mode, if the supplied result buffer is too small, you
> > > > > should allocate a new result buffer using driver_alloc(size),
> > > > > and the caller (the emulator) will free it it has fetched the
> > > > > result from it. Otherwise just write the result in the supplied
> > > > > result buffer. The emulator compares the returned buffer address
> > > > > with the supplied buffer address to see if you return an allocated
> > > > > buffer. So, if you set *rbuf, it _must_ be to something allocated
> > > > > with driver_alloc(size), because it _will_ be freed with
> > > > > driver_free(*rbuf). You can of course return an allocated buffer
> > > > > for any size, also less than 64 bytes.
> > > > >
> > > > >
> > > > >
> > > > > casper2000a@REDACTED (Casper) writes:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > There are 2 parameters, **rbuf and rlen in Port Driver entry
> > > \\\"control.
> > > > In
> > > > > > PORT_CONTROL_FLAG_BINARY mode, I should pass a driver_binary in
> > > > **rbuf.
> > > > > > Usually rlen is 64, which mean **rbuf has 64 bytes, if I need to use
> > > > that
> > > > > > without creating a new ErlDrvBinary. My questions are,
> > > > > >
> > > > > > 1. If my final ErlDrvBinary size is < 64 bytes and I use **rbuf
> > > > without
> > > > > > allocating a new ErlDrvBinary, should I still call
> > driver_free_binary
> > > > or
> > > > > > driver_free to free that **rbuf memory before \\\"control\\\" function
> > > > returns?
> > > > > >
> > > > > > 2. If my final ErlDrvBinary size is > 64 bytes and I allocate a new
> > > > > > driver_binary and assign to **rbuf, what will happen to the previous
> > > > 64
> > > > > > buffer which was in **rbuf? Will the port or erlang garbage
> > collecter
> > > > free
> > > > > > that? Or do I need to free it inside the \\\"control\\\" function by
> > > calling
> > > > > > driver_free_binary or driver_free?
> > > > > >
> > > > > > 3. I (2) above, what will be the case if I use driver_realloc or
> > > > > > driver_realloc_binary? Will the previous 64 byes get released?
> > > > > >
> > > > > > 4. I (2) above, I still need to call driver_free_binary to free the
> > > > newly
> > > > > > created driver_binary inside \\\"control\\\" function, correct?
> > > > > >
> > > > > > 5. If I call \\\"set_port_control_flags(dd->port, 0)\\\" to set the port
> > > > output
> > > > > to
> > > > > > non-binary, do I need to free or clear the non-used space of **rbuf?
> > > > > >
> > > > > > 6. In above cased which free function, driver_free_binary or
> > > > driver_free
> > > > > and
> > > > > > which allocation function, driver_alloc, driver_alloc_binary,
> > > > > > driver_realloc, driver_realloc_binary should I use?
> > > > > >
> > > > > > Thanks in advance!
> > > > > > - Eranga
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > / Raimo Niskanen, Erlang/OTP, Ericsson AB
> > > > >
> > > >
> > > > --
> > > >
> > > > / Raimo Niskanen, Erlang/OTP, Ericsson AB
> > > >
> > > > --------------This mail sent through OmniBIS.com--------------
> > > >
> > >
> > > --
> > >
> > > / Raimo Niskanen, Erlang/OTP, Ericsson AB
> > >
> >
> > --
> >
> > / Raimo Niskanen, Erlang/OTP, Ericsson AB
> >
>
> --------------This mail sent through OmniBIS.com--------------
>
>
>
>
--
/ Raimo Niskanen, Erlang/OTP, Ericsson AB
More information about the erlang-questions
mailing list