Efficiency of big return functions?

[Dmytro Lytovchenko]

   1. ETS and Mnesia (with underlying ETS storage) will copy values on
   return. They are good if you do the query using their own mechanisms, and
   return only small data that is needed by your logic.
   2. Returning a root of a huge structure on the process heap is cheap
   (copying a pointer), but only for this same process. If you pass it to
   another process, a full copy will be created.
   (This matches your use case, if you do not let the data cross process
   boundaries, it will be efficient but consume more memory)
   3. Returning a static literal value hardcoded in your .ERL file is
   always cheap, until you reload the module and the static data you used may
   possibly be "garbage-collected" and moved to a regular process heap
   becoming just a regular big value (same problem as in (2) you can work
   efficiently in the same process but do not pass it between processes).


On Thu, 27 Aug 2020 at 11:51, Oliver Korpilla <oliver.korpilla@REDACTED>
wrote:

> Hello,
>
> I have some data that's between 100K and 1M in size, depending if I use
> the whole data set or just a part. Access is read-only.
>
> So far we kept a copy in each process for latency reasons and
> performance has been quite good. There can be quite a lot of processes
> (1,000s) but we have some big machines to run them on...
>
> So far we haven't considered ETS or Mnesia because all these processes
> would have to go through a single bottleneck in rather short order (they
> are truly parallel and independent from each other and we have lots of
> cores to schedule them on) - or am I wrong? How well does it scale?
>
> That said, we had good experiences with moving some static configuration
> information to code. Performance is really good, but that data was
> roughly of the format of a keyed map or smaller lists.
>
> The data we're looking at are big lists (potentially 1,000s of entries)
> of medium-sized maps, or maybe a map serving as index into these other
> maps.
>
> My question is - how do I efficiently return a big static value (a list
> of maps with no parameters to change their construction) from a
> function? Does BEAM optimize this? Or is the value constructed when the
> function is called? And is there anything I can do to improve it?
>
> Thank you!
> Oliver
>
>
> --
> Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
> https://www.avast.com/antivirus
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20200827/3562ede1/attachment.htm>