[erlang-questions] term_to_binary and record improvements

[Kenneth Lundin]

Hi,

We have discussed if we should implement sharing of data structures
while copying and when
creating the external representation. Our thoughts goes like this:

- The lack of sharing has never been brought up as a significant
problem for telecom and other
com or HA products built with erlang. Therefore we have not
prioritized a solution for this.

- Keeping the sharing while copying has impact on:
    - Message passing
    - insertion in ets-tables
    - the external format for Erlang terms among other things used in
the Erlang distribution and produced by term_to_binary.

A sharing preserving solution for the cases above will be more costly
than the current implementation since it is not possibly to destroy
the source term as can be done in the GC case.

Because of the negative impact on performance in cases where there is
no sharing or need for sharing, which definitely is the dominating
case we don't think it is feasible to implement and use a sharing
solution as default for the cases above.

What could be done is to offer functions for the programmer to
specifically use sharing,
for example by variants or new options to the send, ets:insert and
term_to_binary BIF's
We would also need to add and support new tags in the external format
and fix some
compatibility issues like not sending the new external format to old
Erlang nodes.
A significant amount of work in total.

Currently this is not one of the most prioritized items on our development list.
But there might be possibilities to change that if someone has strong
arguments, financing etc.

/Regards Kenneth, Erlang/OTP team, Ericsson

On Fri, Aug 29, 2008 at 10:03 AM, Ulf Wiger (TN/EAB)
<ulf.wiger@REDACTED> wrote:
> Richard A. O'Keefe skrev:
>>
>> Question:  in existing Erlang use, which is more important,
>> speed of generating a binary encoding for a term, or how big
>> it is (which relates to how quickly it can be sent across a
>> network and how fast it can be decoded, amongst other things).
>
> Having a sharing-preserving term_to_binary() would be a start,
> but as long as local message passing doesn't preserve sharing
> (neither does spawn()), we would need some big fat warning
> labels on the deliberate use of sharing in data structures.
>
> Come to think of it, we need those warning labels even today.
>
> There are very few corners in Erlang where a seemingly
> innocent code modification can cause havoc, but accidental
> serialization of a complex data structure is one of them.
>
> I can say from experience, as can John Hughes, that it's no
> fun at all when a seemingly well-working program suddenly
> blows up, rendering your computer useless for minutes while
> it tries to allocate an enormous glob of memory.
>
> In both cases, sharing was deliberately used, and in both
> cases the fact that the data structure is flattened when
> sent to another local process came as a surprise. In my
> case, a subtle mistake (inheriting the whole record in a
> fun, instead of just a single attribute) caused the blowup.
> In John's case, there was truly a need to pass the data to
> another process, but no way to do it without causing memory
> explosion.
>
> BEAM already has the ability to do sharing-preserving copy:
> the garbage collector obviously does it. But the feature is
> not available to the lowly Erlang programmer.
>
> BR,
> Ulf W
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>