[erlang-questions] Changing the representation of sets

[Michael Truog]

On 04/25/2017 06:17 AM, Jesper Louis Andersen wrote:
> On Tue, Apr 25, 2017 at 1:17 PM Attila Rajmund Nohl <attila.r.nohl@REDACTED <mailto:attila.r.nohl@REDACTED>> wrote:
>
>     It would affect everybody who saves the opaque data using an earlier
>     OTP version, then reads it in newer OTP version (e.g. after upgrade).
>     Or those who run two nodes on different OTP versions (e.g. during an
>     upgrade).
>
>
> I like to think of dict and sets to be "opaque" data structures. If you are relying on their internal representation, you will run into trouble. So changing the representation going forward should definitely be possible.
>
> This leaves backwards compatibility. If the maps-optimized module can read and dynamically change old sets implementations into the new format on the fly, it may be possible to gradually replace the old representations with the new.
>
> Some things to look out for in that process is data-at-rest, stored years ago in a database. At some point you would have to reprocess such data, or supply a conversion module which can handle these old formats.
>
> I also think that any system must provide some measure of pushing things forward. That is, each major release of Erlang could contain a limited set of things you now have to do differently, with a clear upgrade path from earlier versions. As long as the set is limited, we can probably handle the rewrites needed. If you value backwards compatibility for forever, you run the risk of getting stale, never upgrading anything.
>
> As for the increased memory copy pressure: I think this should be fixed in the context of "maps" and not be part of the argument as to why one would keep the old sets representation.
The memory allocation related to maps is likely not something to solve, and should be related to its data structure (HAMT (Hash-Array Mapped Trie)).  Keeping things stable is important and there should never be changes that sacrifice stability for a passing whim.  It is easy to add new modules for maps used for sets and it is important to avoid adversely affecting legacy source code.  That doesn't block future improvements, it only makes sure they are focused on being improvements.

Best Regards,
Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20170425/902585ba/attachment.htm>