[erlang-questions] shared records in .hrl files

[Felix Gallo]

Consider a system that receives application messages in an extensibly large
number of formats (e.g. json, xml, yaml, pictograms, esperanto).  This
system wishes to translate the messages into a unified data format, do
various centralized bookkeeping tasks, and then pass those normalized
messages off to workers for further processing.  The system further
distrusts the application messages (and the format conversion libraries)
and so wishes to spawn a worker for each message and let it crash if
nonsense or tomfoolery is involved.

A picture:

Queue of Incoming Messages -> [Pool of Specialized Translators] ->
[Processing Manager] -> [Pool of Workers]

The "natural" approach from a C / OO background is to define a layout for
the unified format, perhaps as an object or struct, and share that across
the specialized translators, the processing manager, and the worker
processes.  And indeed that solves the problem fairly neatly here as well,
except you can't walk the streets of Los Angeles without tripping over a
hobo brought to desperate circumstances who regales you with his woeful
cautionary tale of attempting to perform a live upgrade with a record
definition shared across multiple modules in an hrl file.

After pondering it, it seems like you can choose at most two of { live
upgrades, separation into multiple processes, simple and sensible records }.

Needing to choose live upgrades and separation into multiple processes, one
then starts thinking of bloodcurdling ways to handle the shared record
issue, including:

1.  Treat record formats as immutable and just keep adding them
('mystate1', 'mystate2', ... 'mystate11'...) along with all of the
accompanied handling.
2.  Keep a version field in every record format and start them out being as
big as you might ever want them to get.
3.  Clone the OTP source repository and hack in record upgrade messages.

I'd be interested in hearing if anyone has any better ideas, or how others
have handled this scenario in real world conditions.

F.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140703/f931858b/attachment.htm>