<html>
<head>
<meta http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<meta name=Generator content="Microsoft Word 11 (filtered)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:Arial;}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
p.MsoAutoSig, li.MsoAutoSig, div.MsoAutoSig
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
span.Typed
{font-family:"Courier New";}
span.tty
{font-family:"Courier New";}
span.Name
{font-style:italic;}
span.Variable
{font-family:"Times New Roman";
font-style:italic;}
span.EmailStyle22
{font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal>While an Erlang system has the ability to update its program
on the fly, updating data structures on the fly seems a bit more difficult.
Unless you can upgrade all nodes simultaneously, some nodes will be expecting
the old data structure while others then new. My question therefore, is how to
structure my data? Is there an approach that I am missing that is both upgrade-friendly
and ETS/Mnesia-compatible? Please see the following paragraphs for my analysis
so far.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Suppose we are writing an inventory control application. We
decide to create a record to contain our information about items in our
inventory. Not much to say about items, really, so we’re just going to
hold the item’s name in a record. If something else ever needs to be
tracked regarding these items, we can always upgrade our data, right?</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-record(item,</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> {
name</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> }).</span></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>So we roll out our new inventory system to 3,000 nodes in
our 25 warehouses in 6 different countries, and everything works swimmingly.
For a while.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>However, some time later, our accounting department decides
we need a way to value our inventory, and each item should have a value
associated with it. That way, we can calculate inventory value simply by
multiplying value by quantity at each location. Unfortunately, we cannot now
use our record structure. What to do?</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Well, naïvely, we decide to just modify our item record.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-record(item,</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> {
name</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
value</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> }).</span></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>This new record structure is incompatible with the old item
record structure, so we will also write some code that upgrades our items in
the system to the new structure when we upgrade the system. Unfortunately,
unless our entire worldwide operation is upgraded all at once, any process
using the old structure will crash when it encounters a new-style item, and
vice versa. Simultaneous upgrading all 3,000 nodes is impractical, so
we’ll have to rethink our original decision.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>We could have created the original record structure with
expansion slots available for future use.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-record(item,</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> {
name</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
'X1'</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
'X2'</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
'X3'</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> }).</span></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Now when Accounting wants us to add the value of the item to
the item record, we simply redefine one of the expansion slots.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-record(item,</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> {
name</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
value</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
'X2'</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
'X3'</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> }).</span></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>This will not crash any process, since the size of resulting
tuple is still the same. Unfortunately, we might run out of expansion slots if
we don’t allocate enough of them. The example runs out of slots once
Accounting also gets their cost-basis and GL-class elements added, leaving us
in the same boat as before. We simply delayed the inevitable. We might get
bright and allocate the new slots hierarchically by department, for instance,
so Accounting gets only one slot for all of its information, and we define a
new record for the information in that slot.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-record(item,</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> {
name</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
acctg</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
'X2'</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
'X3'</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> }).</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-record(acctg_item,</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> {
value</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
cost_basis</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
gl_class</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
'X1'</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
'X2'</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
'X3'</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> }).</span></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>However, this approach once again only delays the
inevitable. When Inventory Control and Manufacturing take up the other two
expansion slots, there is no room for Engineering’s data. Plus, we have
multiplied this problem, since it occurs for each of our subrecords, which can
also run out of expansion slots.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Another alternative might be to have only one expansion
slot, which is filled in by the next version of the item record.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-record(item,</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> {
name</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
v2</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> }).</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-record(item_v2,</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> {
value</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
cost_basis</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
gl_class</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
v3</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> }).</span></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Now when we have more elements to add, we create an item_v3
record (with a v4 element to accommodate future expansion), and so on. The
problems with this, however, are that programmers need to know which version of
the record a certain data element is, and that by the time we go through a few
score enhancements and we’re up to version 68, it becomes quite
cumbersome, and is little better than had we used a linked list.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>In fact, a linked list may well be better. Instead of
writing functions with the record syntax, we can use lists.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>item_value([item,
_Name, Value | _])</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> -></span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>
Value</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> .</span></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>To retrieve the value, we only need to know its position in
the list. This approach suffers from a couple of problems: (1) You need to
know the position of each element in the list; (2) This list will be repeated
quite frequently, so when you have 300 attributes your code will be brittle,
repetitive, and difficult to maintain.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Perhaps an alternative approach is to define each record
version independently, instead of additively as we tried earlier.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-record(item1,</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> {
name</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> }).</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-record(item2,</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> {
item</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
value</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
cost_basis</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
gl_class</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> }).</span></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Now in our code, we have versions of each function matching
on the record structure, and a function that handles the no-match case (in case
you’re running v2 code when you receive a v3 record). Once again,
however, we run into a couple of obstacles: (1) We must implement a different
version of each function for each version of the record (this will get tiresome
around version 68); (2) new versions are not backward compatible: a node
running a previous version of the code will not recognize future-versioned data
structures, even though the only fields it needs are those from its own
version.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Let’s borrow a page from object-oriented design principles.
Why not let the item provide its own methods for data access through functions
contained on the structure. We define a record “class” which has
two slots: one for the methods, and one for the data. By doing this, items
carry around their own methods and so it doesn’t really matter what
version of an item something is, so long as the item knows how to use its own
data. First we define some infrastructure.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-module(class).</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-export([invoke/3]).</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-record(class,</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> {
methods</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> ,
data</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> }).</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>invoke(Method_ID,
Object = #class{methods = Methods}, Args)</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> -></span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>
Method = Methods(Method_ID),</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>
Method(Object, Args)</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> .</span></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>To call a method on an object, syntax is simply “<span
style='font-family:Consolas'>invoke(Method_ID, Object, Args)</span>”, such
as</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>X
= item:new ("X"), % Create a new item "X"</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>X_Name
= class:invoke(get_name, X, []), % Returns "X"</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>Y
= class:invoke(set_name, X, ["Y"]). % Changes item name</span></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>This is great for encapsulation! The implementation is
straightforward.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-module(item).</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-export([new/1]).</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-include("class.hrl").</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>-record(item,</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> {
name</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> }).</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> </span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>new(Name)</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> -></span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>
#class{ methods = fun(get_name) -> fun get_name/2</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>
; (set_name) -> fun set_name/2</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>
end</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>
, data = #item{ name = Name }</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>
}</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> .</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> </span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>get_name(#class{data
= #item{name = Name}}, _)</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> -></span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>
Name</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> .</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> </span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>set_name(Object
= #class{data = Item}, [Name])</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> -></span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>
Object#class{data = Item#item{name = Name}}</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'> .</span></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Alas, there is a fly in this ointment, too. While it would
appear that the method functions are being carried around along with the data
(in fact, the item tuple is <span style='font-family:Consolas'>“{class,#Fun<item.0.96410792>,{item,"X"}}</span>”),
those functions are really <i>not</i> carried around from node to node.
Instead, Erlang only carries around references to the functions. This means if
this item shows up on a node where the function does not exist, an error will
occur when a method is invoked.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>The fact that you cannot safely sling functions around with
your data from node to node indicates that perhaps we need a very simple
interface with functions that will never change. Maybe instead of using
records at all, we can use basic OTP library functions to associate item
properties with their values. Sounds kind of like what proplists were designed
for.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>X
= [{name, "X"}], % Create a new item "X"</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>X_Name
= proplists:get_value(name, X), % Returns "X"</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>Y
= [{name, "Y"} | proplists:delete(name, X1)]. % Changes item name</span></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>A similar effect can be had with dicts, with the decision
probably to be made based on performance. (Not only that, but the decision can
be made dynamically at run-time, since there are functions for converting
between the two.)</p>
<p class=MsoNormal> </p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>X
= dict:from_list([{name, "X"}]), % Create a new item "X"</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>X_Name
= dict:fetch(name, X), % Returns "X"</span></p>
<p class=MsoNormal style='margin-left:.5in'><span style='font-family:Consolas'>Y
= dict:store(name, "Y", X). % Changes item name</span></p>
<p class=MsoNormal> </p>
<p class=MsoNormal>This approach has the advantage of being completely
backward-compatible with respect to my code-base. Should a later version of
our inventory application add a property, it will not change the operation of
any previous version. Once again, however, there are problems with this
approach: (1) property values cannot be used for matching in function
definitions; (2) these structures are not easily indexed: ETS and Mnesia
require record data types. While Disadvantage 1 might be easily managed by
performing lookups and conditionals within the function, Disadvantage 2 is probably
intractable.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>To repeat my question, gentle readers, how ought I structure
my data? Is there an approach that I am missing that is both upgrade-friendly
and ETS/Mnesia-compatible?</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Thank-you.</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>Cheers,</p>
<p class=MsoNormal> </p>
<p class=MsoNormal>David</p>
<p class=MsoNormal> </p>
<p class=MsoNormal> </p>
</div>
</body>
</html>