[erlang-questions] Key-value stores and value part

[Jesper Louis Andersen]

On 2/25/12 3:37 AM, Vinayak Pawar wrote:
>
> Question: is it possible to store values in KV store so that I don't 
> have to update the KV store whenever there is change in tuple 
> structure? store values in JSON form?
>
The "Trick" is to version your values. That is, given data

{E1, E2}

you build a tagged variant:

{'version1', {E1, E2}}

Now when you need to alter the structure, you change the version tag:

{'version2', {E1, E2, E3}}

and henceforth, when you load the data from the underlying store, you 
define a function that is akin to a code_change/3 function in a gen_server:

data_change({'version1', {E1, E2}}) ->
   produce_version2_structure_here;
data_change({'version2', {E1, E2, E3}}) ->
   {E1, E2, E3].

This means you now factorize every load of your data through 
data_change/1 and you always store data in the newest version you know 
about. This means you can lazily update your data as you go along 
processing them. You can also force a persist in the new version right 
away so subsequent reads will get the new version.

This is not a new idea. To a certain extent this is what Facebook does 
with their data as they have too much in their store to be able to 
convert all of it. It also means you can do database changes easily: 
First make your erlang code cope with the new format. Then gradually 
change the data store to the new format.

It takes some more work, but this is one way to do it.

-- 
Jesper Louis Andersen
   Erlang Solutions Ltd., Copenhagen, DK