[erlang-questions] json_to_term EEP

[Richard A. O'Keefe]

On 30 Jul 2008, at 10:07 pm, Hynek Vychodil wrote:
[it was rather hard to figure out what was just quoting
  and what was actual response]

> What performance drawback?  What call to binary_to_list()?  Whoever  
> said
> the binary EXISTED in the first place?  The EEP is a proposal for  
> putting
> these conversion functions in the Erlang core, eventually to be
> implemented in C.  So implemented, the alleged performance drawback  
> simply
> does not exist.
>
> All JSON data coming outside Erlang are binary in first state,
> there is no Erlang lists outside Erlang.

True and irrelevant:  the ONLY lists that json_to_term/[1,2] should
construct are the ones in the results.  NO list construction whatsoever
is implied in the handling of strings.  Remember, this is an EEP based
on Joe Armstrong's suggestion that there should be new built in
functions!


> You are overlooking the fact, that there are another scenarios.

ABSOLUTELY NOT!  Remember, options are OPTIONS.

> For example:
>
> 1/ Read and parse JSON {"a":1, "b":2, "c":3} on one erlang node with  
> one set of existing atoms (a,b).
>
> 2/ Store Erlang term to file [{a,1}, {b,2}, {<<"c">>, 3}]

Remember, this does *NOT* happen by default.
For labels to be converted to existing atoms,
the programmer HAS TO ASK FOR IT EXPLICITLY.

You are 100% right that the DEFAULT options should be safe.

However, the real danger here has nothing to do with atoms.
The danger is this:  if you want to store JSON data, you should
store it *as* JSON, not as something else.  (I am counting
compressed JSON as JSON here.)  The EEP points out other ways
in which Erlang-encoded-JSON may vary:  numbers might be
integers or floats, {key,value} pairs may be reordered in
many ways.

Nor does this have anything to do with Erlang specifically.
For ALL languages, if you want to store JSON or transmit it
or in any way cause JSON data known to one node to become
known to another node you should store or transmit it
*AS* (possibly compressed) JSON, not as something else.

Anyone who keeps this straight will not run into trouble.


>
> 3/ In another erlang node with existing atom list {a,c} (for examle  
> in some module you want detect c key of data take from JSON) you  
> load and parse same JSON {"a":1, "b":2, "c":3} and from parser you  
> get [{a,1}, {<<"b">>,2}, {c, 3}]

Remember, {label,existing_atom} is meant for a module that
wants to receive a JSON term and process it, looking for keys that
are mentioned in that module.  If an Erlang process holds a JSON
term in Erlang form and wants to pass it to another node or
another time, it should send it AS JSON.

> 4/ Than you load stored erlang term from file and two think happend.  
> You take [{a,1}, {b,2}, {<<"c">>, 3}] and existing atoms are now  
> {a,b,c}.
>
> 5/ Read and poarse JSON {"a":1, "b":2, "c":3} again and you take [{a, 
> 1}, {b,2}, {c, 3}]
>
> 6/ Great, you have terms [{a,1}, {b,2}, {c, 3}], [{a,1}, {b,2},  
> {<<"c">>, 3}] and [{a,1}, {<<"b">>,2}, {c, 3}] as Erlang term  
> representing same JSON input {"a":1, "b":2, "c":3}. What the hell,  
> there is some totaly wrong, isn't it?

Yes, and what is wrong is seriously incompetent programming.
There are other round trip issues, including the handling of numbers,
and the order of {key,value} pairs.  Recall that the default is
{label,binary}.  So in one node we convert a JSON form to an
Erlang term.  Another node does the same.  One of the nodes then
sends its term to the other, which compares the two terms.
Are they guaranteed to be the same?  Nope.  [hint

FOR ANY PROGRAMMING LANGUAGE AND LIBRARY THE ONLY COMPLETELY
RELIABLE WAY TO TRANSMIT JSON DATA IS *AS* *JSON*.

Got that?

{label,existing_atom} simply is not meant for the use case you
present.
>
>
> Erlang is way how to make things safe and reliable. Converting keys  
> to atoms is not safe and reliable so don't do it, It hurts you!

No, it only hurts stupid people.
Converting keys to existing atoms is perfectly safe for SOME
uses, and there seems to be no good reason to forbid letting
people do that when they are willing to take responsibility
for it being safe.

Expecting JSON forms to convert to identical Erlang terms
at all times and in all places, now THAT is not safe and not
reliable and WILL hurt you.

I could make similar remarks about any language, and about
many formats including XML.