[erlang-questions] Records / Proplists / JSON / BSON

Fri Jun 14 14:34:38 CEST 2013

That makes sense. My only hesitation there is that I might eventually
need the ability to embed more information than type specifications
allow me to: Leaning on type specs feels like a dead end.

Thomas

On Fri, Jun 14, 2013, at 05:16 AM, Steve Strong wrote:
> Hi Thomas, 
> 
> The first view projects where we did the mapping "by hand" were exactly
> using record_info, and for very simple records it works just fine.
> 
> The problems that we had with that approach is that we don't have any
> type information, so it makes it hard to get some types (atoms,
> timestamps) to roundtrip to JSON and back with full type fidelity. For
> example, you have to convert atoms to strings to pass to the browser but
> on the way back it's impossible to know whether the string you're
> receiving from the browser should stay a string to be turned back into an
> atom.  From a parse transform with type information, all these
> transformations can be done automatically, and nested structures can be
> handled easily.
> 
> If type information was available at runtime, that would be great but
> unfortunately the only way to get that is via the parse_transform
> mechanisms.
> 
> Cheers,
> 
> Steve 
> 
> -- 
> Steve Strong
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> 
> 
> On Friday, 14 June 2013 at 14:06, Thomas Allen wrote:
> 
> > Hi Steve,
> > 
> > I've done some similar things, except rather than using a parse
> > transform to do this (I do not like parse transforms except as a last
> > resort, I find macros a more explicit way to express compile-time
> > things) I simply store my record info in application env when my
> > application starts, and define functions that access these values:
> > db:fields(RecName), etc. Then, I have higher level functions that use
> > this metadata. It's easy to test, and I reason it's pretty fast, with
> > application env being backed by ETS (it's never been a bottleneck).
> > 
> > %% in .hrl
> > -define(DB_FIELDS(Model), {Model, record_info(fields, Model)}).
> > 
> > %% in app:start/0:
> > application:set_env(db, model_fields,
> > [?DB_FIELDS(blog_post),
> > ?DB_FIELDS(contact_message)]),
> > 
> > %% in record utils
> > fields() ->
> > {ok, Fields} = application:get_env(db, model_fields),
> > Fields.
> > 
> > fields(Model) ->
> > proplists:get_value(Model, fields()).
> > 
> > %% etc.
> > 
> > Maybe others here with more experience can correct me, but I think some
> > Erlang developers are too eager to introduce parse transforms to solve
> > simple problems.
> > 
> > Thomas
> > 
> > 
> > On Fri, Jun 14, 2013, at 03:34 AM, Steve Strong wrote:
> > > My plan for the actual JSON encode / decode is for the mapper to produce
> > > a structure that's compatible with jsx
> > > (https://github.com/talentdeficit/jsx) - that's the encoder / decoder
> > > that we currently use. If we go the open-source route, I'd probably aim
> > > to make that pluggable so that folk can use their preferred JSON encoder.
> > > 
> > > Cheers,
> > > 
> > > Steve 
> > > 
> > > -- 
> > > Steve Strong
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > 
> > > 
> > > On Friday, 14 June 2013 at 12:31, Slava Yurin wrote:
> > > 
> > > > See https://github.com/iskra/jsonx. Maybe it decoder/encoder will help you.
> > > > 
> > > > 14.06.2013, 17:13, "Steve Strong" <steve@REDACTED (mailto:steve@REDACTED)>:
> > > > > Hi,
> > > > > 
> > > > > We have built a number of projects recently that have a mongodb backend, an HTML / javascript frontend and Erlang in the middle - pretty standard stuff. One of the things the we end up repeating over and over is mapping data from BSON (the mongo format) to Records (our preferred in-memory format for Erlang) and JSON (to send / receive from the browser). To add to the mix, we also like using proplists in configuration files, so have mappings from those to records as well.
> > > > > 
> > > > > On the last project I finally got sick of doing it by hand, so knocked up a fairly simple parse transform to take the records (with their type specifications) and generate the mapping code, which has resulted in being able to do things like (note - pseudocode only!):
> > > > > 
> > > > > Foo = build_my_record(),
> > > > > mongo:insert(collection, mapper:record_to_bson(Foo)),
> > > > > web_socket:send(Client, mapper:record_to_json(Foo)),
> > > > > 
> > > > > receive
> > > > > {client, Response} ->
> > > > > do_stuff_with(mapper:json_to_record(foo, Response)
> > > > > end
> > > > > 
> > > > > This has worked very well, and handles about 80% of the types we throw at it. The sorts of types that it doesn't deal with are unions and tuples, e.g.
> > > > > 
> > > > > -record(bla, {
> > > > > metadata :: x() | y(),
> > > > > ratio :: {integer(), integer()}
> > > > > }). 
> > > > > 
> > > > > Due to the value the simple version has had to us, I'm about to embark on a re-work that is going to aim to handle (pretty much) any type you can throw at it, and give full two-directional fidelity on the conversions (e.g., you can assert that Data == xxx_to_record(record_to_xxx(Data)) ). 
> > > > > 
> > > > > So, a couple of questions:
> > > > > 
> > > > > 1. Does such a thing already exist? I'm no fan of re-inventing the wheel :)
> > > > > 2. If not, would anyone be interested in it being open-sourced?
> > > > > 3. If 2., then does anyone have opinions on the functionality / API etc?
> > > > > 
> > > > > Interested in any feedback,
> > > > > 
> > > > > Cheers,
> > > > > 
> > > > > Steve
> > > > > -- 
> > > > > Steve Strong
> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > 
> > > > > 
> > > > > 
> > > > > ,
> > > > > _______________________________________________
> > > > > erlang-questions mailing list
> > > > > erlang-questions@REDACTED (mailto:erlang-questions@REDACTED)
> > > > > http://erlang.org/mailman/listinfo/erlang-questions 
> > > > > 
> > > > 
> > > > 
> > > 
> > > 
> > > _______________________________________________
> > > erlang-questions mailing list
> > > erlang-questions@REDACTED (mailto:erlang-questions@REDACTED)
> > > http://erlang.org/mailman/listinfo/erlang-questions
> > > 
> > 
> > 
> > 
> 
> 