<br><br><div class="gmail_quote">On Thu, Nov 17, 2011 at 4:12 PM, Max Bourinov <span dir="ltr"><<a href="mailto:bourinov@gmail.com" target="_blank">bourinov@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>Hi Joe,</div><div><br></div><div>I also have many term_to_binary calls and as a result many-many binary data chunks stored in DB.</div><div><br></div><div>As I understood to use your technique I need pimped versions of hd, tl and element functions. Can you please provide a little bit more details about it plase?</div>
</blockquote><div><br></div><div>The external format is described in</div><div><br></div><div><a href="http://www.erlang.org/doc/apps/erts/erl_ext_dist.html" target="_blank">http://www.erlang.org/doc/apps/erts/erl_ext_dist.html</a></div>
<div><br></div><div>To test my understanding of this I wrote a simple program that</div><div>reconstructs a term from the external format (enclosed)</div><div><br></div><div>A javascript program that does almost the same thing as my program is here:</div>
<div><br></div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><a href="https://github.com/rtomayko/node-bertrpc/blob/master/src/bert.js">https://github.com/rtomayko/node-bertrpc/blob/master/src/bert.js</a></div>
<div><br></div><div>As you can see decoding a term is easy.</div><div><br></div><div>The tuple {a,b,c} gets encoded as</div><div><br></div><div><<131,104,3,100,0,1,97,100,0,1,98,100,0,1,99>></div><div><br></div>
<div>131 means "external format"</div><div>194,3 means a tuple with 3 elements</div><div>100,1,97 means the atom a and so on</div><div><br></div><div>If you keep a pointer into this structure pointing at the second word (call this P)</div>
<div>and want to implement element(3, P) then you just check that P points to a tuple</div><div>then skip to the third element.</div><div><br></div><div>The external format is not actually designed for rapid random access but it's not too</div>
<div>bad, so this pretty easy. </div><div><br></div><div>Turning the code that I posted here into a set of access functions is easy.</div><div>you only need element(K) - to step into tuples hd and tail</div><div>to step into lists (though n'th would be a good idea)</div>
<div><br></div><div>For this to be efficient you would need to do this as a NIF since the erlang code</div><div>that does the same thing would create some garbage as it executes.</div><div><br></div><div>I haven't done any of this - my goal is to create binaries in Erlang</div>
<div>and decode them in javascript.</div><div> </div><div>Cheers</div><div><br></div><div>/Joe</div><div><br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>Best regards,</div><div>Max</div><br><br>
<br><br><div class="gmail_quote"><div><div></div><div>On Thu, Nov 17, 2011 at 5:52 PM, Joe Armstrong <span dir="ltr"><<a href="mailto:erlang@gmail.com" target="_blank">erlang@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div></div><div>
<div>Here's a programing technique that might be useful which I haven't seen</div><div>described before ...</div><div><br></div><div>I've playing with unpacking binaries produced by term_to_binary(Term) in</div>
<div>other languages. Specifically I do term_to_binary in Erlang creating binary and I send the</div><div>binary to javascript. The javascript code does not by default decode the entire binary, </div><div>but accesses sub-terms through selector functions (you only need element, hd and tl)</div>
<div><br></div><div>This technique seems much nicer than mucking around with JSON </div><div>binary formats are way easier to manipulate than than text formats that need parsing.</div><div><br></div><div>Now of course you can do the same thing in Erlang, you do not have to</div>
<div>do binary_to_term(B) to extract a sub-term, but can traverse the internal structure</div><div>of the external format and pull out exactly what you want and nothing else.</div><div><br></div><div>I often store large terms in files and databases using term_to_binary</div>
<div>and I extract data by first doing binary_to_term and then pattern matching on the result.</div><div><br></div><div>For example if I create a binary with:</div><div><br></div><div> > B = term_to_binary({foo,bar,[a,b]})</div>
<div><br></div><div>And I want to extract the 'b' sub term, I'd normally write</div><div><br></div><div> {_, _, [_,X]} = binary_to_term(B)</div><div><br></div><div>But why bother to unpack? I could just as well write</div>
<div><br></div><div> X = hd(tl(element(3,B))) </div><div><br></div><div>This is not the regular hd/tl/and element but a hacked version that can traverse the external format.</div><div><br></div><div>If the term inside the external format is large and if I only want to extract a few parameters </div>
<div>then this method should be lot faster than actually building a large term, just to throw it away after pattern matching.</div><div>This should be a GC and cache friendly way of doing things.</div><div><br></div><div>
In a similar vein one could think of pattern matching being extended over packed terms.</div><div><br></div><div>If this were so I could write:</div><div><br></div><div> T = {foo,bac,[a,b]}</div><div> B = term_to_binary(T),</div>
<div> match(B).</div><div><br></div><div>match({_,_,[_,X]}) -> X</div><div><br></div><div>Doing so would mean that once we have packed terms using term_to_binary we could leave them</div><div>alone and extract data from them without having to completely unpack them.</div>
<div><br></div><div>This should be very cache friendly - Erlang terms can be scatter all over the place in virtual memory</div><div>but in the external form all the term is kept together in memory</div><div><br></div><div>
This is actually pretty useful - I have a data structure representing a book - somewhere near the beginning there is a title</div><div>the entire book is stored on disk as a term_to_binary encoded blob. Now I have a large numbers of these</div>
<div>representing ebooks. If I want to list all titles I certainly do not want to complete unpack everything, </div><div>I only want to extract the title field and nothing else. ...</div><div><br></div><div>Cheers</div><span><font color="#888888"><div>
<br></div><div>/Joe</div><div><br></div><div><br></div>
</font></span><br></div></div>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br>
</blockquote></div><br>