[erlang-questions] Coming Back (maybe improving lists:reverse/1)

[Ivan Carmenates Garcia]

Hi Richard,

Yes I will definably like to build queries like list comprehensions or like
your nice example, but I don't understand so well parse transform because I
didn't find so much documentation about it in Erlang doc, I was trying with
qlc but yet nothing I could archive.
If you could point me in the right direction to get a good reference and
documentation for doing parse transforms, that would be nice because that
will be definitely some well form improvement for my little framework.

Regards,
Ivan (son of Gilberio).


-----Original Message-----
From: Richard A. O'Keefe [mailto:ok@REDACTED] 
Sent: Tuesday, October 13, 2015 1:58 AM
To: Ivan Carmenates Garcia
Cc: erlang-questions@REDACTED
Subject: Re: [erlang-questions] Coming Back (maybe improving
lists:reverse/1)


On 9/10/2015, at 4:40 pm, Ivan Carmenates Garcia <co7eb@REDACTED> wrote:

> Regards,
> 
> Well seems to me that there is no more optimizations for the algorithm 
> (except for yours 'my_reverse_join' see below), I tested yours and 
> mine and they both take exactly the same for the final result the 
> unparsed "string", well mine is 200 millisecond faster in 1 million of 
> iterations, yours take 64XX main 62XX approx. The problem is that 
> using lists:flatten is the hell slow,

The problem is that I suggested *NOT* using lists:flatten/1.

By the way, you really do not *have* to use any particular built in
function.  A few minutes' coding produced this result for flattening a tree
with a million elements:

- Using lists:flatten/1 : 123,939 microseconds
- Using my own code     :  63,134 microseconds

The effect of my code isn't precisely the same as that of lists:flatten/1,
but for a tree of strings that doesn't matter.

my_flatten([A,B,C|Xs], R) ->
    my_flatten(A, my_flatten(B, my_flatten(C, my_flatten(Xs, R))));
my_flatten([A|Xs], R) ->
    my_flatten(A, my_flatten(Xs, R));
my_flatten([], R) ->
    R;
my_flatten(X, R) ->
    [X|R].


> I also tested your algorithm just after the io lists return and it is 
> very faster its took 14XX milliseconds, so when you use the final
> lists:flatten/1 everything goes to crap, sorry about the word. So it 
> is amazing who faster it is and then lists:flatten by its own take the 
> another
> 5 seconds, I did knew that because I read Erlang session about 7 myths 
> off bla bla .. and optimizations stuffs some time ago and they say 
> lists:flatten is slow also I tested it when I was constructing the 
> algorithm first time I did avoid constructing the io lists because of
that.

No, the whole point of io lists is to *NOT* flatten them.

The advice, in short, was to structure the rest of your program so that you
don't NEED to do the flattening.
> 
> I did clear some things like unparse well I need another name I don't 
> like unparse either and parse is wrong, I will come up with something.

Maybe you don't *like* "unparse", but it *is* the standard technical term.
Type "unparsing" into the search box of your browser and read!
> I also have questions for you if you are so kind to answer.
> 
> Regards proplists, well, it is necessary to make lists of options as 
> proplists?

No, of course not.  Even when you encode a list of options as something
*like* a proplist, the structure you want will usually be stricter.  In any
case, what you have here is simply not a list of options.  The elements are
not options, they are descriptions of columns you want to get back from some
data base query.

> yes, proplists:compact/1 and expand/1 for [a] to [{a, true}] and all 
> those stuffs but what I was trying to do is to make it simple for the 
> user, because more tuples are more complicate to write in the case of 
> {users, name, alias}

(a) Remember, one of the essential things about *being* a proplist is that
the consumer is going to IGNORE anything that is not an atom or a pair.  If
you are not going to do that, you are not treating the data *as* a proplist,
and it is "lying to the user" to call it one.

(b) Far from making things simpler for the user, you made it much more
confusing.  There are places where an alias *can* be supplied and places
where it *can't*, and there doesn't seem to be any reason for that.  Well,
it confused the heck out of *me*.

> well that could be right adding more brackets like {users, {name, 
> alias}} because it organizes the thing. But i.e.: I have another 
> algorithm for match specs in which I can do {name | {table, name} | 
> value, op, name | {table, name} | value } also [{..., op, ...}, ...], 
> logic_op, ... so it will be easy for the user to build something like 
> [
>    [{id, '==', 1}, {age, '==', 31}],
>    'or',
>    {username, '==', "john"}
> ]

There are programming languages in which building queries as data structures
is easy.

  In Lisp,
    (make-query '(or (and (= id 1) (= age 31)) (= username "john")))
  is easy because that's *exactly* what an expression would look
  like in Lisp.

  In R,
    make_query(id = 1 && age == 31 || username == "john")
  is easy because R functions evaluate their arguments lazily and
  can get at the abstract syntax trees of those arguments, so again
  the syntax is exactly what an expression normally looks like.

Erlang really isn't one of those languages, but there's a feature of Erlang
that means you can get a lot closer.  You could use a parse transform to
take something like

   make_query(Id == 1 andalso Age == 31 orelse UserName == "John")

which the parser turns into an abstract syntax tree, and then your parse
transform could take make_query(...) and turn that into whatever data
structure you like.  It's hardER in Erlang than it is in Lisp, but it's
still much easier than it is with C or Java.  (And of course LFE would do it
exactly the way Lisp does it.)

If I were a user of your system, I would be somewhere between baffled and
outraged at the claim that it would be *easy* for me to construct a query in
either of the forms you mention.

This is perhaps the ideal time to mention the idea of STAGED computation,
also known as partial evaluation or partial execution.  The function we
started discussing looks like a textbook case of something you probably
shouldn't be doing at run time anyway.