App

. ----------------- shell_info.erl ----------------- -module(shell_info).-compile(export_all). get_ver() -> io:format(erlang:system_info(system_version)). However, if I remove the cwd option from the javascript exec function, in the index.html file, I get the following error on the webpage: Erlang shell version: {"init terminating in do_boot",{undef,[{shell_info,get_ver,[],[]},{init,start_it,1,[{file,"init.erl"},{line,1054}]},{init,start_em,1,[{file,"init.erl"},{line,1034}]}]}}, Electron version: 0.33.6 How do I display the expected shell_info:get_ver() output on the webpage? Expected output: C:\Users\eausaig\Downloads\electron-v0.33.6-win32-x64\erlang_studio>erl -noshell -s shell_info get_ver -s erlang haltErlang/OTP 18 [erts-7.1] [64-bit] [smp:4:4] [async-threads:10] C:\Users\eausaig\Downloads\electron-v0.33.6-win32-x64\erlang_studio> Regards, Austin -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Fri Oct 9 21:08:41 2015 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Fri, 9 Oct 2015 14:08:41 -0500 Subject: [erlang-questions] Initializing mnesia In-Reply-To: <1649616.GLHeFUKtGy@burrito> References: <1444345727.2776431@apps.rackspace.com> <1649616.GLHeFUKtGy@burrito> Message-ID: You are troopers all. Much to chew on. Many thanks, Lloyd Sent from my iPad > On Oct 9, 2015, at 3:43 AM, zxq9 wrote: > >> On Friday 09 October 2015 09:39:01 Graham Hay wrote: >> There was another discussion here >> (http://erlang.2086793.n4.nabble.com/Mnesia-create-tables-best-practices-td4710757.html), >> about including an initialised db in a release. > > The non-annoying link: > http://erlang.org/pipermail/erlang-questions/2015-February/083160.html > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From co7eb@REDACTED Sat Oct 10 00:29:13 2015 From: co7eb@REDACTED (Ivan Carmenates Garcia) Date: Fri, 9 Oct 2015 18:29:13 -0400 Subject: [erlang-questions] Coming Back (maybe improving lists:reverse/1) In-Reply-To: <656993BD-5652-4E90-B872-2D2D11876952@cs.otago.ac.nz> References: <041e01d1012f$9220aa40$b661fec0$@frcuba.co.cu> <001001d10152$7e80a080$7b81e180$@frcuba.co.cu> <3DE96CA6-56A4-4996-AEC8-9377749DA20C@cs.otago.ac.nz> <001c01d1015d$c36cf8a0$4a46e9e0$@frcuba.co.cu> <001d01d1015f$24b92790$6e2b76b0$@frcuba.co.cu> <656993BD-5652-4E90-B872-2D2D11876952@cs.otago.ac.nz> Message-ID: <000001d102e1$ee788cf0$cb69a6d0$@frcuba.co.cu> Well, using this function I write instead of 'lists:flatten/1' your algorithm takes 50XX instead of 62XX milliseconds when using 'lists:flatten/1'. But if instead of adding the "," in your iolist generator algorithm we do it here somehow it takes about 42XX milliseconds, I did but I still missing one comma. I get something like this "users.*,name,roles.id,roles.levelusers.name AS alias, comma missing between role.level and user.name. I think we have no control of that in 'concat_io_list/2' function. concat_io_list([], Acc) -> Acc; concat_io_list([[L | _] = List | Rest], Acc) when is_list(L)-> concat_io_list(Rest, Acc ++ join_simple_list(List, [])); concat_io_list([List | Rest], Acc) -> concat_io_list(Rest, Acc ++ List). join_simple_list([], Acc) -> Acc; join_simple_list([[X | _] = Str | Xs], Acc) when is_number(X) -> join_simple_list(Xs, Acc ++ Str); join_simple_list([Str | Xs], Acc) -> join_simple_list(Xs, Acc ++ join_simple_list(Str, [])). Well thanks for all, Best regards, Ivan (son of Gilberio). -----Original Message----- From: Richard A. O'Keefe [mailto:ok@REDACTED] Sent: Thursday, October 8, 2015 10:17 PM To: Ivan Carmenates Garcia Cc: Erlang Questions Mailing List Subject: Re: [erlang-questions] Coming Back (maybe improving lists:reverse/1) On 8/10/2015, at 1:20 pm, Ivan Carmenates Garcia < co7eb@REDACTED> wrote: > For example this is one of the algorithms, I optimize it as well as I could: Start by optimising the documentation. > > Considering here that the order of the fields is very important!. > > %% ------------------------------------------------------------------- > %% @private > %% @doc > %% Parses the specified list of full fields into a string containing > %% all full fields separated my comma. > %% example: > %%


> %%   parse_full_fields([{users, '*'}, name, {roles, [id, level]},

> %%       {users, name, alias}], fun get_postgres_operator/2) ->

> %%       {"users.'*',name,roles.id,roles.level,users.name AS alias",

> %%           [users, roles]}.

> %%

> %% Returns `{[], []}' if called with `[]'. > %% @throws {error, invalid_return_fields_spec, InvalidForm :: any()} > %% @end %% > ------------------------------------------------------------------- Parsing turns a string into structure. Turning structure into a string is the OPPOSITE of parsing. You are not parsing but UNparsing. > -spec parse_return_full_fields(FullFieldsSpecs, Separator, OperatorFinder) -> Str when > FullFieldsSpecs :: proplists:proplist(), > Separator :: string(), > OperatorFinder :: fun(), > Str :: string(). This is seriously confusing. The thing about a proplist, as we've recently discussed in this list, is that it contains - atoms, where 'x' is equivalent to {'x',true} - pairs {Key, Value} - junk, which is completely ignored. While I don't fully understand your example, it's clear that name is NOT equivalent to {name,true} and that {users, name, alias} is NOT ignored as junk. So whatever you have, it certainly isn't a proplist. Finally, your -spec describes a function with three arguments, but the example in your comment has only two arguments. It looks like you are mapping 'atom' -> "atom" {'table', 'field'} -> "table.field" {'table', ['f1',...,'fn']} -> "table.f1" ... "table.fn" {'table', 'field', 'alias'} -> "table.field AS alias" It would seem more logical to have field() = atom() | {atom(),atom()}. % {name,alias} fields() = field() | list(field()). slot() = atom() | {atom(),fields()}. % {table,fields} so that it's obvious how to put an alias inside a list of fields. -module(goo). -export([ tables/1, unparse_return_full_fields/1, unparse_slots/1 ]). unparse_return_full_fields(Slots) -> {lists:flatten(unparse_slots(Slots)), tables(Slots)}. tables([{Table,_}|Slots]) -> Tables = tables(Slots), case lists:member(Table, Tables) of true -> Tables ; false -> [Table|Tables] end; tables([Name|Slots]) when is_atom(Name) -> tables(Slots); tables([]) -> []. unparse_slots([Slot|Slots]) -> [ unparse_slot(Slot) | unparse_remaining_slots(Slots) ]. unparse_remaining_slots([]) -> ""; unparse_remaining_slots(Slots) -> ["," | unparse_slots(Slots)]. unparse_slot({Table,Fields}) -> unparse_fields(Fields, atom_to_list(Table)); unparse_slot(Name) when is_atom(Name) -> atom_to_list(Name). unparse_fields([Field|Fields], Table) -> [ unparse_fields(Field, Table) | unparse_remaining_fields(Fields, Table) ]; unparse_fields({Name,Alias}, Table) -> [Table, ".", atom_to_list(Name), " AS ", atom_to_list(Alias)]; unparse_fields(Name, Table) when is_atom(Name) -> [Table, ".", atom_to_list(Name)]. unparse_remaining_fields([], _) -> ""; unparse_remaining_fields(Fields, Table) -> ["," | unparse_fields(Fields, Table)]. With that definition, we get 1> c(goo). 2> goo:unparse_slots([{users, '*'}, name, {roles, [id, level]}, {users, {name, alias}}]). %% Note difference here.[["users",".","*"], ",","name",",", [["roles",".","id"],",",["roles",".","level"]], ",", ["users",".","name"," AS ","alias"]] This is a tree of strings, not a string. But for many purposes, it's just as good. (It's called an iolist.) Flattening it gives "users.*,name,roles.id,roles.level,users.name AS alias" 3> goo:unparse_return_full_fields([{users, '*'}, name, {roles, [id, level]}, {users, {name, alias}}]). {"users.*,name,roles.id,roles.level,users.name AS alias", [roles,users]} You will note that the functions above don't have any use for reversing a list. And I must repeat that for many purposes, such as sending text to another OS process, there isn't any *need* to flatten the tree of strings. Even in your code, > parse_return_full_fields(FullFieldsSpecs, Separator, OperatorFinder) -> > {ParsedFields, TableNames} = > parse_full_fields2(FullFieldsSpecs, [], [], OperatorFinder), > {string:join(lists:reverse(ParsedFields), Separator), TableNames}. there isn't actually any need to do this: my_reverse_join([X|Xs], Sep) -> my_reverse_join(Xs, X, Sep); my_reverse_join([], _) -> "". my_reverse_join([], Acc, _) -> Acc; my_reverse_join([Y|Ys], Acc, Sep) -> my_reverse_join(Ys, Y++Sep++Acc, Sep). 9> goo:my_reverse_join(["harry","deacon","thom","avery"]). "avery,thom,deacon,harry" We see two ideas here which are likely to be applicable to your code in context. (1) It may be possible to *eliminate* a call to lists:reverse/1 by fusing it with the function the result is being passed to. (2) It may be possible to *eliminate* appends (++) by constructing an iolist (a tree of strings) instead of a string. For example, io:put_chars/2 wants a unicode:chardata(). http://www.erlang.org/doc/man/unicode.html#type-chardata http://www.erlang.org/doc/man/io.html#put_chars-2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From guidao1013@REDACTED Sat Oct 10 11:59:23 2015 From: guidao1013@REDACTED (Dao Gui) Date: Sat, 10 Oct 2015 17:59:23 +0800 Subject: [erlang-questions] =?utf-8?q?why_isn=27t_scope_in_=22begin_end=22?= =?utf-8?b?IGJsb2Nr77yf?= Message-ID: Sorry to my pool English Example: -module(test). -export([test/0]). test() -> begin A = 1 end, A = 2. %% match error %io:format("~p~n", [A]). %normal output The var A is in begin end scope, but it can visited by other scope. Why isn't erlang support this? no this feature? I can't write the next code: -define(OUTPUT_3(A), begin [_,_,B|_]=A, io:format("~p~n", [B]) end). -------------- next part -------------- An HTML attachment was scrubbed... URL: From fernando.benavides@REDACTED Sat Oct 10 14:45:42 2015 From: fernando.benavides@REDACTED (Brujo Benavides) Date: Sat, 10 Oct 2015 09:45:42 -0300 Subject: [erlang-questions] =?utf-8?q?why_isn=27t_scope_in_=22begin_end=22?= =?utf-8?b?IGJsb2Nr77yf?= In-Reply-To: References: Message-ID: You can?t write that but you can write this: output_3(A) -> [_,_,B|_]=A, io:format(?~p~n?, [B]). and then, in your code, instead of ?OUTPUT_3(?something?) you just use output_3(?something?) > On Oct 10, 2015, at 06:59, Dao Gui wrote: > > Sorry to my pool English > Example: > > -module(test). > -export([test/0]). > > test() -> > begin A = 1 end, > A = 2. %% match error > %io:format("~p~n", [A]). %normal output > > The var A is in begin end scope, but it can visited by other scope. > Why isn't erlang support this? > > no this feature? I can't write the next code: > -define(OUTPUT_3(A), begin [_,_,B|_]=A, io:format("~p~n", [B]) end). > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From montuori@REDACTED Sat Oct 10 15:22:33 2015 From: montuori@REDACTED (Kevin Montuori) Date: Sat, 10 Oct 2015 08:22:33 -0500 Subject: [erlang-questions] =?utf-8?q?why_isn=27t_scope_in_=22begin_end=22?= =?utf-8?b?IGJsb2Nr77yf?= In-Reply-To: (Dao Gui's message of "Sat, 10 Oct 2015 17:59:23 +0800") References: Message-ID: >>>>> "dg" == Dao Gui writes: dg> The var A is in begin end scope, but it can visited by other dg> scope. Why isn't erlang support this? Because the documentation says so: The scope for a variable is its function clause. Variables bound in a branch of an if, case, or receive expression must be bound in all branches to have a value outside the expression. Otherwise they are regarded as 'unsafe' outside the expression. http://www.erlang.org/doc/reference_manual/expressions.html#id79254 dg> -define(OUTPUT_3(A), begin [_,_,B|_]=A, io:format("~p~n", [B]) end). -define(OUTPUT_3(A), io:format("~p~n", lists:nth(3, A))). k. -- Kevin Montuori montuori@REDACTED From aschultz@REDACTED Sat Oct 10 15:43:24 2015 From: aschultz@REDACTED (Andreas Schultz) Date: Sat, 10 Oct 2015 15:43:24 +0200 (CEST) Subject: [erlang-questions] RADIUS decode/encode In-Reply-To: References: <5617D5E6.8030608@ipv6.dp.ua> Message-ID: <1604170422.432635.1444484604512.JavaMail.zimbra@tpip.net> ----- Original Message ----- > From: "Vance Shipley" > To: "Artem Teslenko" > Cc: "Questions erlang-questions" > Sent: Friday, October 9, 2015 5:09:05 PM > Subject: Re: [erlang-questions] RADIUS decode/encode > On Fri, Oct 9, 2015 at 8:27 PM, Artem Teslenko wrote: >> Look at https://github.com/ates/radius project > > ... or: https://github.com/vances/radierl ... or : https://github.com/travelping/eradius Andreas From aschultz@REDACTED Sat Oct 10 15:47:11 2015 From: aschultz@REDACTED (Andreas Schultz) Date: Sat, 10 Oct 2015 15:47:11 +0200 (CEST) Subject: [erlang-questions] RADIUS decode/encode In-Reply-To: References: Message-ID: <2127303626.432646.1444484831044.JavaMail.zimbra@tpip.net> ----- Original Message ----- > From: "Kirill Ratkin" > To: erlang-questions@REDACTED > Sent: Friday, October 9, 2015 3:29:15 PM > Subject: [erlang-questions] RADIUS decode/encode > Hi guys, > Who played with RADIUS? > I'm trying to make test aplication which decode request and encode response > (Accept). > Here is code: > handle_info({udp, Socket, IP, Port, Packet}, State) -> > io:format("Packet is ~p~n", [hexlify(Packet)]), > <> = > Packet, > io:format("Packet is ~p,~p,~p,~p,~p~n", [ > Code, > Identifier, > Length, > Authenticator, > hexlify(Attributes) > ]), > <> = Attributes, > io:format("AVP: ~p, ~p, ~p~n", [Len, Type, Body]), > AVPCode = 18, > AVPMessage = <<"You dick">>, > AVPSize = byte_size(AVPMessage) + 2, > AVPResponse = <>, > RCode = 2, % calculated base on logic, accept is now for test > RLength = byte_size(AVPResponse) + 20, > Secret = <<"secret">>, > RAuthenticator = erlang:md5(<>), That should be: RAuthenticator = erlang:md5(<>), The Response-Authenticator is calculate over the response packet, with the Authenticator field set to the Request-Authenticator. Andreas > Response = < AVPResponse/binary>>, > gen_udp:send(Socket, IP, Port, Response), > inet:setopts(Socket, [{active, once}]), > {noreply, State}; > It works but ... 'radclient' says Response Authenticator is not correctly > calculated. > This is its output: > $ echo "User-Name = test" | radclient -x localhost:1812 auth secret > Sending Access-Request Id 68 from 0.0.0.0:38654 to 127.0.0.1:1812 > User-Name = 'test' > Received Access-Accept Id 68 from 127.0.0.1:1812 to 127.0.0.1:38654 length 30 > (0) Reply verification failed: Received Access-Accept packet from home server > 127.0.0.1 port 1812 with invalid Response Authenticator! (Shared secret is > incorrect.) > RFC says: > Response Authenticator > The value of the Authenticator field in Access-Accept, Access- > Reject, and Access-Challenge packets is called the Response > Authenticator, and contains a one-way MD5 hash calculated over > a stream of octets consisting of: the RADIUS packet, beginning > with the Code field, including the Identifier, the Length, the > Request Authenticator field from the Access-Request packet, and > the response Attributes, followed by the shared secret. That > is, ResponseAuth = > MD5(Code+ID+Length+RequestAuth+Attributes+Secret) where + > denotes concatenation. > It seems I do how RFC recommends but ... > I don't see mistake :(. > Please help if you see my fault. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From thomas@REDACTED Sat Oct 10 17:46:20 2015 From: thomas@REDACTED (Thomas Gebert) Date: Sat, 10 Oct 2015 11:46:20 -0400 Subject: [erlang-questions] Why should you ever use atoms? Message-ID: I know this is probably kind of a newbie question, but I figured this would be the place to ask it: if atoms aren't garbage collected, why should I use them? For example, it's a common pattern to have something like: myFunction({user, "tombert","eats pizza"}) -> %% do something When I could easily do something like: myFunction({"user", "tombert", "eats pizza"}) -> %% do something ---- I could be way off here, but wouldn't the string be garbage collected? Is there a benefit to atoms that I'm missing? -Tombert. From pierrefenoll@REDACTED Sat Oct 10 17:59:54 2015 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Sat, 10 Oct 2015 08:59:54 -0700 Subject: [erlang-questions] Why should you ever use atoms? In-Reply-To: References: Message-ID: <3245F9E3-07F8-4B20-9FD2-75D28F657284@gmail.com> Atom comparison is O(1) (and just one word). String comparison is O(n) > On 10 Oct 2015, at 08:46, Thomas Gebert wrote: > > I know this is probably kind of a newbie question, but I figured this would be the place to ask it: if atoms aren't garbage collected, why should I use them? For example, it's a common pattern to have something like: > > myFunction({user, "tombert","eats pizza"}) -> %% do something > > When I could easily do something like: > > myFunction({"user", "tombert", "eats pizza"}) -> %% do something > > ---- > > I could be way off here, but wouldn't the string be garbage collected? Is there a benefit to atoms that I'm missing? > > -Tombert. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From olopierpa@REDACTED Sat Oct 10 18:00:51 2015 From: olopierpa@REDACTED (Pierpaolo Bernardi) Date: Sat, 10 Oct 2015 18:00:51 +0200 Subject: [erlang-questions] Why should you ever use atoms? In-Reply-To: References: Message-ID: On Sat, Oct 10, 2015 at 5:46 PM, Thomas Gebert wrote: > I know this is probably kind of a newbie question, but I figured this would > be the place to ask it: if atoms aren't garbage collected, why should I use > them? For example, it's a common pattern to have something like: > > myFunction({user, "tombert","eats pizza"}) -> %% do something > > When I could easily do something like: > > myFunction({"user", "tombert", "eats pizza"}) -> %% do something > > ---- > > I could be way off here, but wouldn't the string be garbage collected? Is > there a benefit to atoms that I'm missing? Yes. Atoms can be checked for identity with one machine instruction, while strings need to be scanned. From zxq9@REDACTED Sat Oct 10 19:17:11 2015 From: zxq9@REDACTED (zxq9) Date: Sun, 11 Oct 2015 02:17:11 +0900 Subject: [erlang-questions] Why should you ever use atoms? In-Reply-To: References: Message-ID: <2211277.niYdm7qWJz@changa> On 2015?10?10? ??? 11:46:20 Thomas Gebert wrote: > I know this is probably kind of a newbie question, but I figured this > would be the place to ask it: if atoms aren't garbage collected, why > should I use them? ...snip... > I could be way off here, but wouldn't the string be garbage collected? > Is there a benefit to atoms that I'm missing? [Spoiler: This is a non-answer, in the sense that I urge you to discover the answer yourself through experience with the environment.] This reminds me of me about 7 years ago, before I had been able to give myself any time with languages that featured atoms. There are *many* good reasons for having atoms (it would be interesting if ROK shows up and enumerates a few of them -- pun noted, to include probable demolition of the pun). If you have ever written anything in C, used databases with serious features, or messed around with very basic hash-like data structures you will instantly appreciate the utility of enums and enum-like data. Even if you haven't, though, just write a few programs in Erlang and you will discover on your own why atoms are useful. At first you must simply have faith that they are useful enough that people we assume to have deeper knowledge than ourselves saw fit to include them. Until we find a pattern of cases that invalidate that assumption we should just go with it. Life is too short to start second guessing people who have produced demonstrably useful things we wish to use until we actually know better (as opposed to pretending that we do). Anyway, without getting even more preachy than this response already is, just go with the flow for now, use atoms where they seem appropriate, and your sense of "appropriate" and "common use case" will mature. At that point you will suddenly realize why this is useful. For me, at least, this sort of experience has more impact than an academic explanation does initially -- after a bit of time getting used to the mentality, though, the academic explanations become very obviously compelling (and suddenly you will appreciate two things: weird features *and* academic explanations). At the very least, who wants to write `<<"something">>` when they can write `something` instead? There is a very clear semantic case for having a data type that "is its own value". -Craig From co7eb@REDACTED Sat Oct 10 20:38:57 2015 From: co7eb@REDACTED (Ivan Carmenates Garcia) Date: Sat, 10 Oct 2015 14:38:57 -0400 Subject: [erlang-questions] What about this? Message-ID: <000001d1038a$edaf3db0$c90db910$@frcuba.co.cu> Hi fellows, Here I show some examples of how to work with the little framework for cowboy that I am making, I will be glad of hear some opinions from the community in order to help improve it. I must say that I will release it to the community as soon as it is ready for shown (the first version). (This will be about database stuffs) First you configure the backends in a config file app.config. (you can configure as many backends as you want, even in runtime you can update and install database backends). I'll show you how. %% Database backends configuration. {database_manager, [ {main_backend, [ {backend, postgres_backend}, {server, "localhost"}, {username, "postgres"}, {password, "server"}, {database, "my_main_db"}, %% max amount of database connections in the connection pool. {max_reusable_connections, 10}, % 10 connections. %% max time it will wait for an available connection. {wait_for_reusable_connection_timeout, 5000} % 5 seconds. ]}, {test_backend, [ {backend, postgres_backend}, {server, "localhost"}, {username, "postgres"}, {password, "server"}, {database, "test_db"}, %% max amount of database connections in the connection pool. {max_reusable_connections, 50}, % 10 connections. %% max time it will wait for an available connection. {wait_for_reusable_connection_timeout, 3000} % 3 seconds. ]} ]} To install a backend at runtime use database_manager:install_backend/2, passing the backend name and config, also has update_backend/2. Later you will use this in a simple way: database_manager:connection_session(fun(DBSession)-> . end, [{backend, test_backend}]). 'connection_session/1' give us an available connection from the connection pool. Each configured backend has its own connection pool, so we can use they as we like having the possibility of use multiple backend in runtime and at the same time. Let's see some examples of the current database functionalities the framework has: database_manager:connection_session(fun(DBSession) -> database_manager:insert(DBSession, {users, #{username => "John", age => 31}}) end). If no backend option is defined in 'connection_session', 'main_backend' will be used. database_manager:connection_session_transaction(fun(DBSession) -> {ok, #{id := UserId}} = database_manager:insert(DBSession, {users, #{username => "John", age => 31}}, [{return_fields, [id]}]), {ok, 1} = database_manager:insert(DBSession, {roles, #{user_id => UserId, role_level => 1}}) end). We also have 'database_manager:transaction/1' for use with 'connection_session' as we like. It has more functions like update, delete, find. i.e.: database_manager:connection_session(fun(DBSession) -> {ok, DataMap} = database_manager:find(DBSession, users, [{username, '==', "John"}]) end). We can use 'return_fields' option also to select just the fields we want. That was the low level API, it also have a high level API for model validation and of course you can mix between them as you wish. Example: To use the high level model API you need to create a module file that implement a behavior of 'model'. The nice thing here is that you can use validation functions by context, using tags. -module(users_model). -behavior(model). -export([ validation_tests/1, after_validate/1]). validation_tests(ModelDataMap) -> {username := Username, online := Online, password := Password} = ModelDataMap, [ %% tag for common validation tests. {common, [ %% username is required {fun() -> size(Username) =/= 0 end, "username cannot be empty"}]}, {new, [ %% password cannot be empty {fun() -> size(Password) =/= 0 end, "password cannot be empty"}]}, {update, [ %% online must be true or false {fun() -> (Online == true) or (Online == false) end, "online is not true or false"}]} ]. %% optional after_validate(ModelDataMap) -> ok. %% optional id_field_name() -> id. Then you can write {ok, ModelInfo} = model_manager:new_model(users, #{ username => "John", password => "server213*-+", password_old => password}, [common, new]), {ok, UserId} = model_manager:store(ModelInfo, [return_id]), {ok, ModelInfo2} = model_manager:update_model(roles, UserId, #{role_level => 1}, [common, update]), ok = model_manager:store_model(ModelInfo2). Note three things here: - first you can use tags to separate validation tests by context, for example, here I use 'common' to use that validation function for both new and update, and one for each of them. You can choose the name you want. But 'always' is a special name that is used when you don't specify any tag in 'new_model' or 'update_model' functions, them 'always' tag will be used. - second you can use 'return_id' option to return just the id of the inserted record but you can also use 'return_fields' to return any other field. In other to make the framework knows which is the id for 'return_id' option we must use 'id' for the name in the database or specify which is the name of the id in the model module using 'id_field_name/0' function. - third we can do stuffs like this #{username => "John", password => "server213*-+", password_old => password}, meaning that 'passworld_old' will take the value of 'password'. There are many other options and functions in the API, for example you can mix between low level API and High level API using model_manager:store_session(fun(DBSession) -> . end). You of course can use all functionalities form the low level API in the High one, like backend options and many other. Using 'model_manager:store_session/1/2' when using multiple 'store_model' in the same function is recommended because with it you hold the same connection for all 'store_model' and low level API operations. So this will be ok: model_manager:store_session_transaction(fun(_) -> {ok, ModelInfo} = model_manager:new_model(users, #{ username => "John", password => "server213*-+", password_old => password}, [common, new]), {ok, UserId} = model_manager:store(ModelInfo, [return_id]), {ok, ModelInfo2} = model_manager:update_model(roles, UserId, #{role_level => 1}, [common, update]), ok = model_manager:store_model(ModelInfo2) end, [{backend, test_backend}]). If any of the functions fail or the pattern matching fails the hold transaction will rollback. Cheers, Ivan (son of Gilberio). -------------- next part -------------- An HTML attachment was scrubbed... URL: From co7eb@REDACTED Sat Oct 10 20:49:25 2015 From: co7eb@REDACTED (Ivan Carmenates Garcia) Date: Sat, 10 Oct 2015 14:49:25 -0400 Subject: [erlang-questions] Why should you ever use atoms? In-Reply-To: References: Message-ID: <000501d1038c$64295470$2c7bfd50$@frcuba.co.cu> Regards, Even if you use atoms in many functions for pattern matching you won't be using as many as for break down the system or consume the default maximum of atoms allowed by the system. So use them freely in that way, warning a thing you definably must never do is to use it as a conversion from a data that comes from outside of Erlang or other context, for example never do this: my_data_to_atom(StringData) when is_list(StringData) -> lists_to_atom(StringData). This is a simple and pointless example but what I am trying to explain is that StringData is an information that can vary in many ways and one you make a new atom it will be never garbage collected so... unless you have control of the StringData and know that it will be only a few constant values of course. Cheers, Ivan (son of Gilberio). -----Original Message----- From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Thomas Gebert Sent: Saturday, October 10, 2015 11:46 AM To: erlang-questions@REDACTED Subject: [erlang-questions] Why should you ever use atoms? I know this is probably kind of a newbie question, but I figured this would be the place to ask it: if atoms aren't garbage collected, why should I use them? For example, it's a common pattern to have something like: myFunction({user, "tombert","eats pizza"}) -> %% do something When I could easily do something like: myFunction({"user", "tombert", "eats pizza"}) -> %% do something ---- I could be way off here, but wouldn't the string be garbage collected? Is there a benefit to atoms that I'm missing? -Tombert. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From jesper.louis.andersen@REDACTED Sat Oct 10 22:14:22 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 10 Oct 2015 22:14:22 +0200 Subject: [erlang-questions] Why should you ever use atoms? In-Reply-To: References: Message-ID: On Sat, Oct 10, 2015 at 5:46 PM, Thomas Gebert wrote: > I know this is probably kind of a newbie question, but I figured this > would be the place to ask it: if atoms aren't garbage collected, why should > I use them? 1) Pierre notes: runtime characteristics are different. This in itself is an important reason. Having a way to introduce static names in code with O(1) comparison helps in defining structure. This structure can then be used in pattern matches to quickly discriminate between cases. If you define {user, "tombert", "easts pizza"}, you are using a common pattern where the first tuple-argument encodes what the remainder of the tuple contains. It has some reminiscence of algebraic datatypes in languages which support those, and is somewhat Erlang's ways of getting access to these. >From a programmers perspective: atoms: used for statically declared words by the programmer and are avoided for dynamic generation and input controlled by an "enemy". binaries: used in places where space saving is paramount, and for data which contain truly binary information. strings: textual strings and the rest of the systems string-like objects are represented as lists of code-points. Apart from this, there are cases where it is nice to signify "namespace difference" by using atoms. e.g., in the term {user, "tombert", "eats pizza"}, A reader immediately understands that 'user' is something the programmer has supplied, whereas "tombert" and "eats pizza" are probably sattelite data entered by a user at some point. Had we writte, e.g., "user", then you somewhat tell the reader that "user" is a field which might get changed to "anything", whereas the atom 'user' signifies that there might be other kinds of terms used in this place, but there is only a finite amount of those. Of course, YMMV, and some people have other views as to how and where to use the different classes. There are also people who thinks we should GC atoms such that we can reuse them freely. The major reason this has not happened yet, is that one has to come up with a good scheme that doesn't slow the interpreter down too much, or breaks the soft-realtime properties of the system. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From frederic.bonfanti@REDACTED Sun Oct 11 07:46:01 2015 From: frederic.bonfanti@REDACTED (Frederic BONFANTI) Date: Sun, 11 Oct 2015 00:46:01 -0500 Subject: [erlang-questions] Erlang executable that reads command line arguments Message-ID: <0AF84669-C6E1-43B9-B7FF-78A2A969FB0B@gmail.com> Hi guys, given a simple Erlang code that consists in one file, let?s say test123.erl , I?d like to 1. use a Makefile to compile test123.erl into test.beam and then generate a distributable version of test123 (executable) 2. figure-out how to parse the command line arguments once this test command is called from regular shell, for example: test123 -A -x 555 If there are straightforward examples available, that will do. Thanks in advance From sergej.jurecko@REDACTED Sun Oct 11 08:52:28 2015 From: sergej.jurecko@REDACTED (Sergej =?UTF-8?B?SnVyZcSNa28=?=) Date: Sun, 11 Oct 2015 08:52:28 +0200 Subject: [erlang-questions] Erlang executable that reads command line arguments In-Reply-To: <0AF84669-C6E1-43B9-B7FF-78A2A969FB0B@gmail.com> References: <0AF84669-C6E1-43B9-B7FF-78A2A969FB0B@gmail.com> Message-ID: <887870C5-9650-4D47-B1E7-8670B196746E@gmail.com> Use an escript? http://www.erlang.org/doc/man/escript.html Sergej On 11/10/15 07:46, "Frederic BONFANTI" wrote: >Hi guys, > >given a simple Erlang code that consists in one file, let?s say test123.erl , I?d like to > >1. use a Makefile to compile test123.erl into test.beam and then generate a distributable version of test123 (executable) > >2. figure-out how to parse the command line arguments once this test command is called from regular shell, for example: > > test123 -A -x 555 > >If there are straightforward examples available, that will do. > >Thanks in advance > >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions From eric.pailleau@REDACTED Sun Oct 11 10:07:52 2015 From: eric.pailleau@REDACTED (=?ISO-8859-1?Q?=C9ric_Pailleau?=) Date: Sun, 11 Oct 2015 10:07:52 +0200 Subject: [erlang-questions] Erlang executable that reads command line arguments In-Reply-To: <0AF84669-C6E1-43B9-B7FF-78A2A969FB0B@gmail.com> Message-ID: Hello, And Gnu getopt like module : https://github.com/jcomellas/getopt Regards Le?11 oct. 2015 07:46, Frederic BONFANTI a ?crit?: > > Hi guys, > > given a simple Erlang code that consists in one file, let?s say test123.erl , I?d like to > > 1. use a Makefile to compile test123.erl into test.beam and then generate a distributable version of test123 (executable) > > 2. figure-out how to parse the command line arguments once this test command is called from regular shell, for example: > > test123 -A -x 555 > > If there are straightforward examples available, that will do. > > Thanks in advance > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From frederic.bonfanti@REDACTED Sun Oct 11 09:42:45 2015 From: frederic.bonfanti@REDACTED (Frederic BONFANTI) Date: Sun, 11 Oct 2015 02:42:45 -0500 Subject: [erlang-questions] Erlang executable that reads command line arguments In-Reply-To: <887870C5-9650-4D47-B1E7-8670B196746E@gmail.com> References: <0AF84669-C6E1-43B9-B7FF-78A2A969FB0B@gmail.com> <887870C5-9650-4D47-B1E7-8670B196746E@gmail.com> Message-ID: The first time I launched it, it requested me to accept socket monitoring (can?t remember exactly) then I couldn?t reproduce that issue. Besides this, it should do Thanks > On Oct 11, 2015, at 1:52 AM, Sergej Jure?ko wrote: > > Use an escript? > > http://www.erlang.org/doc/man/escript.html > > > Sergej > > > > > On 11/10/15 07:46, "Frederic BONFANTI" wrote: > >> Hi guys, >> >> given a simple Erlang code that consists in one file, let?s say test123.erl , I?d like to >> >> 1. use a Makefile to compile test123.erl into test.beam and then generate a distributable version of test123 (executable) >> >> 2. figure-out how to parse the command line arguments once this test command is called from regular shell, for example: >> >> test123 -A -x 555 >> >> If there are straightforward examples available, that will do. >> >> Thanks in advance >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From dmkolesnikov@REDACTED Sun Oct 11 12:40:40 2015 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Sun, 11 Oct 2015 13:40:40 +0300 Subject: [erlang-questions] Erlang executable that reads command line arguments In-Reply-To: <0AF84669-C6E1-43B9-B7FF-78A2A969FB0B@gmail.com> References: <0AF84669-C6E1-43B9-B7FF-78A2A969FB0B@gmail.com> Message-ID: <9FAC6E44-8C3B-418E-94BF-82CB9FBEB9FA@gmail.com> And if you are using rebar and you have deps to other projects you might add following lines to rebar.config {escript_incl_apps, [ getopt ]}. {escript_emu_args, "%%! +K true +P 10000000\n"}. Sad story, I've not figure out how to add erlang runtime to same package. Anyone, who receives you script needs to have one. Best Regards, Dmitry >-|-|-(*> > On 11 Oct 2015, at 08:46, Frederic BONFANTI wrote: > > Hi guys, > > given a simple Erlang code that consists in one file, let?s say test123.erl , I?d like to > > 1. use a Makefile to compile test123.erl into test.beam and then generate a distributable version of test123 (executable) > > 2. figure-out how to parse the command line arguments once this test command is called from regular shell, for example: > > test123 -A -x 555 > > If there are straightforward examples available, that will do. > > Thanks in advance > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From vimal7370@REDACTED Sun Oct 11 16:18:50 2015 From: vimal7370@REDACTED (Vimal Kumar) Date: Sun, 11 Oct 2015 19:48:50 +0530 Subject: [erlang-questions] If you make WhatsApp today... Message-ID: Hi, Imagine that WhatsApp does not exist today. If you are asked to write WhatsApp in Erlang and scale it like how they successfully did, will you still be opting for Mnesia, or something else like Riak? Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Sun Oct 11 19:48:41 2015 From: erlang@REDACTED (Joe Armstrong) Date: Sun, 11 Oct 2015 19:48:41 +0200 Subject: [erlang-questions] If you make WhatsApp today... In-Reply-To: References: Message-ID: On Sun, Oct 11, 2015 at 4:18 PM, Vimal Kumar wrote: > Hi, > > Imagine that WhatsApp does not exist today. If you are asked to write > WhatsApp in Erlang and scale it like how they successfully did, will you > still be opting for Mnesia, or something else like Riak? Neither - I'd define an API that I'd use and implement it with any appropriate database. Then *if* the app took off and I ran into performance problems I'd tweak whatever law behind the API. If the application never flew I'd be saved the trouble of making an efficient back-end. To implement the API I'd choose whatever I was most comfortable with. My goal would be to get something up and running as soon as possible and get users - and not worry about implementation details. As the wise man said "Premature optimisation is the root of all evil". I'd probably use the file system, then ets or dets and "roll my own" first. Data bases using *only* the file system and simple locking can get you a lot of milage before you need to optimize :-) /Joe > > Thank you! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From erlang@REDACTED Sun Oct 11 20:01:08 2015 From: erlang@REDACTED (Joe Armstrong) Date: Sun, 11 Oct 2015 20:01:08 +0200 Subject: [erlang-questions] Why should you ever use atoms? In-Reply-To: References: Message-ID: Using atoms is fine. The dangerous primitive is list_to_atom(X) if your program *never* calls list_to_atom then don't worry. If you call list_to_atom a lot then worry. Note "a lot" means tens of millions of times on a machine with GBytes if memory. There is your friend list_to_existing_atom(X) which exits if X represents a non existent atom. You can, or course, use strings or binaries instead of atoms but your programs will run slower and you'll put more pressure on the garbage collector. Rule of thumb: don't worry, use atoms until you run into problems. Write your code in such a way that swapping from atoms to binaries is no big deal. This is actually about structuring your code - if you use atoms and get problems and have to edit for a hour or so to fix the problem then this is no big deal - if this involves weeks of work then you have a bad design. Make it work - them make it fast/stable/whatever /Joe On Sat, Oct 10, 2015 at 5:46 PM, Thomas Gebert wrote: > I know this is probably kind of a newbie question, but I figured this would > be the place to ask it: if atoms aren't garbage collected, why should I use > them? For example, it's a common pattern to have something like: > > myFunction({user, "tombert","eats pizza"}) -> %% do something > > When I could easily do something like: > > myFunction({"user", "tombert", "eats pizza"}) -> %% do something > > ---- > > I could be way off here, but wouldn't the string be garbage collected? Is > there a benefit to atoms that I'm missing? > > -Tombert. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From gianfranco.alongi@REDACTED Sun Oct 11 20:50:16 2015 From: gianfranco.alongi@REDACTED (Gianfranco Alongi) Date: Sun, 11 Oct 2015 20:50:16 +0200 Subject: [erlang-questions] If you make WhatsApp today... In-Reply-To: References:

Message-ID: This is the correct answer. On Sunday, 11 October 2015, Joe Armstrong wrote: > On Sun, Oct 11, 2015 at 4:18 PM, Vimal Kumar > wrote: > > Hi, > > > > Imagine that WhatsApp does not exist today. If you are asked to write > > WhatsApp in Erlang and scale it like how they successfully did, will you > > still be opting for Mnesia, or something else like Riak? > > Neither - I'd define an API that I'd use and implement it with any > appropriate database. Then *if* the app took off and I ran into performance > problems I'd tweak whatever law behind the API. If the application never > flew I'd be saved the trouble of making an efficient back-end. > > To implement the API I'd choose whatever I was most comfortable with. > > My goal would be to get something up and running as soon as possible > and get users - and not worry about implementation details. > > As the wise man said "Premature optimisation is the root of all evil". > > I'd probably use the file system, then ets or dets and "roll my own" first. > > Data bases using *only* the file system and simple locking can get you > a lot of milage before you need to optimize :-) > > > /Joe > > > > > Thank you! > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Sent from Gmail Mobile -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Sun Oct 11 23:27:54 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 12 Oct 2015 00:27:54 +0300 Subject: [erlang-questions] Why should you ever use atoms? In-Reply-To: References:

Message-ID: You can think about atoms as of a predefined words of your domain logic. Each domain description has a limited amount of words that describe it, so number of atoms is also limited by design. In a properly designed program you should never meet dynamic creation by list_to_atom, only if you know that it is a string representation of already described atom. In other words, atoms are always written by hands by some human as a part of described functionality. Maybe this will help you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Sun Oct 11 23:44:54 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sun, 11 Oct 2015 23:44:54 +0200 Subject: [erlang-questions] If you make WhatsApp today... In-Reply-To: References:

Message-ID: On Sun, Oct 11, 2015 at 7:48 PM, Joe Armstrong wrote: > I'd probably use the file system, then ets or dets and "roll my own" first. To much extent, this is what WhatsApp is doing according to what I've been able to dissect out of their talks. They use mnesia for storing meta-data, but they use an UFS2 file system on FreeBSD to store the majority of data and just pick it off of the disk whenever they need it. The other trick they use is to use the clients to store the majority of data and only use the servers as a temporary transient store until the client comes back up and requests its missing data. Were I to start, I'd definitely do the API modularity split as well, but I'd probably pick postgresql as a backing store over mnesia, perhaps with mnesia as an in-memory cache for active users. The assumption is user data is not changing that often, and postgresql have the added value of being a system I *know* how to operate and handle. There is much to be said for picking technology which is tried and proven and you know how to operate. A possible switch is Riak for the UFS2 store, but it does require the ability to "append" to an object in Riak. Otherwise it is kind of moot to load data, add a couple of bytes and store the object back again. Perhaps this can be done by "chaining" documents in Riak with content addressing. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Oct 12 01:02:12 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 12 Oct 2015 12:02:12 +1300 Subject: [erlang-questions] =?utf-8?q?why_isn=27t_scope_in_=22begin_end=22?= =?utf-8?b?IGJsb2Nr77yf?= In-Reply-To: References: Message-ID: <0D41BBB1-2CF6-4048-906A-3EE1533D4DB2@cs.otago.ac.nz> On 10/10/2015, at 10:59 pm, Dao Gui wrote: > Sorry to my pool English > Example: > > -module(test). > -export([test/0]). > > test() -> > begin A = 1 end, > A = 2. %% match error > %io:format("~p~n", [A]). %normal output > > The var A is in begin end scope, but it can visited by other scope. > Why isn't erlang support this? Because it doesn't. It never has. It is so documented. There is no such thing as "begin end scope" in Erlang. You might as well ask why %start...%finish in Atlas Autocode and IMP aren't scopes; why do;...end; in PL/I and XPL and PL/M aren't scopes; why begin...end is not a scope in Pascal. | without this feature? I can't write the next code: > -define(OUTPUT_3(A), begin [_,_,B|_]=A, io:format("~p~n", [B]) end). You can get the *effect* you want: -define(OUTPUT_3(A), fun ([_,_,B|_]) -> io:format("~p~n", [B]) end(A)). In general, you can express let Pat1 = Exp1 Pat2 = Exp2 ... Patn = Expn in Body end as fun (Pat1, Pat2, ..., Patn) -> Body end(Exp1, Exp2, ..., Expn) From ok@REDACTED Mon Oct 12 01:22:16 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 12 Oct 2015 12:22:16 +1300 Subject: [erlang-questions] Why should you ever use atoms? In-Reply-To: <000501d1038c$64295470$2c7bfd50$@frcuba.co.cu> References: <000501d1038c$64295470$2c7bfd50$@frcuba.co.cu> Message-ID: <037DD6F4-BCF7-4522-9F6D-CB1E88157357@cs.otago.ac.nz> For what it's worth, SWI Prolog now has a lock-free atom symbol table and has garbage collected the atom table for a very long time. It is long past time for Erlang's fixed size bottlenecking atom table to be replaced by something less troublesome. There is an EEP proposing an approach to lots-of-atoms based on the LOGIX implementation of Flat Concurrent Prolog. From what I've heard, the SWI Prolog lock-free atom table sounds superior. For what it's worth, benchmarking is always able to surprise us. I benchmarked comparing atoms, comparing strings, and comparing binaries. Atoms: 1.0 Strings: 1.3 Binaries: 1.4 With hindsight, it was obvious: most string comparisons fail at the first character. Of course, if you have lots of strings (maybe file names or URIs) with long common prefixes, comparing those is going to be costly, but you're not going to use atoms for those anyway, and you should probably use a data structure like A/B/C -> [C,B,A] so that the different bits get compared first. From kenji@REDACTED Mon Oct 12 03:04:02 2015 From: kenji@REDACTED (Kenji Rikitake) Date: Mon, 12 Oct 2015 10:04:02 +0900 Subject: [erlang-questions] Building OTP on OS X 10.11 El Capitan Message-ID: <20151012010402.GA9242@k2r.org> OS X 10.11 El Capitan seems to have ditched /usr/include altogether (presumably due to System Integrity Protection policies, even untouchable with XCode installation). This prevents Erlang/OTP 18.1.1 to be built with crypto-related modules including ssh and ssl modules. What I did was to install HomeBrew OpenSSL (after creating /usr/local) and use --with-ssl=/usr/local/opt/openssl when doing Configure. OTOH, I find XCode 7.1 places the directory under /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-migrator/sdk/MacOSX.sdk/usr/include if you really want to use the stock OpenSSL 0.9.8 (though I haven't tried it). Any other possible practical solutions on this? Regards, Kenji Rikitake From vasdeveloper@REDACTED Mon Oct 12 03:59:38 2015 From: vasdeveloper@REDACTED (Theepan) Date: Mon, 12 Oct 2015 07:29:38 +0530 Subject: [erlang-questions] If you make WhatsApp today... In-Reply-To: References:

Message-ID: Vimal, As Joe mentioned, focus of a new product should be on key-differentiators and time-to-market. This way you can check the product-market fit, and, if needed, can fail-fast. If you are familiar with Mnesia, it does not take more than 10 minutes to set it up. Since it is a Erlang-integrated DBMS, it will help you code fast in case you need transactional guarantees, and it withstands decent load before you need to look at performance optimisation options. As Jesper mentioned, when you look at storing data, you have to select the right mix of mechanism. It is not always database alone. This worries you only later. Some of them of course you should have it at the initial stage itself -- e.g. storing media files in file system. Theepan On Mon, Oct 12, 2015 at 3:14 AM, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > > On Sun, Oct 11, 2015 at 7:48 PM, Joe Armstrong wrote: > >> I'd probably use the file system, then ets or dets and "roll my own" >> first. > > > To much extent, this is what WhatsApp is doing according to what I've been > able to dissect out of their talks. They use mnesia for storing meta-data, > but they use an UFS2 file system on FreeBSD to store the majority of data > and just pick it off of the disk whenever they need it. The other trick > they use is to use the clients to store the majority of data and only use > the servers as a temporary transient store until the client comes back up > and requests its missing data. > > Were I to start, I'd definitely do the API modularity split as well, but > I'd probably pick postgresql as a backing store over mnesia, perhaps with > mnesia as an in-memory cache for active users. The assumption is user data > is not changing that often, and postgresql have the added value of being a > system I *know* how to operate and handle. There is much to be said for > picking technology which is tried and proven and you know how to operate. > > A possible switch is Riak for the UFS2 store, but it does require the > ability to "append" to an object in Riak. Otherwise it is kind of moot to > load data, add a couple of bytes and store the object back again. Perhaps > this can be done by "chaining" documents in Riak with content addressing. > > > -- > J. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas@REDACTED Mon Oct 12 01:13:05 2015 From: thomas@REDACTED (Thomas Gebert) Date: Sun, 11 Oct 2015 19:13:05 -0400 Subject: [erlang-questions] Why should you ever use atoms? In-Reply-To: References:

Message-ID: <561AED01.8060209@gebert.sexy> I understood for the most part how to use atoms and what they were used for, I was just curious what technical advantages they had over a string (or binary string) considering that they aren't garbage collected. The O(1) comparison is a valid enough reason that I hadn't thought about, and the idea that they are limited by design actually kind of makes valid sense. On 10/11/2015 05:27 PM, Max Lapshin wrote: > You can think about atoms as of a predefined words of your domain logic. > > Each domain description has a limited amount of words that describe > it, so number of atoms is also limited by design. > > In a properly designed program you should never meet dynamic creation > by list_to_atom, only if you know that it is a > string representation of already described atom. > > In other words, atoms are always written by hands by some human as a > part of described functionality. > > > Maybe this will help you. > > From ulf@REDACTED Mon Oct 12 08:03:08 2015 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 12 Oct 2015 08:03:08 +0200 Subject: [erlang-questions] If you make WhatsApp today... In-Reply-To: References:

Message-ID: <1F5C5B03-1CAB-47DB-AFEB-306584201D10@feuerlabs.com> > On 11 Oct 2015, at 23:44, Jesper Louis Andersen wrote: > > Were I to start, I'd definitely do the API modularity split as well, but I'd probably pick postgresql as a backing store over mnesia, perhaps with mnesia as an in-memory cache for active users. The assumption is user data is not changing that often, and postgresql have the added value of being a system I *know* how to operate and handle. There is much to be said for picking technology which is tried and proven and you know how to operate. FWIW, while playing around* with the extended mnesia version at Klarna, we also experimented with a pgsql backend to mnesia. It worked pretty well, although there were some bootstrapping challenges esp. during testing**, but ultimately, the leveldb backend was a better fit for their needs. * They stopped playing around and are now using ?mnesia_ext? with a leveldb backend ** You could either create a pgsql instance inside the mnesia directory, or connect to a running pgsql, in which case consistency checks where needed As I understand it, Klarna will publish the leveldb backend (the pgsql backend was an experiment, so I don?t know about that). For those who want to experiment with *some* backend, there?s https://github.com/klarna/otp/blob/OTP-17.5.6-mnesia_ext/lib/mnesia/test/mnesia_ext_filesystem.erl which is used in the test suite. A very simple example of how to get started with it can be found here: https://github.com/klarna/otp/blob/OTP-17.5.6-mnesia_ext/lib/mnesia/test/mnesia_ext_test.erl BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Mon Oct 12 11:08:03 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Mon, 12 Oct 2015 11:08:03 +0200 Subject: [erlang-questions] Exometer, Recon and prim_inet:send/2? Message-ID: All, This is an operational question, i.e. how to pin point alerts and co. I monitor process queues and get the longest queue with Recon, using: recon:proc_count(message_queue_len, 1). I'm using exometer_core to send data to HostedGraphite. Tonight I received errors stating that the queue exceeded this number, and this is the queue info that Recon gives me: [exometer_report_graphite,{current_function,{prim_inet,send,3}},{initial_call,{proc_lib,init_p,3}}] I got reports every 5 seconds (the interval of time used by exometer to report) from around 00:30 until 08:20, with a message queue growing from a few hundreds to 6k or so. After 08:20, the queue went back to 0. I can see that my server was up (no other errors, all functionalities and pings were correct during the same period). Therefore, it looks to me that exometer was unable to send the data to the HostedGraphite servers during the whole period 00:30 - 08:20, and when it was able to it emptied the queue that went then back to normal. However: HostedGraphite only has a "hole" of data from around 07:40 until 08:20, i.e. not the whole period where the queue kept growing. Therefore, my questions is: Is exometer_core performing some local cache, i.e. if it's unable to send data it will retry? And if so, is this cache limited? Otherwise I don't see what could be causing the queue lenght of exometer_report_graphite to grow. But if this is so, then why am I seeing a hole in data? Than kyou, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Mon Oct 12 11:13:12 2015 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 12 Oct 2015 11:13:12 +0200 Subject: [erlang-questions] If you make WhatsApp today... In-Reply-To: <1F5C5B03-1CAB-47DB-AFEB-306584201D10@feuerlabs.com> References:

<1F5C5B03-1CAB-47DB-AFEB-306584201D10@feuerlabs.com> Message-ID: On Mon, Oct 12, 2015 at 8:03 AM, Ulf Wiger wrote: > * They stopped playing around and are now using ?mnesia_ext? with a > leveldb backend > Correct, we have been using leveldb as a backend for mnesia for some time now, and it works very well. The space savings (both on disk and in memory) are enormous. As I understand it, Klarna will publish the leveldb backend (the pgsql > backend was an experiment, so I don?t know about that). > We intended to do this sooner, but as usual, we've been kept busy with more pressing things. There is one major performance-improving change that we want to do before we push this out, so that people don't start relying on the old behaviour. Hope to have this done soonish. /Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From knv.suresh2009@REDACTED Mon Oct 12 11:39:23 2015 From: knv.suresh2009@REDACTED (suresh knv) Date: Mon, 12 Oct 2015 15:09:23 +0530 Subject: [erlang-questions] Facing problem with the installation of rabbitmqserver on arm64 Message-ID: Hi, I am facing problems with the installation of rabbitmq-server can anyone please helpme out with this. I am trying to install openstack using devstack on arm64 machine,while running ./stack.sh it fails saying that the 2015-10-12 09:16:15.437 | wget is already the newest version. 2015-10-12 09:16:15.437 | 0 upgraded, 0 newly installed, 0 to remove and 30 not upgraded. 2015-10-12 09:16:15.437 | 1 not fully installed or removed. 2015-10-12 09:16:15.437 | After this operation, 0 B of additional disk space will be used. 2015-10-12 09:16:15.466 | Setting up rabbitmq-server (3.2.4-1) ... 2015-10-12 09:16:15.645 | * Starting message broker rabbitmq-server 2015-10-12 09:16:20.142 | * FAILED - check /var/log/rabbitmq/startup_\{log, _err\} 2015-10-12 09:16:20.145 | ...fail! 2015-10-12 09:16:20.149 | invoke-rc.d: initscript rabbitmq-server, action "start" failed. 2015-10-12 09:16:20.155 | dpkg: error processing package rabbitmq-server (--configure): 2015-10-12 09:16:20.155 | subprocess installed post-installation script returned error exit status 1 2015-10-12 09:16:20.264 | Errors were encountered while processing: 2015-10-12 09:16:20.266 | rabbitmq-server 2015-10-12 09:16:20.815 | E: Sub-process /usr/bin/dpkg returned an error code (1) 2015-10-12 09:16:20.822 | + exit_trap 2015-10-12 09:16:20.822 | + local r=100 2015-10-12 09:16:20.825 | ++ jobs -p 2015-10-12 09:16:20.829 | + jobs= 2015-10-12 09:16:20.829 | + [[ -n '' ]] 2015-10-12 09:16:20.829 | + kill_spinner 2015-10-12 09:16:20.829 | + '[' '!' -z '' ']' 2015-10-12 09:16:20.829 | + [[ 100 -ne 0 ]] 2015-10-12 09:16:20.830 | + echo 'Error on exit' 2015-10-12 09:16:20.830 | Error on exit 2015-10-12 09:16:20.830 | + [[ -z /opt/stack/logs ]] 2015-10-12 09:16:20.830 | + /home/ubuntu/devstack/tools/worlddump.py -d /opt/stack/logs 2015-10-12 09:16:21.654 | + exit 100 current erlang version which I am having is R16B03 and /var/log/rabbitmq/start_err is like this Crash dump was written to: erl_crash.dump Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{k ubuntu@REDACTED:/var/log/rabbitmq$ cat startup_log {error_logger,{{2015,10,12},{9,16,17}},"Protocol: ~tp: register/listen error: ~tp~n",["inet_tcp",econnrefused]} {error_logger,{{2015,10,12},{9,16,17}},crash_report,[[{initial_call,{net_kernel,init,['Argument__1']}},{pid,<0.21.0>},{registered_name,[]},{error_info,{exit,{error,badarg},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,320}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[net_sup,kernel_sup,<0.10.0>]},{messages,[]},{links,[#Port<0.93>,<0.18.0>]},{dictionary,[{longnames,false}]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,811}],[]]} {error_logger,{{2015,10,12},{9,16,17}},supervisor_report,[{supervisor,{local,net_sup}},{errorContext,start_error},{reason,{'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel},{mfargs,{net_kernel,start_link,[[rabbitmqprelaunch3622,shortnames]]}},{restart_type,permanent},{shutdown,2000},{child_type,worker}]}]} {error_logger,{{2015,10,12},{9,16,17}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}},{offender,[{pid,undefined},{name,net_sup},{mfargs,{erl_distribution,start_link,[]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]} {error_logger,{{2015,10,12},{9,16,17}},crash_report,[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.9.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,133}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[<0.8.0>]},{messages,[{'EXIT',<0.10.0>,normal}]},{links,[<0.8.0>,<0.7.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,117}],[]]} {error_logger,{{2015,10,12},{9,16,17}},std_info,[{application,kernel},{exited,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}}},{type,permanent}]} {"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}}}"} my hostname is arm64 in etc/hosts 127.0.0.1 localhost arm64 ::1 localhost arm64 ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters Thanks Suresh KN V From anna.grzybowska@REDACTED Mon Oct 12 12:45:10 2015 From: anna.grzybowska@REDACTED (Anna Grzybowska) Date: Mon, 12 Oct 2015 11:45:10 +0100 Subject: [erlang-questions] Berlin Erlang Factory Lite 1 Dec - Early Bird rates open! Message-ID: Berlin Erlang Factory Lite 1 Dec - Early Bird rates open! The third edition of the Erlang Factory Lite in Berlin will take place on 1 Dec! Berlin Erlang Factory Lite is a whole day full of Erlang and Elixir and, on top of that, three days of hands-on training on OTP, Advanced Erlang Techniques and Phoenix on 2-4 Dec. So far our speakers are Heinz Gies and Sonny Scroggin and new speakers will be announced shortly! The Early Bird tickets are on sale now and the price for one ticket is 89 EUR + VAT. In addition to that we have a special discount for students - 50% off the regular price. To get it please email conferences@REDACTED from your university email account for details. -- Anna Grzybowska Conference Executive Erlang Solutions Ltd ?New Loom House?Back Church Lane?London? E1 1LU -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Mon Oct 12 13:19:01 2015 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 12 Oct 2015 13:19:01 +0200 Subject: [erlang-questions] Exometer, Recon and prim_inet:send/2? In-Reply-To: References: Message-ID: > On 12 Oct 2015, at 11:08, Roberto Ostinelli wrote: > > Therefore, my questions is: Is exometer_core performing some local cache, i.e. if it's unable to send data it will retry? And if so, is this cache limited? Otherwise I don't see what could be causing the queue lenght of exometer_report_graphite to grow. But if this is so, then why am I seeing a hole in data? No, exometer_report does no such thing, and neither does exometer_report_graphite (is that the one you?re using?) If a cache would be added, it would need to be in the graphite reporter. That is, it could well be a core library function, but the caching would need to be done by the reporter instance. In the case you?re describing, 8 hours of data could end up being quite a lot, so depending on your workload, spooling to disk might be required. BR, Ulf Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Mon Oct 12 13:24:40 2015 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 12 Oct 2015 13:24:40 +0200 Subject: [erlang-questions] Exometer, Recon and prim_inet:send/2? In-Reply-To: References:

Message-ID: > On 12 Oct 2015, at 13:19, Ulf Wiger wrote: > > In the case you?re describing, 8 hours of data could end up being quite a lot, so depending on your workload, spooling to disk might be required. ?or, crazy thought, using RabbitMQ as an intermediary. :) http://www.somic.org/2009/05/21/graphite-rabbitmq-integration/ https://github.com/Feuerlabs/exometer/blob/master/doc/exometer_report_amqp.md Some assembly might be required ? BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Mon Oct 12 13:58:11 2015 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 12 Oct 2015 13:58:11 +0200 Subject: [erlang-questions] Facing problem with the installation of rabbitmqserver on arm64 In-Reply-To: References: Message-ID: On Mon, Oct 12, 2015 at 11:39 AM, suresh knv wrote: > > {application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}} > This is the underlying problem. Usually it means you have some odd configuration in and around the epmd application. Common errors are firewall rules, or that you have no/oddly-configured localhost interfaces. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Mon Oct 12 15:09:31 2015 From: roberto@REDACTED (Roberto Ostinelli) Date: Mon, 12 Oct 2015 15:09:31 +0200 Subject: [erlang-questions] Exometer, Recon and prim_inet:send/2? In-Reply-To: References:

Message-ID: Thank you Ulf.? Yes, I'm using exometer_report_graphite [1]. Unfortunately this means that I have no idea what happened, i.e. what would make its queue grow like that when a connection is unavailable. [1] https://github.com/Feuerlabs/exometer/blob/master/src/exometer_report_graphite.erl -------------- next part -------------- An HTML attachment was scrubbed... URL: From t@REDACTED Mon Oct 12 15:14:36 2015 From: t@REDACTED (Tristan Sloughter) Date: Mon, 12 Oct 2015 08:14:36 -0500 Subject: [erlang-questions] If you make WhatsApp today... In-Reply-To: References:

<1F5C5B03-1CAB-47DB-AFEB-306584201D10@feuerlabs.com> Message-ID: <1444655676.545392.407872329.7AC25FB7@webmail.messagingengine.com> I'd love to see the postgres one even if it was just experimental. Someone else might take it up and continue with it. -- Tristan Sloughter t@REDACTED On Mon, Oct 12, 2015, at 04:13 AM, Richard Carlsson wrote: > On Mon, Oct 12, 2015 at 8:03 AM, Ulf Wiger wrote: >> * They stopped playing around and are now using ?mnesia_ext? with a leveldb backend >> > > Correct, we have been using leveldb as a backend for mnesia for some time now, and it works very well. The space savings (both on disk and in memory) are enormous. >> As I understand it, Klarna will publish the leveldb backend (the pgsql backend was an experiment, so I don?t know about that). > > We intended to do this sooner, but as usual, we've been kept busy with more pressing things. There is one major performance-improving change that we want to do before we push this out, so that people don't start relying on the old behaviour. Hope to have this done soonish. > ??? /Richard > > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.j.thompson@REDACTED Mon Oct 12 15:55:32 2015 From: s.j.thompson@REDACTED (Simon Thompson) Date: Mon, 12 Oct 2015 14:55:32 +0100 Subject: [erlang-questions] CFP: Workshop on Real World Domain Specific Languages, March 2016 Message-ID: <57F1B9D7-FD15-4D3C-ACB8-FB0EA1AD1BA8@kent.ac.uk> CALL FOR PAPERS ================================= Workshop on Real World Domain Specific Languages https://sites.google.com/site/realworlddsl In conjunction with The International Symposium on Code Generation and Optimisation 2016. http://cgo.org/cgo2016 Barcelona, 12-13 March, 2016 ================================= As the use of computers proliferates, the complexity and variety of systems continues to grow. As a result, it is becoming increasingly inflexible to "hard wire? behaviours into software. Software developers can enable more control over their software configurations by exploiting Domain Specific Languages (DSLs). Such DSLs provide a systematic way to structure the underlying computational components: to coin a phrase, a DSL is a library with syntax. There is an enormous variety of DSLs for a very wide range of domains. Most DSLs are highly idiosyncratic, reflecting both the specific natures of their application domains and their designers? own preferences. This workshop will bring together constructors of DSLs for ?real world? domains; that is, DSLs intended primarily to aid in building software to solve real world problems rather than to explore the more theoretical aspects of language design and implementation. We are looking for submissions that present the motivation, design, implementation, use and evaluation of such DSLs. ================================= Key Dates: ---------------- Paper submission deadline: 10 January 2016. Author notification: 2 February 2016. Final manuscript due: 21 February 2016. Workshop: 12-13 March 2016. ---------------- Submission Instructions: The EasyChair submission page for this workshop is https://easychair.org/conferences/?conf=rwdsl16 Accepted submissions will be published in the ACM Digital Library within its International Conference Proceedings Series. Submissions should be 8-10 in ACM double-column format. Authors should follow the information for formatting ACM SIGPLAN conference papers, which can be found at http://www.sigplan.org/Resources/Author. We will provide more details of the submissions format and process at the beginning of November. ================================= Program Chairs Rob Stewart, Heriot-Watt University Greg Michaelson, Heriot-Watt University PC Members Jost Berthold, Commonwealth Bank of Australia Andy Gill, University of Kansas Kevin Hammond, University of St Andrews Rita Loogen, Philipps Universitat Marburg Patrick Maier, University of Glasgow Josef Svenningsson, Chalmers University Simon Thompson, University of Kent Phil Trinder, University of Glasgow Contact Please email inquiries concerning the workshop to: R.Stewart@REDACTED Simon Thompson | Professor of Logic and Computation School of Computing | University of Kent | Canterbury, CT2 7NF, UK s.j.thompson@REDACTED | M +44 7986 085754 | W www.cs.kent.ac.uk/~sjt From ulf@REDACTED Mon Oct 12 16:30:19 2015 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 12 Oct 2015 16:30:19 +0200 Subject: [erlang-questions] Exometer, Recon and prim_inet:send/2? In-Reply-To: References:

Message-ID: > On 12 Oct 2015, at 15:09, Roberto Ostinelli wrote: > > Thank you Ulf.? Yes, I'm using exometer_report_graphite [1]. > > Unfortunately this means that I have no idea what happened, i.e. what would make its queue grow like that when a connection is unavailable. > > > > [1] https://github.com/Feuerlabs/exometer/blob/master/src/exometer_report_graphite.erl Well, the two potentially blocking operations it performs are gen_tcp:connect() (5-second timeout by default) and gen_tcp:send() If you want to create some form of buffering behavior, perhaps exometer_report_logger.erl might be useful. It?s a bit intricate, perhaps, but you can e.g. check the following code for ideas: https://github.com/Feuerlabs/exometer_core/blob/master/test/exometer_report_SUITE.erl https://github.com/Feuerlabs/exometer_core/blob/master/test/exometer_test_udp_reporter.erl But I think that in general, the idea with metrics is that you push them off-host if possible, otherwise let it be, until a connection is re-established. BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From Tobias.Schlager@REDACTED Mon Oct 12 17:14:13 2015 From: Tobias.Schlager@REDACTED (Tobias Schlager) Date: Mon, 12 Oct 2015 15:14:13 +0000 Subject: [erlang-questions] Exometer, Recon and prim_inet:send/2? In-Reply-To: References:

, Message-ID: <12F2115FD1CCEE4294943B2608A18FA301A2710ED2@MAIL01.win.lbaum.eu> Hi, Graphite does also support UDP. Maybe that could be an option for you. Fire and forget might in this case be better than loading your system. Another possibility would be to automatically close the TCP session when the other end gets unresponsive using the send_timeout option. Regards Tobias -------------- next part -------------- An HTML attachment was scrubbed... URL: From shefys@REDACTED Mon Oct 12 18:16:26 2015 From: shefys@REDACTED (Yury Shefer) Date: Mon, 12 Oct 2015 09:16:26 -0700 Subject: [erlang-questions] httpc request - stream to file Message-ID: Hi, I'm studying Erlang and have a simple question. In my http request function i'm using stream to save the output to the file. The question is how to replace the file content instead of adding additional data to it? fetch_db_ver() -> httpc:request(get, {"http://www.domain.com/CURRENTDBVERSION",[]}, [], [{stream, "CURRENTDBVERSION"}] ). current_version() -> file:read_file("CURRENTDBVERSION"). So, instead of 32154810 I'm getting "3215334632153346321533463215334632154810" 19> db:current_version(). {ok,<<"3215334632153346321533463215334632154810">>} -- Thanks a lot, Yury. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.meadows.jonsson@REDACTED Mon Oct 12 20:11:51 2015 From: eric.meadows.jonsson@REDACTED (=?UTF-8?Q?Eric_Meadows=2DJ=C3=B6nsson?=) Date: Mon, 12 Oct 2015 13:11:51 -0500 Subject: [erlang-questions] httpc request - stream to file In-Reply-To: References: Message-ID: Delete the file before calling httpc:request. On Mon, Oct 12, 2015 at 11:16 AM, Yury Shefer wrote: > Hi, > > I'm studying Erlang and have a simple question. In my http request > function i'm using stream to save the output to the file. The question is > how to replace the file content instead of adding additional data to it? > > fetch_db_ver() -> > httpc:request(get, > {"http://www.domain.com/CURRENTDBVERSION",[]}, > [], > [{stream, "CURRENTDBVERSION"}] > ). > > current_version() -> > file:read_file("CURRENTDBVERSION"). > > > So, instead of > > 32154810 > > I'm getting "3215334632153346321533463215334632154810" > > 19> db:current_version(). > {ok,<<"3215334632153346321533463215334632154810">>} > > -- > Thanks a lot, > Yury. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- Eric Meadows-J?nsson -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennethlakin@REDACTED Mon Oct 12 21:48:41 2015 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Mon, 12 Oct 2015 12:48:41 -0700 Subject: [erlang-questions] If you make WhatsApp today... In-Reply-To: References:

<1F5C5B03-1CAB-47DB-AFEB-306584201D10@feuerlabs.com> Message-ID: <561C0E99.4070803@gmail.com> On 10/12/2015 02:13 AM, Richard Carlsson wrote: > As I understand it, Klarna will publish the leveldb backend (the > pgsql backend was an experiment, so I don?t know about that). The Mnesia leveldb backend relies on mnesia_ext, which is a patch to Mnesia, right? If I'm not wrong about that, will one be required to make use of a patched Erlang/OTP, or are there plans to get the patch into Erlang, or -somehow- make it a library that can be included in one's software? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From steven.charles.davis@REDACTED Tue Oct 13 02:08:12 2015 From: steven.charles.davis@REDACTED (Steve Davis) Date: Mon, 12 Oct 2015 19:08:12 -0500 Subject: [erlang-questions] Mnesia Message-ID: Hi, I?m almost ashamed to say that it?s taken me over 5 years to come around to understanding the value of mnesia. I bought the ?well-known? negatives too fast. I have explored relational connectors, DHT solutions and a number of other approaches... It?s dawned on me *finally* that 90% of the time, a well implemented mnesia solution would have been better, faster, and cheaper. Did I mention "better"? Has anyone else had this experience? /s From steven.charles.davis@REDACTED Tue Oct 13 02:36:01 2015 From: steven.charles.davis@REDACTED (Steve Davis) Date: Mon, 12 Oct 2015 19:36:01 -0500 Subject: [erlang-questions] Mnesia In-Reply-To: References: Message-ID: Yes :) Let?s discuss what the tick boxes are. And be specific on where they break down (e.g. terabyte data storage). And given where they break down, what paths to take. I suspect there?s not too many use cases that aren?t addressed, and those unreasonably dominate discussion about mnesia. /s In general, > On Oct 12, 2015, at 7:29 PM, T Ty wrote: > > Always lol > > Use mnesia until it is no longer useful. As long as you have all the tick boxes marked mnesia is great. And once it breaks down then either consider moving to another system or radically changing your architecture the way WhatsApp did. > > On Tue, Oct 13, 2015 at 1:08 AM, Steve Davis > wrote: > Hi, > > I?m almost ashamed to say that it?s taken me over 5 years to come around to understanding the value of mnesia. > > I bought the ?well-known? negatives too fast. I have explored relational connectors, DHT solutions and a number of other approaches... > > It?s dawned on me *finally* that 90% of the time, a well implemented mnesia solution would have been better, faster, and cheaper. > > Did I mention "better"? > > Has anyone else had this experience? > > /s > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martink@REDACTED Tue Oct 13 03:04:27 2015 From: martink@REDACTED (Martin Karlsson) Date: Tue, 13 Oct 2015 14:04:27 +1300 Subject: [erlang-questions] Mnesia In-Reply-To: References: Message-ID: Hi Steve, I'm with you:) The negatives are being mentioned a lot and unfortunately put people off instead of just being used as good things to know about when you are picking your technology. If you know about the quirks you can usually come up with a workaround that matches your use-case. It is about weighing pros and cons. Why not list a few of the negatives and what you can do to work around them. PROBLEM: Net splits must be manually handled. WORKAROUND: - https://github.com/uwiger/unsplit for automatic healing - Use the majority options (when you need to heal you choose the majority partition when you pick master nodes). This should reduce (eliminate?) risk of data loss. PROBLEM: disc_only_copies uses dets and is limited to 2GB WORKAROUND: - Partition the tables - Use the mnesia_ex patch with leveldb backend (to be released soon I think, it was mentioned in another email just recently) - Perhaps you can use disc_copies and hold your data-set in RAM PROBLEM: Prone to overloading WORKAROUND: Load regulate. Reduce concurrency. I think I read somewhere that mnesia and ets works best with a max concurrency of the number of scheduler threads and degrades from there on. PROBLEM: Slow startup WORKAROUND: Increase the number of table loaders. PROBLEM: Upgrading table definition. transform_table can have problems with big and distributed tables. WORKAROUND: Treat large tables like a key value store to reduce risk of having to modify the record I'm sure there are more problems and workarounds especially if you start pushing it (for example like whatsapp did). Also I've heard dirty_write is not safe with replicated tables. I don't know if this is true but is holding me back from using dirty transactions rather than transactions (which I don't need apart from safety on replicated tables). If anyone knows about this one please let me know. Feel free to add/remove/correct items:) Cheers, Martin On 13 October 2015 at 13:08, Steve Davis wrote: > Hi, > > I?m almost ashamed to say that it?s taken me over 5 years to come around to understanding the value of mnesia. > > I bought the ?well-known? negatives too fast. I have explored relational connectors, DHT solutions and a number of other approaches... > > It?s dawned on me *finally* that 90% of the time, a well implemented mnesia solution would have been better, faster, and cheaper. > > Did I mention "better"? > > Has anyone else had this experience? > > /s > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From g@REDACTED Tue Oct 13 03:48:11 2015 From: g@REDACTED (Garrett Smith) Date: Mon, 12 Oct 2015 20:48:11 -0500 Subject: [erlang-questions] Mnesia In-Reply-To: References:

Message-ID: There's no substitute for production experience. If Mnesia is what you really really really want to use - it's super awesome - just dive in and use it. As you say, if you run into issues, you can just work around them. Personally, I'm interested in the physics of a system at scale, less so the brands involved. Get something in motion, measure it, fix it. On Mon, Oct 12, 2015 at 8:04 PM, Martin Karlsson wrote: > Hi Steve, > > I'm with you:) > > The negatives are being mentioned a lot and unfortunately put people > off instead of just being used as good things to know about when you > are picking your technology. > > If you know about the quirks you can usually come up with a workaround > that matches your use-case. > > It is about weighing pros and cons. Why not list a few of the > negatives and what you can do to work around them. > > PROBLEM: Net splits must be manually handled. > WORKAROUND: > - https://github.com/uwiger/unsplit for automatic healing > - Use the majority options (when you need to heal you choose the > majority partition when you pick master nodes). This should reduce > (eliminate?) risk of data loss. > > PROBLEM: disc_only_copies uses dets and is limited to 2GB > WORKAROUND: > - Partition the tables > - Use the mnesia_ex patch with leveldb backend (to be released soon I > think, it was mentioned in another email just recently) > - Perhaps you can use disc_copies and hold your data-set in RAM > > PROBLEM: Prone to overloading > WORKAROUND: Load regulate. Reduce concurrency. I think I read > somewhere that mnesia and ets works best with a max concurrency of the > number of scheduler threads and degrades from there on. > > PROBLEM: Slow startup > WORKAROUND: Increase the number of table loaders. > > PROBLEM: Upgrading table definition. transform_table can have problems > with big and distributed tables. > WORKAROUND: Treat large tables like a key value store to reduce risk > of having to modify the record > > I'm sure there are more problems and workarounds especially if you > start pushing it (for example like whatsapp did). > > Also I've heard dirty_write is not safe with replicated tables. I > don't know if this is true but is holding me back from using dirty > transactions rather than transactions (which I don't need apart from > safety on replicated tables). If anyone knows about this one please > let me know. > > Feel free to add/remove/correct items:) > > Cheers, > Martin > > > On 13 October 2015 at 13:08, Steve Davis wrote: >> Hi, >> >> I?m almost ashamed to say that it?s taken me over 5 years to come around to understanding the value of mnesia. >> >> I bought the ?well-known? negatives too fast. I have explored relational connectors, DHT solutions and a number of other approaches... >> >> It?s dawned on me *finally* that 90% of the time, a well implemented mnesia solution would have been better, faster, and cheaper. >> >> Did I mention "better"? >> >> Has anyone else had this experience? >> >> /s >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From zhuoyikang@REDACTED Tue Oct 13 05:19:10 2015 From: zhuoyikang@REDACTED (yikang zhuo) Date: Tue, 13 Oct 2015 11:19:10 +0800 Subject: [erlang-questions] Memory Leak Message-ID: after i eval gc manually: X=[erlang:garbage_collect(P) || P <- erlang:processes(), {status, waiting} == erlang:process_info(P, status)]. top res -> 8.3g erlang:memory() -> 3.6G [{total,3694253080}, {processes,3301211768}, {processes_used,3300343808}, {system,393041312}, {atom,744345}, {atom_used,715567}, {binary,254047488}, {code,18840350}, {ets,61244696}] wow.. does memory Leak in ejabberd or erlang 17.5 erts-6.4 -------------- next part -------------- An HTML attachment was scrubbed... URL: From vances@REDACTED Tue Oct 13 05:43:29 2015 From: vances@REDACTED (Vance Shipley) Date: Tue, 13 Oct 2015 09:13:29 +0530 Subject: [erlang-questions] Memory Leak In-Reply-To: References: Message-ID: On Tue, Oct 13, 2015 at 8:49 AM, yikang zhuo wrote: > top res -> 8.3g > erlang:memory() -> 3.6G ... > wow.. does memory Leak in ejabberd or erlang 17.5 erts-6.4 Not in the buggy way you may suspect: https://blog.heroku.com/archives/2013/11/7/logplex-down-the-rabbit-hole Jump down to the section titled "I Just Keep on Bleeding and I Won't Die". -- -Vance From cchalasani@REDACTED Tue Oct 13 03:02:51 2015 From: cchalasani@REDACTED (Chaitanya Chalasani) Date: Tue, 13 Oct 2015 06:32:51 +0530 Subject: [erlang-questions] Mnesia In-Reply-To: References: Message-ID: <85BE6422-4309-465C-B830-6066CA939282@me.com> I have almost always managed to delivery solutions using just mnesia. In one case when the data volumes were too high, I have partitioned the data model to use only mnesia for transactions and the rest to oracle. > On 13-Oct-2015, at 5:38 AM, Steve Davis wrote: > > Hi, > > I?m almost ashamed to say that it?s taken me over 5 years to come around to understanding the value of mnesia. > > I bought the ?well-known? negatives too fast. I have explored relational connectors, DHT solutions and a number of other approaches... > > It?s dawned on me *finally* that 90% of the time, a well implemented mnesia solution would have been better, faster, and cheaper. > > Did I mention "better"? > > Has anyone else had this experience? > > /s > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mjtruog@REDACTED Tue Oct 13 07:04:35 2015 From: mjtruog@REDACTED (Michael Truog) Date: Mon, 12 Oct 2015 22:04:35 -0700 Subject: [erlang-questions] If you make WhatsApp today... In-Reply-To: <561C0E99.4070803@gmail.com> References:

<1F5C5B03-1CAB-47DB-AFEB-306584201D10@feuerlabs.com> <561C0E99.4070803@gmail.com> Message-ID: <561C90E3.10900@gmail.com> On 10/12/2015 12:48 PM, Kenneth Lakin wrote: > On 10/12/2015 02:13 AM, Richard Carlsson wrote: >> As I understand it, Klarna will publish the leveldb backend (the >> pgsql backend was an experiment, so I don?t know about that). > The Mnesia leveldb backend relies on mnesia_ext, which is a patch to > Mnesia, right? > > If I'm not wrong about that, will one be required to make use of a > patched Erlang/OTP, or are there plans to get the patch into Erlang, or > -somehow- make it a library that can be included in one's software? > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions It looks like the current patch was being tracked at https://github.com/erlang/otp/pull/783 and that the changes needed more changes before merging on a future pull request (based on this email thread). -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Tue Oct 13 07:18:27 2015 From: mjtruog@REDACTED (Michael Truog) Date: Mon, 12 Oct 2015 22:18:27 -0700 Subject: [erlang-questions] Memory Leak In-Reply-To: References:

Message-ID: <561C9423.5070907@gmail.com> On 10/12/2015 08:43 PM, Vance Shipley wrote: > On Tue, Oct 13, 2015 at 8:49 AM, yikang zhuo wrote: >> top res -> 8.3g >> erlang:memory() -> 3.6G > ... >> wow.. does memory Leak in ejabberd or erlang 17.5 erts-6.4 > Not in the buggy way you may suspect: > https://blog.heroku.com/archives/2013/11/7/logplex-down-the-rabbit-hole > > Jump down to the section titled "I Just Keep on Bleeding and I Won't Die". > > There was a GC bug in 17.5 mentioned at http://erlang.org/pipermail/erlang-questions/2015-June/085032.html as OTP-12821 . You may not be running into that though. I have seen abnormal GC memory growth with the 17.5 release (from the download page) but not the 18.0 release, and am not sure about the exact changes it was related to (only that it caused the Erlang VM to be killed by the Linux OS, if you are interested in the testing, it is at https://groups.google.com/forum/#!topic/cloudi-questions/bw2D7YOFtKU). From ok@REDACTED Tue Oct 13 07:57:38 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 13 Oct 2015 18:57:38 +1300 Subject: [erlang-questions] Coming Back (maybe improving lists:reverse/1) In-Reply-To: <001801d10244$4095dd10$c1c19730$@frcuba.co.cu> References: <041e01d1012f$9220aa40$b661fec0$@frcuba.co.cu> <001001d10152$7e80a080$7b81e180$@frcuba.co.cu> <3DE96CA6-56A4-4996-AEC8-9377749DA20C@cs.otago.ac.nz> <001c01d1015d$c36cf8a0$4a46e9e0$@frcuba.co.cu> <001d01d1015f$24b92790$6e2b76b0$@frcuba.co.cu> <656993BD-5652-4E90-B872-2D2D11876952@cs.otago.ac.nz> <001801d10244$4095dd10$c1c19730$@frcuba.co.cu> Message-ID: On 9/10/2015, at 4:40 pm, Ivan Carmenates Garcia wrote: > Regards, > > Well seems to me that there is no more optimizations for the algorithm > (except for yours 'my_reverse_join' see below), I tested yours and mine and > they both take exactly the same for the final result the unparsed "string", > well mine is 200 millisecond faster in 1 million of iterations, yours take > 64XX main 62XX approx. The problem is that using lists:flatten is the hell > slow, The problem is that I suggested *NOT* using lists:flatten/1. By the way, you really do not *have* to use any particular built in function. A few minutes' coding produced this result for flattening a tree with a million elements: - Using lists:flatten/1 : 123,939 microseconds - Using my own code : 63,134 microseconds The effect of my code isn't precisely the same as that of lists:flatten/1, but for a tree of strings that doesn't matter. my_flatten([A,B,C|Xs], R) -> my_flatten(A, my_flatten(B, my_flatten(C, my_flatten(Xs, R)))); my_flatten([A|Xs], R) -> my_flatten(A, my_flatten(Xs, R)); my_flatten([], R) -> R; my_flatten(X, R) -> [X|R]. > I also tested your algorithm just after the io lists return and it is > very faster its took 14XX milliseconds, so when you use the final > lists:flatten/1 everything goes to crap, sorry about the word. So it is > amazing who faster it is and then lists:flatten by its own take the another > 5 seconds, I did knew that because I read Erlang session about 7 myths off > bla bla .. and optimizations stuffs some time ago and they say lists:flatten > is slow also I tested it when I was constructing the algorithm first time I > did avoid constructing the io lists because of that. No, the whole point of io lists is to *NOT* flatten them. The advice, in short, was to structure the rest of your program so that you don't NEED to do the flattening. > > I did clear some things like unparse well I need another name I don't like > unparse either and parse is wrong, I will come up with something. Maybe you don't *like* "unparse", but it *is* the standard technical term. Type "unparsing" into the search box of your browser and read! > I also have questions for you if you are so kind to answer. > > Regards proplists, well, it is necessary to make lists of options as > proplists? No, of course not. Even when you encode a list of options as something *like* a proplist, the structure you want will usually be stricter. In any case, what you have here is simply not a list of options. The elements are not options, they are descriptions of columns you want to get back from some data base query. > yes, proplists:compact/1 and expand/1 for [a] to [{a, true}] and > all those stuffs but what I was trying to do is to make it simple for the > user, because more tuples are more complicate to write in the case of > {users, name, alias} (a) Remember, one of the essential things about *being* a proplist is that the consumer is going to IGNORE anything that is not an atom or a pair. If you are not going to do that, you are not treating the data *as* a proplist, and it is "lying to the user" to call it one. (b) Far from making things simpler for the user, you made it much more confusing. There are places where an alias *can* be supplied and places where it *can't*, and there doesn't seem to be any reason for that. Well, it confused the heck out of *me*. > well that could be right adding more brackets like > {users, {name, alias}} because it organizes the thing. But i.e.: I have > another algorithm for match specs in which I can do {name | {table, name} | > value, op, name | {table, name} | value } also [{..., op, ...}, ...], > logic_op, ... so it will be easy for the user to build something like > [ > [{id, '==', 1}, {age, '==', 31}], > 'or', > {username, '==', "john"} > ] There are programming languages in which building queries as data structures is easy. In Lisp, (make-query '(or (and (= id 1) (= age 31)) (= username "john"))) is easy because that's *exactly* what an expression would look like in Lisp. In R, make_query(id = 1 && age == 31 || username == "john") is easy because R functions evaluate their arguments lazily and can get at the abstract syntax trees of those arguments, so again the syntax is exactly what an expression normally looks like. Erlang really isn't one of those languages, but there's a feature of Erlang that means you can get a lot closer. You could use a parse transform to take something like make_query(Id == 1 andalso Age == 31 orelse UserName == "John") which the parser turns into an abstract syntax tree, and then your parse transform could take make_query(...) and turn that into whatever data structure you like. It's hardER in Erlang than it is in Lisp, but it's still much easier than it is with C or Java. (And of course LFE would do it exactly the way Lisp does it.) If I were a user of your system, I would be somewhere between baffled and outraged at the claim that it would be *easy* for me to construct a query in either of the forms you mention. This is perhaps the ideal time to mention the idea of STAGED computation, also known as partial evaluation or partial execution. The function we started discussing looks like a textbook case of something you probably shouldn't be doing at run time anyway. From carlsson.richard@REDACTED Tue Oct 13 09:01:47 2015 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 13 Oct 2015 09:01:47 +0200 Subject: [erlang-questions] If you make WhatsApp today... In-Reply-To: <561C90E3.10900@gmail.com> References:

<1F5C5B03-1CAB-47DB-AFEB-306584201D10@feuerlabs.com> <561C0E99.4070803@gmail.com> <561C90E3.10900@gmail.com> Message-ID: Yeah, I'm working on a new pull request based on 18.1.1. /Richard On Tue, Oct 13, 2015 at 7:04 AM, Michael Truog wrote: > On 10/12/2015 12:48 PM, Kenneth Lakin wrote: > > On 10/12/2015 02:13 AM, Richard Carlsson wrote: > > As I understand it, Klarna will publish the leveldb backend (the > pgsql backend was an experiment, so I don?t know about that). > > The Mnesia leveldb backend relies on mnesia_ext, which is a patch to > Mnesia, right? > > If I'm not wrong about that, will one be required to make use of a > patched Erlang/OTP, or are there plans to get the patch into Erlang, or > -somehow- make it a library that can be included in one's software? > > > > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > It looks like the current patch was being tracked at > https://github.com/erlang/otp/pull/783 and that the changes needed more > changes before merging on a future pull request (based on this email > thread). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhuoyikang@REDACTED Tue Oct 13 10:21:46 2015 From: zhuoyikang@REDACTED (yikang zhuo) Date: Tue, 13 Oct 2015 16:21:46 +0800 Subject: [erlang-questions] Memory Leak In-Reply-To: References:

Message-ID: "I Just Keep on Bleeding and I Won't Die " very humorous... i follow your post the see what happend in my erlang node , and the binary allocted is also my biggest problem, it almost have 5G?and the usage rate is very low rp(recon_alloc:fragmentation(current)). {mbcs_carriers_size,1438941360 {mbcs_carriers_size,908361904 {mbcs_carriers_size,928284848 {mbcs_carriers_size,817135792 {mbcs_carriers_size,556302512 {mbcs_carriers_size,479494320 {mbcs_carriers_size,399278256 {mbcs_carriers_size,271876272 {mbcs_carriers_size,32944 sum -> 5 799 708 208 {mbcs_usage,0.09427699263575272}, {mbcs_usage,0.010110173004349157}, {mbcs_usage,0.05721490350125805}, {mbcs_usage,0.03553723173589733}, {mbcs_usage,0.0037855806051078915}, {mbcs_usage,0.002756687503618395}, {mbcs_usage,0.0015088825673492223}, {mbcs_usage,0.0015768937717374615}, {mbcs_usage,0.003399708596406022}, but it look like a bug more than alloctor fragment , the default alloctor strage is {as,aoffcbf}]}, does aoffcbf can as bad as such low usage... my erlang is erts6.4 r17.5 , this post describle a memory leak about r17.3 http://erlang.2086793.n4.nabble.com/Possibly-memory-leak-in-R17-td4690007.html https://github.com/erlang/otp/blob/maint/erts/emulator/internal_doc/CarrierMigration.md#searching-the-pool but i use r17.5. rp(erlang:system_info({allocator,binary_alloc})). [{instance,0, [{versions,"0.9","3.0"}, {options,[{e,true}, {t,true}, {ramv,false}, {sbct,524288}, {asbcst,4145152}, {rsbcst,20}, {rsbcmt,80}, {rmbcmt,50}, {mmbcs,32768}, {mmmbc,18446744073709551615}, {mmsbc,256}, {lmbcs,5242880}, {smbcs,262144}, {mbcgs,10}, {acul,0}, {as,aoffcbf}]}, does -->> all [{{binary_alloc,1}, [{sbcs_usage,1.0}, {mbcs_usage,0.09427699263575272}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,135659064}, {mbcs_carriers_size,1438941360}]}, {{binary_alloc,4}, [{sbcs_usage,1.0}, {mbcs_usage,0.010110173004349157}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,9183696}, {mbcs_carriers_size,908361904}]}, {{binary_alloc,2}, [{sbcs_usage,1.0}, {mbcs_usage,0.05721490350125805}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,53111728}, {mbcs_carriers_size,928284848}]}, {{binary_alloc,3}, [{sbcs_usage,1.0}, {mbcs_usage,0.03553723173589733}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,29038744}, {mbcs_carriers_size,817135792}]}, {{binary_alloc,5}, [{sbcs_usage,1.0}, {mbcs_usage,0.0037855806051078915}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,2105928}, {mbcs_carriers_size,556302512}]}, {{binary_alloc,6}, [{sbcs_usage,1.0}, {mbcs_usage,0.002756687503618395}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,1321816}, {mbcs_carriers_size,479494320}]}, {{binary_alloc,7}, [{sbcs_usage,1.0}, {mbcs_usage,0.0015088825673492223}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,602464}, {mbcs_carriers_size,399278256}]}, {{binary_alloc,8}, [{sbcs_usage,1.0}, {mbcs_usage,0.0015768937717374615}, {sbcs_block_size,0}, {sbcs_carriers_size,0}, {mbcs_block_size,428720}, {mbcs_carriers_size,271876272}]}, ? 2015-10-13 11:43 GMT+08:00 Vance Shipley : > On Tue, Oct 13, 2015 at 8:49 AM, yikang zhuo wrote: > > top res -> 8.3g > > erlang:memory() -> 3.6G > ... > > wow.. does memory Leak in ejabberd or erlang 17.5 erts-6.4 > > Not in the buggy way you may suspect: > https://blog.heroku.com/archives/2013/11/7/logplex-down-the-rabbit-hole > > Jump down to the section titled "I Just Keep on Bleeding and I Won't Die". > > > -- > -Vance > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikpelinux@REDACTED Tue Oct 13 09:41:54 2015 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Tue, 13 Oct 2015 09:41:54 +0200 Subject: [erlang-questions] If you make WhatsApp today... In-Reply-To: <561C0E99.4070803@gmail.com> References:

<1F5C5B03-1CAB-47DB-AFEB-306584201D10@feuerlabs.com> <561C0E99.4070803@gmail.com> Message-ID: <22044.46530.616247.300935@gargle.gargle.HOWL> Kenneth Lakin writes: > On 10/12/2015 02:13 AM, Richard Carlsson wrote: > > As I understand it, Klarna will publish the leveldb backend (the > > pgsql backend was an experiment, so I don?t know about that). > > The Mnesia leveldb backend relies on mnesia_ext, which is a patch to > Mnesia, right? > > If I'm not wrong about that, will one be required to make use of a > patched Erlang/OTP, or are there plans to get the patch into Erlang, or > -somehow- make it a library that can be included in one's software? There is an open P.R. to include mnesia_ext in OTP, and we have published versions for R15, R16, and 17 on Klarna's github for those that prefer stable releases over bleeding edge. The R15 and R16 versions have had substantial testing in production. You can bundle a patched OTP, or just a patched mnesia, with your application (mnesia itself is just a library, and mnesia_ext doesn't patch the VM). Our Erlang systems use locally patched stable OTP releases anyway, so adding one more patch for mnesia_ext is no big deal. From mikpelinux@REDACTED Tue Oct 13 09:43:33 2015 From: mikpelinux@REDACTED (Mikael Pettersson) Date: Tue, 13 Oct 2015 09:43:33 +0200 Subject: [erlang-questions] If you make WhatsApp today... In-Reply-To: <1444655676.545392.407872329.7AC25FB7@webmail.messagingengine.com> References:

<1F5C5B03-1CAB-47DB-AFEB-306584201D10@feuerlabs.com> <1444655676.545392.407872329.7AC25FB7@webmail.messagingengine.com> Message-ID: <22044.46629.802283.964778@gargle.gargle.HOWL> Tristan Sloughter writes: > I'd love to see the postgres one even if it was just experimental. > Someone else might take it up and continue with it. I'm working on getting that one open-sourced, hopefully this week. From max.lapshin@REDACTED Tue Oct 13 13:54:15 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 13 Oct 2015 14:54:15 +0300 Subject: [erlang-questions] driver_alloc high usage Message-ID: Hi. Customer came with our flussonic eating too much memory. (flussonic@REDACTED)20> recon_alloc:memory(allocated_types). [{binary_alloc,36737328}, {driver_alloc,13561794864}, {eheap_alloc,101822768}, {ets_alloc,83661104}, {fix_alloc,7377200}, {ll_alloc,82317416}, {sl_alloc,823600}, {std_alloc,2396464}, {temp_alloc,3279800}] Recon alloc told me that 13 GB is used by driver_alloc. There is my driver mpegts_udp that reads UDP with mpegts (unfortunately native inet driver cannot do it, because it consumes too much CPU for each message, so we glue several packets into one long packet, reducing amount of packets by 10-20 and load by 2-3 times). Perhaps my driver can leak. Is it possible to get any other information about this driver_alloc, how is this memory used, etc? -------------- next part -------------- An HTML attachment was scrubbed... URL: From garazdawi@REDACTED Tue Oct 13 14:39:01 2015 From: garazdawi@REDACTED (Lukas Larsson) Date: Tue, 13 Oct 2015 14:39:01 +0200 Subject: [erlang-questions] driver_alloc high usage In-Reply-To: References: Message-ID: You can use instrument and some combination of "+Mis true" and "+Mim true" to get some more details about where the memory is used. See http://www.erlang.org/doc/man/instrument.html for details. Unfortunately it will group all allocations done by driver_alloc into one group. On Tue, Oct 13, 2015 at 1:54 PM, Max Lapshin wrote: > Hi. > > Customer came with our flussonic eating too much memory. > > (flussonic@REDACTED)20> recon_alloc:memory(allocated_types). > > [{binary_alloc,36737328}, > > {driver_alloc,13561794864}, > > {eheap_alloc,101822768}, > > {ets_alloc,83661104}, > > {fix_alloc,7377200}, > > {ll_alloc,82317416}, > > {sl_alloc,823600}, > > {std_alloc,2396464}, > > {temp_alloc,3279800}] > > > > Recon alloc told me that 13 GB is used by driver_alloc. > > > There is my driver mpegts_udp that reads UDP with mpegts (unfortunately > native inet driver cannot do it, because it consumes too much CPU for each > message, so we glue several packets into one long packet, reducing amount > of packets by 10-20 and load by 2-3 times). Perhaps my driver can leak. > > > Is it possible to get any other information about this driver_alloc, how > is this memory used, etc? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue Oct 13 14:48:09 2015 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 13 Oct 2015 15:48:09 +0300 Subject: [erlang-questions] driver_alloc high usage In-Reply-To: References: Message-ID: Ok, will try to. Also I've looked at difference between driver_alloc and driver_free count and possibly I've found problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Tue Oct 13 14:59:20 2015 From: sergej.jurecko@REDACTED (=?UTF-8?Q?Sergej_Jure=C4=8Dko?=) Date: Tue, 13 Oct 2015 14:59:20 +0200 Subject: [erlang-questions] unpacking big/little endian speed difference Message-ID: How come unpacking integers as big endian is faster than little endian, when I'm running on a little endian platform (intel osx)? I know big endian is erlang default, but if you're turning binaries into integers, you must turn them into little endian anyway, so unpacking should basically be memmcpy. The test: % Takes: ~1.3s -define(ENDIAN,unsigned-big). % Takes: ~1.8s % -define(ENDIAN,unsigned-little). loop() -> L = [<<(random:uniform(1000000000)):32/?ENDIAN,1:32>> || _ <- lists:seq(1,1000)], S = os:timestamp(), loop1(1000,L), timer:now_diff(os:timestamp(),S). loop1(0,_) -> ok; loop1(N,L) -> lists:sort(fun(<>,<>) -> A =< B end, L), loop1(N-1,L). -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony@REDACTED Tue Oct 13 16:21:01 2015 From: tony@REDACTED (Tony Rogvall) Date: Tue, 13 Oct 2015 16:21:01 +0200 Subject: [erlang-questions] unpacking big/little endian speed difference In-Reply-To: References: Message-ID: Checking the code with a simple disassembler, that can inspect the loaded code, using erts_debug:disassemble: big-endian: > dis:func(decode_bench, '-loop/2-fun-0-', 2). 000000001813E9A8: [i_func_info_IaaI,0,decode_bench,'-loop/2-fun-0-',2] 000000001813E9D0: [i_bs_start_match2_rfIId,{x,0},{f,403958184},2,0,{x,0}] 000000001813E9F8: [i_bs_get_integer_32_rfId,{x,0},{f,403958480},2,{x,2}] 000000001813EA18: [i_bs_skip_bits_all2_frI,{f,403958480},{x,0},8] 000000001813EA30: [i_bs_start_match2_xfIId,{x,1},{f,403958480},3,0,{x,3}] 000000001813EA60: [i_bs_get_integer_32_xfId,{x,3},{f,403958480},4,{x,4}] 000000001813EA88: [bs_test_unit8_fx,{f,403958480},{x,3}] 000000001813EAA0: [i_fetch_xx,{x,2},{x,4}] 000000001813EAB0: [i_bif2_body_bd,'=<','/',2,{x,0}] 000000001813EAC8: [return] 000000001813EAD0: [bs_context_to_binary_r,{x,0}] 000000001813EAD8: [i_is_lt_spec_frr,{f,403958184},{x,0},{x,0}] The same for little-endian: > dis:func(decode_bench, '-loop/2-fun-0-', 2). 0000000018121D30: [i_func_info_IaaI,0,decode_bench,'-loop/2-fun-0-',2] 0000000018121D58: [i_bs_start_match2_rfIId,{x,0},{f,403840304},2,0,{x,0}] 0000000018121D80: [i_bs_get_integer_small_imm_rIfId, {x,0}, 32, {f,403840616}, 2, {x,2}] 0000000018121DA8: [i_bs_skip_bits_all2_frI,{f,403840616},{x,0},8] 0000000018121DC0: [i_bs_start_match2_xfIId,{x,1},{f,403840616},3,0,{x,3}] 0000000018121DF0: [i_bs_get_integer_small_imm_xIfId, {x,3}, 32, {f,403840616}, 2, {x,4}] 0000000018121E20: [bs_test_unit8_fx,{f,403840616},{x,3}] 0000000018121E38: [i_fetch_xx,{x,2},{x,4}] 0000000018121E48: [i_bif2_body_bd,'=<','/',2,{x,0}] 0000000018121E60: [return] 0000000018121E68: [bs_context_to_binary_r,{x,0}] 0000000018121E70: [i_is_lt_spec_frr,{f,403840304},{x,0},{x,0}] The difference between i_bs_get_integer_32_rfId and i_bs_get_integer_small_imm_xIfId this is most probably the cause of the performance difference. /Tony > On 13 okt 2015, at 14:59, Sergej Jure?ko wrote: > > How come unpacking integers as big endian is faster than little endian, when I'm running on a little endian platform (intel osx)? > I know big endian is erlang default, but if you're turning binaries into integers, you must turn them into little endian anyway, so unpacking should basically be memmcpy. > > The test: > > % Takes: ~1.3s > -define(ENDIAN,unsigned-big). > > % Takes: ~1.8s > % -define(ENDIAN,unsigned-little). > > loop() -> > L = [<<(random:uniform(1000000000)):32/?ENDIAN,1:32>> || _ <- lists:seq(1,1000)], > S = os:timestamp(), > loop1(1000,L), > timer:now_diff(os:timestamp(),S). > > loop1(0,_) -> > ok; > loop1(N,L) -> > lists:sort(fun(<>,<>) -> A =< B end, L), > loop1(N-1,L). > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From kostis@REDACTED Tue Oct 13 16:31:05 2015 From: kostis@REDACTED (Kostis Sagonas) Date: Tue, 13 Oct 2015 16:31:05 +0200 Subject: [erlang-questions] unpacking big/little endian speed difference In-Reply-To: References: Message-ID: <561D15A9.70104@cs.ntua.gr> On 10/13/2015 02:59 PM, Sergej Jure?ko wrote: > How come unpacking integers as big endian is faster than little endian, > when I'm running on a little endian platform (intel osx)? > I know big endian is erlang default, but if you're turning binaries into > integers, you must turn them into little endian anyway, so unpacking > should basically be memmcpy. > > The test: > > % Takes: ~1.3s > -define(ENDIAN,unsigned-big). > > % Takes: ~1.8s > % -define(ENDIAN,unsigned-little). Well, if you are interested in performance, simply compile to native code your module and the time will drop to less than half... ================================================================ Eshell V7.1 (abort with ^G) 1> c(endian). {ok,endian} 2> endian:loop(). 1097971 3> endian:loop(). 1327621 4> endian:loop(). 1162506 5> endian:loop(). 1103813 6> c(endian, [native]). {ok,endian} 7> endian:loop(). 414000 8> endian:loop(). 441327 ================================================================ Kostis From co7eb@REDACTED Tue Oct 13 16:40:47 2015 From: co7eb@REDACTED (Ivan Carmenates Garcia) Date: Tue, 13 Oct 2015 10:40:47 -0400 Subject: [erlang-questions] Coming Back (maybe improving lists:reverse/1) In-Reply-To: References: <041e01d1012f$9220aa40$b661fec0$@frcuba.co.cu> <001001d10152$7e80a080$7b81e180$@frcuba.co.cu> <3DE96CA6-56A4-4996-AEC8-9377749DA20C@cs.otago.ac.nz> <001c01d1015d$c36cf8a0$4a46e9e0$@frcuba.co.cu> <001d01d1015f$24b92790$6e2b76b0$@frcuba.co.cu> <656993BD-5652-4E90-B872-2D2D11876952@cs.otago.ac.nz> <001801d10244$4095dd10$c1c19730$@frcuba.co.cu> Message-ID: <003301d105c5$2767b820$76372860$@frcuba.co.cu> Hi Richard, Yes I will definably like to build queries like list comprehensions or like your nice example, but I don't understand so well parse transform because I didn't find so much documentation about it in Erlang doc, I was trying with qlc but yet nothing I could archive. If you could point me in the right direction to get a good reference and documentation for doing parse transforms, that would be nice because that will be definitely some well form improvement for my little framework. Regards, Ivan (son of Gilberio). -----Original Message----- From: Richard A. O'Keefe [mailto:ok@REDACTED] Sent: Tuesday, October 13, 2015 1:58 AM To: Ivan Carmenates Garcia Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] Coming Back (maybe improving lists:reverse/1) On 9/10/2015, at 4:40 pm, Ivan Carmenates Garcia wrote: > Regards, > > Well seems to me that there is no more optimizations for the algorithm > (except for yours 'my_reverse_join' see below), I tested yours and > mine and they both take exactly the same for the final result the > unparsed "string", well mine is 200 millisecond faster in 1 million of > iterations, yours take 64XX main 62XX approx. The problem is that > using lists:flatten is the hell slow, The problem is that I suggested *NOT* using lists:flatten/1. By the way, you really do not *have* to use any particular built in function. A few minutes' coding produced this result for flattening a tree with a million elements: - Using lists:flatten/1 : 123,939 microseconds - Using my own code : 63,134 microseconds The effect of my code isn't precisely the same as that of lists:flatten/1, but for a tree of strings that doesn't matter. my_flatten([A,B,C|Xs], R) -> my_flatten(A, my_flatten(B, my_flatten(C, my_flatten(Xs, R)))); my_flatten([A|Xs], R) -> my_flatten(A, my_flatten(Xs, R)); my_flatten([], R) -> R; my_flatten(X, R) -> [X|R]. > I also tested your algorithm just after the io lists return and it is > very faster its took 14XX milliseconds, so when you use the final > lists:flatten/1 everything goes to crap, sorry about the word. So it > is amazing who faster it is and then lists:flatten by its own take the > another > 5 seconds, I did knew that because I read Erlang session about 7 myths > off bla bla .. and optimizations stuffs some time ago and they say > lists:flatten is slow also I tested it when I was constructing the > algorithm first time I did avoid constructing the io lists because of that. No, the whole point of io lists is to *NOT* flatten them. The advice, in short, was to structure the rest of your program so that you don't NEED to do the flattening. > > I did clear some things like unparse well I need another name I don't > like unparse either and parse is wrong, I will come up with something. Maybe you don't *like* "unparse", but it *is* the standard technical term. Type "unparsing" into the search box of your browser and read! > I also have questions for you if you are so kind to answer. > > Regards proplists, well, it is necessary to make lists of options as > proplists? No, of course not. Even when you encode a list of options as something *like* a proplist, the structure you want will usually be stricter. In any case, what you have here is simply not a list of options. The elements are not options, they are descriptions of columns you want to get back from some data base query. > yes, proplists:compact/1 and expand/1 for [a] to [{a, true}] and all > those stuffs but what I was trying to do is to make it simple for the > user, because more tuples are more complicate to write in the case of > {users, name, alias} (a) Remember, one of the essential things about *being* a proplist is that the consumer is going to IGNORE anything that is not an atom or a pair. If you are not going to do that, you are not treating the data *as* a proplist, and it is "lying to the user" to call it one. (b) Far from making things simpler for the user, you made it much more confusing. There are places where an alias *can* be supplied and places where it *can't*, and there doesn't seem to be any reason for that. Well, it confused the heck out of *me*. > well that could be right adding more brackets like {users, {name, > alias}} because it organizes the thing. But i.e.: I have another > algorithm for match specs in which I can do {name | {table, name} | > value, op, name | {table, name} | value } also [{..., op, ...}, ...], > logic_op, ... so it will be easy for the user to build something like > [ > [{id, '==', 1}, {age, '==', 31}], > 'or', > {username, '==', "john"} > ] There are programming languages in which building queries as data structures is easy. In Lisp, (make-query '(or (and (= id 1) (= age 31)) (= username "john"))) is easy because that's *exactly* what an expression would look like in Lisp. In R, make_query(id = 1 && age == 31 || username == "john") is easy because R functions evaluate their arguments lazily and can get at the abstract syntax trees of those arguments, so again the syntax is exactly what an expression normally looks like. Erlang really isn't one of those languages, but there's a feature of Erlang that means you can get a lot closer. You could use a parse transform to take something like make_query(Id == 1 andalso Age == 31 orelse UserName == "John") which the parser turns into an abstract syntax tree, and then your parse transform could take make_query(...) and turn that into whatever data structure you like. It's hardER in Erlang than it is in Lisp, but it's still much easier than it is with C or Java. (And of course LFE would do it exactly the way Lisp does it.) If I were a user of your system, I would be somewhere between baffled and outraged at the claim that it would be *easy* for me to construct a query in either of the forms you mention. This is perhaps the ideal time to mention the idea of STAGED computation, also known as partial evaluation or partial execution. The function we started discussing looks like a textbook case of something you probably shouldn't be doing at run time anyway. From gpadovani@REDACTED Tue Oct 13 18:03:53 2015 From: gpadovani@REDACTED (Gianluca Padovani) Date: Tue, 13 Oct 2015 18:03:53 +0200 Subject: [erlang-questions] Delete key doesn't work in erl/iex shell Message-ID: Hi, I'm using Ubuntu 14.04 and in Elixir and Erlang shell the delete key doesn't work. I found this issue[1] on elixir repo but they suggest to fix this problem on erlang shell. Do you consider this issue a bug? Is it possible to fix it or find a workaround? Thank you very much Gianluca [1] https://github.com/elixir-lang/elixir/issues/3471 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Tue Oct 13 19:43:44 2015 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Tue, 13 Oct 2015 10:43:44 -0700 Subject: [erlang-questions] Delete key doesn't work in erl/iex shell In-Reply-To: References: Message-ID: Just wait a little :) https://github.com/erlang/otp/commit/2c7e387961251af59f14bca39cbf8fbbe880383e Cheers, -- Pierre Fenoll On 13 October 2015 at 09:03, Gianluca Padovani wrote: > Hi, > I'm using Ubuntu 14.04 and in Elixir and Erlang shell the delete key > doesn't work. I found this issue[1] on elixir repo but they suggest to fix > this problem on erlang shell. Do you consider this issue a bug? Is it > possible to fix it or find a workaround? > > Thank you very much > Gianluca > > [1] https://github.com/elixir-lang/elixir/issues/3471 > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd@REDACTED Tue Oct 13 19:46:14 2015 From: lloyd@REDACTED (lloyd@REDACTED) Date: Tue, 13 Oct 2015 13:46:14 -0400 (EDT) Subject: [erlang-questions] Mnesia In-Reply-To: References:

Message-ID: <1444758374.470121944@apps.rackspace.com> Hello, During a break at the Chicago Erlang Workshop Fred Hebert mentioned that Mnesia is a fine database for the 1990s. Mnesia was a major draw when I was first attracted to Erlang in part because I hate transiting syntactic boundaries when I'm programming. I asked Fred what it would it take to upgrade Mnesia for the 21st century (or, at least, for the next decade). He didn't know. Martin's list of negatives may be a good place to start. So, just how much effort and knowledge would it take to overcome or ameliorate these and other unspecified negatives of Mnesia to produce Mnesia2? Best, LRP -----Original Message----- From: "Martin Karlsson" Sent: Monday, October 12, 2015 9:04pm To: "Steve Davis" Cc: "Erlang Questions" Subject: Re: [erlang-questions] Mnesia Hi Steve, I'm with you:) The negatives are being mentioned a lot and unfortunately put people off instead of just being used as good things to know about when you are picking your technology. If you know about the quirks you can usually come up with a workaround that matches your use-case. It is about weighing pros and cons. Why not list a few of the negatives and what you can do to work around them. PROBLEM: Net splits must be manually handled. WORKAROUND: - https://github.com/uwiger/unsplit for automatic healing - Use the majority options (when you need to heal you choose the majority partition when you pick master nodes). This should reduce (eliminate?) risk of data loss. PROBLEM: disc_only_copies uses dets and is limited to 2GB WORKAROUND: - Partition the tables - Use the mnesia_ex patch with leveldb backend (to be released soon I think, it was mentioned in another email just recently) - Perhaps you can use disc_copies and hold your data-set in RAM PROBLEM: Prone to overloading WORKAROUND: Load regulate. Reduce concurrency. I think I read somewhere that mnesia and ets works best with a max concurrency of the number of scheduler threads and degrades from there on. PROBLEM: Slow startup WORKAROUND: Increase the number of table loaders. PROBLEM: Upgrading table definition. transform_table can have problems with big and distributed tables. WORKAROUND: Treat large tables like a key value store to reduce risk of having to modify the record I'm sure there are more problems and workarounds especially if you start pushing it (for example like whatsapp did). Also I've heard dirty_write is not safe with replicated tables. I don't know if this is true but is holding me back from using dirty transactions rather than transactions (which I don't need apart from safety on replicated tables). If anyone knows about this one please let me know. Feel free to add/remove/correct items:) Cheers, Martin On 13 October 2015 at 13:08, Steve Davis wrote: > Hi, > > I?m almost ashamed to say that it?s taken me over 5 years to come around to understanding the value of mnesia. > > I bought the ?well-known? negatives too fast. I have explored relational connectors, DHT solutions and a number of other approaches... > > It?s dawned on me *finally* that 90% of the time, a well implemented mnesia solution would have been better, faster, and cheaper. > > Did I mention "better"? > > Has anyone else had this experience? > > /s > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From zxq9@REDACTED Wed Oct 14 02:03:20 2015 From: zxq9@REDACTED (zxq9) Date: Wed, 14 Oct 2015 09:03:20 +0900 Subject: [erlang-questions] Mnesia In-Reply-To: <1444758374.470121944@apps.rackspace.com> References:

<1444758374.470121944@apps.rackspace.com> Message-ID: <2643813.LQAQ6q5oQX@changa> On 2015?10?13? ??? 13:46:14 lloyd@REDACTED wrote: > Hello, > > During a break at the Chicago Erlang Workshop Fred Hebert mentioned that Mnesia is a fine database for the 1990s. > > Mnesia was a major draw when I was first attracted to Erlang in part because I hate transiting syntactic boundaries when I'm programming. > > I asked Fred what it would it take to upgrade Mnesia for the 21st century (or, at least, for the next decade). He didn't know. > > Martin's list of negatives may be a good place to start. > > So, just how much effort and knowledge would it take to overcome or ameliorate these and other unspecified negatives of Mnesia to produce Mnesia2? As with any view, perspective is what drives the perception of utility. If we assume that the only thing that matters is web applications or server-oriented architectures where every potential user in the world is going to be pounding the crap out of a central data center or, even worse, a floating set of vaguely provisioned services within something like AWS, then sure. Maybe Mnesia isn't the thing for that, and certainly a LOT of development is focused on that case. Most of that development is iterative rehashes of what everyone else has already done, though. This is a red flag to me. Social networks, messaging systems, click analytics, GIS frameworks for click analytics, and lots of other things called "analytics" but really boil down to adtech, adtech, adtech, adtech, and some more adtech, and marketing campaigns about campaigns about adtech. The elephant in the room being that the ad market is falling apart because it isn't worth anything near like what we imagined. If views were what mattered it would be impossible for Twitter to be in such trouble, Arstechnica be struggling to maintain its editorial standards, or sites like Huffington Post and Forbes to be forced to display clickbait panels and load 40 ~ 70 third party plants per page load. For this case -- the forefront of tech buzz and where a huge, and very publicly visible percentage of investment and development effort go -- maybe Mnesia is not the best tool. But for other use cases it is immensely useful, *especially* as a live cache with db features. Small and medium business represents about ~ 90% of the activity in many economies. Here in Japan the figure floats around 92%; in the US it is something comparable. When small businesses latch on to a solution that *actually* helps them become more efficient (instead of shelling out for yet another upgrade to a spreadsheet program that will be used to do exactly what they would have done with a spreadsheet in the 80's -- or even funnier, for a version of that spreadsheet that still does the same things, but slower, in a browser), and that solution becomes widespread enough to have an actual impact *the entire economy* improves without anyone really noticing why. That is AMAZING. This is much more interesting than chasing click statistics in the interest of brokering ad sales at the speed of light so that users can continue to ignore them. I mention the SMB use case because it is one with which I am familiar. Fred holds his views because of the sector he deals with. Mnesia is by no means the *only* database useful in developing for the SMB market, but it is a profoundly useful tool among several necessary to make cross-platform development for SMBs a non-suicidal task. A hilariously low percentage of tech investment targets the SMB market (aiming instead at huge business, the web, and the consumer electronics tie-in (the only interesting part of the list)), and there are accordingly very few tools available to make development for that market reasonable. [Insert rant about the deleterious effects of a legally enforced monopoly on small business software.] So is mnesia a db for the 1990s? Maybe. I don't know. That is a very broad (and totally unqualified) statement to make, so it is hard to argue about. I suppose Postgres, Oracle, DB2 and anything else with a schema fall in the same category, but last I heard those were pretty darn useful for a wide variety of very real problems people experienced in the 1990s and continue to experience today. What I can say for certain is that I continue to find Mnesia to be of profound utility today, in 2015, whether or not it is "a good database for the 1990s". The *variety* of data I have dealt with in business is so broad that no single database paradigm, and certainly no particular database system, can manage it. Not reasonably, anyway. I'm not aware of any other systems that compare very well with Mnesia in functionality, especially interoperating with Erlang (or almost any language/runtime) the same way, so its hard to compare. That leaves Mnesia in the "unique tool" category, as opposed to the "easy to compare within a commodity market of similar alternatives" category. -Craig From lloyd@REDACTED Wed Oct 14 02:28:38 2015 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Tue, 13 Oct 2015 20:28:38 -0400 Subject: [erlang-questions] Mnesia In-Reply-To: <2643813.LQAQ6q5oQX@changa> References:

<1444758374.470121944@apps.rackspace.com> <2643813.LQAQ6q5oQX@changa> Message-ID: <8CB0B8A6-842C-4B01-ADBF-8F7F3DABA3B9@writersglen.com> Hi Craig, Among the most wise things I've heard on the topic so far. How can we get a crisp summary of your points and implications of Martin's negatives high up in the official mnesia docs. (I really don't know the process.) Would have saved me many hours of uncertainty. All the best, Lloyd Sent from my iPad > On Oct 13, 2015, at 8:03 PM, zxq9 wrote: > >> On 2015?10?13? ??? 13:46:14 lloyd@REDACTED wrote: >> Hello, >> >> During a break at the Chicago Erlang Workshop Fred Hebert mentioned that Mnesia is a fine database for the 1990s. >> >> Mnesia was a major draw when I was first attracted to Erlang in part because I hate transiting syntactic boundaries when I'm programming. >> >> I asked Fred what it would it take to upgrade Mnesia for the 21st century (or, at least, for the next decade). He didn't know. >> >> Martin's list of negatives may be a good place to start. >> >> So, just how much effort and knowledge would it take to overcome or ameliorate these and other unspecified negatives of Mnesia to produce Mnesia2? > > As with any view, perspective is what drives the perception of utility. > > If we assume that the only thing that matters is web applications or server-oriented architectures where every potential user in the world is going to be pounding the crap out of a central data center or, even worse, a floating set of vaguely provisioned services within something like AWS, then sure. Maybe Mnesia isn't the thing for that, and certainly a LOT of development is focused on that case. > > Most of that development is iterative rehashes of what everyone else has already done, though. This is a red flag to me. Social networks, messaging systems, click analytics, GIS frameworks for click analytics, and lots of other things called "analytics" but really boil down to adtech, adtech, adtech, adtech, and some more adtech, and marketing campaigns about campaigns about adtech. > > The elephant in the room being that the ad market is falling apart because it isn't worth anything near like what we imagined. If views were what mattered it would be impossible for Twitter to be in such trouble, Arstechnica be struggling to maintain its editorial standards, or sites like Huffington Post and Forbes to be forced to display clickbait panels and load 40 ~ 70 third party plants per page load. > > For this case -- the forefront of tech buzz and where a huge, and very publicly visible percentage of investment and development effort go -- maybe Mnesia is not the best tool. > > But for other use cases it is immensely useful, *especially* as a live cache with db features. Small and medium business represents about ~ 90% of the activity in many economies. Here in Japan the figure floats around 92%; in the US it is something comparable. When small businesses latch on to a solution that *actually* helps them become more efficient (instead of shelling out for yet another upgrade to a spreadsheet program that will be used to do exactly what they would have done with a spreadsheet in the 80's -- or even funnier, for a version of that spreadsheet that still does the same things, but slower, in a browser), and that solution becomes widespread enough to have an actual impact *the entire economy* improves without anyone really noticing why. > > That is AMAZING. This is much more interesting than chasing click statistics in the interest of brokering ad sales at the speed of light so that users can continue to ignore them. > > I mention the SMB use case because it is one with which I am familiar. Fred holds his views because of the sector he deals with. Mnesia is by no means the *only* database useful in developing for the SMB market, but it is a profoundly useful tool among several necessary to make cross-platform development for SMBs a non-suicidal task. A hilariously low percentage of tech investment targets the SMB market (aiming instead at huge business, the web, and the consumer electronics tie-in (the only interesting part of the list)), and there are accordingly very few tools available to make development for that market reasonable. > > [Insert rant about the deleterious effects of a legally enforced monopoly on small business software.] > > So is mnesia a db for the 1990s? Maybe. I don't know. That is a very broad (and totally unqualified) statement to make, so it is hard to argue about. I suppose Postgres, Oracle, DB2 and anything else with a schema fall in the same category, but last I heard those were pretty darn useful for a wide variety of very real problems people experienced in the 1990s and continue to experience today. What I can say for certain is that I continue to find Mnesia to be of profound utility today, in 2015, whether or not it is "a good database for the 1990s". The *variety* of data I have dealt with in business is so broad that no single database paradigm, and certainly no particular database system, can manage it. Not reasonably, anyway. I'm not aware of any other systems that compare very well with Mnesia in functionality, especially interoperating with Erlang (or almost any language/runtime) the same way, so its hard to compare. That leaves Mnesia in the "unique tool" category, as opposed to the "easy to compare within a commodity market of similar alternatives" category. > > -Craig > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ok@REDACTED Wed Oct 14 02:58:16 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 14 Oct 2015 13:58:16 +1300 Subject: [erlang-questions] Coming Back (maybe improving lists:reverse/1) In-Reply-To: <003301d105c5$2767b820$76372860$@frcuba.co.cu> References: <041e01d1012f$9220aa40$b661fec0$@frcuba.co.cu> <001001d10152$7e80a080$7b81e180$@frcuba.co.cu> <3DE96CA6-56A4-4996-AEC8-9377749DA20C@cs.otago.ac.nz> <001c01d1015d$c36cf8a0$4a46e9e0$@frcuba.co.cu> <001d01d1015f$24b92790$6e2b76b0$@frcuba.co.cu> <656993BD-5652-4E90-B872-2D2D11876952@cs.otago.ac.nz> <001801d10244$4095dd10$c1c19730$@frcuba.co.cu> <003301d105c5$2767b820$76372860$@frcuba.co.cu> Message-ID: <79D9E2E6-AAF4-4B68-AC7D-6F3BDE080112@cs.otago.ac.nz> On 14/10/2015, at 3:40 am, Ivan Carmenates Garcia wrote: > Hi Richard, > > Yes I will definably like to build queries like list comprehensions or like > your nice example, but I don't understand so well parse transform because I > didn't find so much documentation about it in Erlang doc, I was trying with > qlc but yet nothing I could archive. The basic idea is that the standard tools parse a module to get an abstract syntax tree for the whole module, hand it to your code to transform, and continue with whatever you give back. The source code does NOT have to pass semantic checks; the result of transforming does. http://www.erlang-factory.com/upload/presentations/521/yrashk_parse_transformations_sf12.pdf erl_id_trans is an example module that does nothing: http://www.erlang.org/doc/man/erl_id_trans.html It's worth noting that you can use an identity parse transform as a way of plugging in a static check of some kind that is not already done by the compiler. The warnings in that are a little strong. Parse transformations are not something to do *lightly*, but for implementing a query language as an Embedded Domain Specific Language they are a good choice. The Stack Overflow query "Is there a good, complete tutorial on Erlang parse transforms available?" http://stackoverflow.com/questions/2416192/is-there-a-good-complete-tutorial-on-erlang-parse-transforms-available has some useful answers. http://seancribbs.com/tech/2009/06/21/building-a-parser-generator-in-erlang-part-4/ talks about parse transforms in the context of building a PEG parser generator for Erlang. I liked it, except that it is a *perfect* example of why syntax colouring is **evil**. The official definition of abstract syntax trees is the documentation for the erl_syntax module: http://erlang.org/doc/man/erl_syntax.html Justin Calleja has a nice introduction to that: http://www.ricston.com/blog/erlang-erlsyntax/ Parse transforms are an extremely powerful tool that you want to use *sparingly*; only when it will improve the *total* maintainability of your program. I haven't used them in anger yet in Erlang myself. I've used the equivalent in other languages to do things like embedding XML syntax in a host language and generate specialised code from a template. (The effort of trying to take three versions of the "same" thing and boil them down to a single template from which the originals can be recovered is, sadly, a good way to discover *unintentional* differences between the versions, which is why I think maintaining a master copy and generating specialised versions from it is a good thing for overall maintainability.) There are at least four ways in which a parse transform can help with an embedded query language: - queries can be much easier for people to read and write, so the queries as such can be more maintainable - what the translation *is* can be varied without the source code of any query having to be changed - the translation is done at compile time, not run time which is good for efficiency *and* timeliness of error reporting - the temptation to leave a sanity check out on the grounds that it will cost too much at run time goes away because you know the check will be done at compile time. *Can* help is not *will* help. But it's worth exploring. From ok@REDACTED Wed Oct 14 03:16:38 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 14 Oct 2015 14:16:38 +1300 Subject: [erlang-questions] Mnesia In-Reply-To: <1444758374.470121944@apps.rackspace.com> References:

<1444758374.470121944@apps.rackspace.com> Message-ID: On 14/10/2015, at 6:46 am, wrote: > > I asked Fred what it would it take to upgrade Mnesia for the 21st century (or, at least, for the next decade). He didn't know. There's one thing that strikes me. I could go to a shop today and buy a 1 TB external drive for NZD 75, including Goods and Services Tax of 15%. (At least that's what the ad I saw a couple of days ago said.) That's almost exactly USD 50. This is a drive that fits in a shirt pocket, with room left over for all sorts of junk. To make Mnesia a data base for the 2010s, it has to be able to handle at least 1TB of data. Heck, I've got enough goodies-for- research money left that I could get the department to buy me ten of these gadgets, so let's say Mnesia - should be able to handle a single table in the low TB - should be able to handle a collection of tables in the tens of TB - where "handle" includes creating, populating, checking, recovering, and accessing in "a reasonable time". That's a "single machine data base for the 2010s". Of course there are multicore, cluster, and network issues as well. From lloyd@REDACTED Wed Oct 14 05:19:56 2015 From: lloyd@REDACTED (Lloyd R. Prentice) Date: Tue, 13 Oct 2015 23:19:56 -0400 Subject: [erlang-questions] Mnesia In-Reply-To: References:

<1444758374.470121944@apps.rackspace.com> Message-ID: Hi Richard, So if we dig into the code, what exactly needs to change to make that happen? This is a great teachable moment where the wizards of Erlang can help us with less understanding significantly advance our Erlang skills--- if nothing else, guide us through the design decisions that shaped mnesia, the architecture, and the significant code passages that impose current limitations. Why? To broaden the base of folks capable of extending and advancing the Erlang legacy. All the best, Lloyd Sent from my iPad > On Oct 13, 2015, at 9:16 PM, "Richard A. O'Keefe" wrote: > > >> On 14/10/2015, at 6:46 am, wrote: >> >> I asked Fred what it would it take to upgrade Mnesia for the 21st century (or, at least, for the next decade). He didn't know. > > There's one thing that strikes me. > > I could go to a shop today and buy a 1 TB external drive for > NZD 75, including Goods and Services Tax of 15%. (At least > that's what the ad I saw a couple of days ago said.) > That's almost exactly USD 50. This is a drive that fits in > a shirt pocket, with room left over for all sorts of junk. > > To make Mnesia a data base for the 2010s, it has to be able to > handle at least 1TB of data. Heck, I've got enough goodies-for- > research money left that I could get the department to buy me > ten of these gadgets, so let's say Mnesia > - should be able to handle a single table in the low TB > - should be able to handle a collection of tables in the > tens of TB > - where "handle" includes creating, populating, checking, > recovering, and accessing in "a reasonable time". > > That's a "single machine data base for the 2010s". > Of course there are multicore, cluster, and network issues as > well. > > > From ok@REDACTED Wed Oct 14 07:24:12 2015 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 14 Oct 2015 18:24:12 +1300 Subject: [erlang-questions] Mnesia In-Reply-To: References:

<1444758374.470121944@apps.rackspace.com>

Message-ID: <76B2A53E-914A-4C15-A116-536CC57E7AAB@cs.otago.ac.nz> On 14/10/2015, at 4:19 pm, Lloyd R. Prentice wrote: > Hi Richard, > > So if we dig into the code, what exactly needs to change to make that happen? How should I know? I couldn't implement a data base to save my life. It will certainly be more than just finding the place where it says "2GB" and changing a number. Things *change* at scale. I've got a data set that's 18GB as raw text, and a student wants to do some data mining on a recent data set that's too big to fit on the 8GB memory stick he keeps bringing me excerpts on; it would be a good fit for Mnesia... mnesia_ext + LevelDB looks really good; it would be nice to know, on downloading a new release of Erlang/OTP, that it would be _there_ and *the documentation integrated*. By the way, a key fact about why Mnesia is the way it is can be found in the documentation: Mnesia is primarily intended to be a memory-resident database. Some of its design tradeoffs reflect this. But on a 16GiB 64-bit machine (yeah, I know it's small, but it's a couple of years old) a 32-bit limit doesn't make as much sense for a memory-resident data base as it used to either. From kennethlakin@REDACTED Wed Oct 14 08:55:23 2015 From: kennethlakin@REDACTED (Kenneth Lakin) Date: Tue, 13 Oct 2015 23:55:23 -0700 Subject: [erlang-questions] Mnesia In-Reply-To: <76B2A53E-914A-4C15-A116-536CC57E7AAB@cs.otago.ac.nz> References:

<1444758374.470121944@apps.rackspace.com>

<76B2A53E-914A-4C15-A116-536CC57E7AAB@cs.otago.ac.nz> Message-ID: <561DFC5B.9030705@gmail.com> On 10/13/2015 10:24 PM, Richard A. O'Keefe wrote: > But on a 16GiB 64-bit machine ... a 32-bit limit doesn't > make as much sense for a memory-resident data base as it > used to either. I'm slightly confused. As I understood it, the size of a disc_copies Mnesia table was only limited by the system's available RAM, and one didn't need to worry about the size of the on-disk representation of the data. Did I misunderstand this, or were you referring to DETS-backed disc_only_copies tables? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From jan.chochol@REDACTED Wed Oct 14 09:05:42 2015 From: jan.chochol@REDACTED (Jan Chochol) Date: Wed, 14 Oct 2015 09:05:42 +0200 Subject: [erlang-questions] unpacking big/little endian speed difference In-Reply-To: <561D15A9.70104@cs.ntua.gr> References: <561D15A9.70104@cs.ntua.gr> Message-ID: The difference is really in "i_bs_get_integer_32_rfId" and "i_bs_get_integer_small_imm_xIfId". When looking in code, the "i_bs_get_integer_32_rfId" can extract 32 bit big endian integer from bitstring - it is quite simple function, which can not do much more, and even has fast path for binary (bitstring with byte boundary). On the other side "i_bs_get_integer_small_imm_xIfId" is quite complex function, which can extract integer with any bit size, signed, or unsigned, little, or big endian. It is quite complex, and even use some temporary allocated buffers. So the reason is, that there is special "fast" opcode for extracting 32 bit big endian integer (probably because authors of BEAM VM thought, that it will be more useful). On Tue, Oct 13, 2015 at 4:31 PM, Kostis Sagonas wrote: > On 10/13/2015 02:59 PM, Sergej Jure?ko wrote: > >> How come unpacking integers as big endian is faster than little endian, >> when I'm running on a little endian platform (intel osx)? >> I know big endian is erlang default, but if you're turning binaries into >> integers, you must turn them into little endian anyway, so unpacking >> should basically be memmcpy. >> >> The test: >> >> % Takes: ~1.3s >> -define(ENDIAN,unsigned-big). >> >> % Takes: ~1.8s >> % -define(ENDIAN,unsigned-little). >> > > Well, if you are interested in performance, simply compile to native code > your module and the time will drop to less than half... > > ================================================================ > Eshell V7.1 (abort with ^G) > 1> c(endian). > {ok,endian} > 2> endian:loop(). > 1097971 > 3> endian:loop(). > 1327621 > 4> endian:loop(). > 1162506 > 5> endian:loop(). > 1103813 > 6> c(endian, [native]). > {ok,endian} > 7> endian:loop(). > 414000 > 8> endian:loop(). > 441327 > ================================================================ > > Kostis > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gguthrie@REDACTED Wed Oct 14 09:25:33 2015 From: gguthrie@REDACTED (Gordon Guthrie) Date: Wed, 14 Oct 2015 08:25:33 +0100 Subject: [erlang-questions] Mnesia In-Reply-To: References:

<1444758374.470121944@apps.rackspace.com> Message-ID: <3880B774-DB69-487A-8BD7-6DA3E8DE6766@basho.com> Mnesia is also a pain when it partitions - you have to write your own reconciliation programmes. There has been a lot of work done on eventual consistency in *the other* Erlang database - Riak (disclaimer I am working at Basho now) Riak implements eventual consistency - and post-partition self-healing using Consistent Replicated Data Types (or CRDTs) and the canonical set of standalone CRDT libraries is written in Erlang: https://github.com/basho/riak_dt There is a comprehensive reading list here: https://christophermeiklejohn.com/crdt/2014/07/22/readings-in-crdts.html The combination of using Klarna?s (forthcoming) leveldb backend and a CRDT eventual consistency layer on top would be an interesting start offering a distributed transactional database with eventual consistency Gordon > On 14 Oct 2015, at 02:16, Richard A. O'Keefe wrote: > > > On 14/10/2015, at 6:46 am, wrote: >> >> I asked Fred what it would it take to upgrade Mnesia for the 21st century (or, at least, for the next decade). He didn't know. > > There's one thing that strikes me. > > I could go to a shop today and buy a 1 TB external drive for > NZD 75, including Goods and Services Tax of 15%. (At least > that's what the ad I saw a couple of days ago said.) > That's almost exactly USD 50. This is a drive that fits in > a shirt pocket, with room left over for all sorts of junk. > > To make Mnesia a data base for the 2010s, it has to be able to > handle at least 1TB of data. Heck, I've got enough goodies-for- > research money left that I could get the department to buy me > ten of these gadgets, so let's say Mnesia > - should be able to handle a single table in the low TB > - should be able to handle a collection of tables in the > tens of TB > - where "handle" includes creating, populating, checking, > recovering, and accessing in "a reasonable time". > > That's a "single machine data base for the 2010s". > Of course there are multicore, cluster, and network issues as > well. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From rizkhan@REDACTED Wed Oct 14 09:56:07 2015 From: rizkhan@REDACTED (Rizwan Khan) Date: Wed, 14 Oct 2015 12:56:07 +0500 Subject: [erlang-questions] Erlang in the Insurance Sector Message-ID: Hi, Wanted to know if Erlang is being used in the Insurance sector anywhere? I have seem multiple platforms which are based on Java. Some of them still use COBOL. Most of them follow SOA practices and also include the business and calculation rules engines. I see Erlang/Elixir as a good fit for Microservices based architecture. How useful Erlang/Elixir stack is going to be for the underwriting purpose which is the main component for Insurance companies? Rizwan Khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From lcastro@REDACTED Wed Oct 14 10:27:06 2015 From: lcastro@REDACTED (Laura M. Castro) Date: Wed, 14 Oct 2015 10:27:06 +0200 Subject: [erlang-questions] Erlang in the Insurance Sector In-Reply-To: References: Message-ID: A few years ago, we did develop a system for risk management (including not only incident report, but best-fitting insurance policy determination, and also payment lifecycle processing), with an Erlang back-end and a Java front-end. As one can imagine, pattern-matching proved very useful in the kind of logic implementation that was needed: the use case was that multiple insurance policies could have been contracted that could possible be used for covering a given incident, and a number of user-defined criteria could be used to determine with one to use. It started as a proof-of-concept but ended up growing into a fully functional enterprise system that was used by a international corporation for a number of years; I don't know which is the situation of the system nowadays. We published a number of papers around it, you can check them out: http://dl.acm.org/citation.cfm?id=940884 http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=1562879 http://www.tandfonline.com/doi/abs/10.3166/jds.17.501-521 (if you don't have access to any of those, just let me know) From carlsson.richard@REDACTED Wed Oct 14 10:46:33 2015 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 14 Oct 2015 10:46:33 +0200 Subject: [erlang-questions] Mnesia In-Reply-To: <3880B774-DB69-487A-8BD7-6DA3E8DE6766@basho.com> References:

<1444758374.470121944@apps.rackspace.com> <3880B774-DB69-487A-8BD7-6DA3E8DE6766@basho.com> Message-ID: The simple view of Mnesia is that it's a transaction layer on top of ETS tables, with some varying forms of backing storage. Transactions are pessimistic, based on locking. Some small optimizations have been done to the locking mechanism in recent years, but it's maybe more could be done in that area. It might also be possible to add built-in support for optimistic transactions based on timestamps or compare-and-swap, for certain usage patterns. The semantics of dirty reads/writes need to be better documented, and if possible cleaned up a bit, because the behaviour can depend on table type and whether or not the tables are local or remote. The table size problems can probably be considered to be solved by mnesia_ext with leveldb or other backends. None of this will make it less of a 90's database though. Adding eventual consistency (e.g. based on vector clocks) as alternative to transactions would make it more modern. The big thing that would help, as Gordon mentioned, is a new distribution/replication layer. The existing one basically assumes that tables are not huge and the network between nodes is fast and reliable with netsplits being rare, like in a small cluster in a telecom base station. We use Mnesia, but not in distributed mode - we have a custom distribution layer (stable, but very limited) on top of local Mnesia instances that are not directly aware of each other. /Richard /Richard On Wed, Oct 14, 2015 at 9:25 AM, Gordon Guthrie wrote: > Mnesia is also a pain when it partitions - you have to write your own > reconciliation programmes. > > There has been a lot of work done on eventual consistency in *the other* > Erlang database - Riak (disclaimer I am working at Basho now) > > Riak implements eventual consistency - and post-partition self-healing > using Consistent Replicated Data Types (or CRDTs) and the canonical set of > standalone CRDT libraries is written in Erlang: > https://github.com/basho/riak_dt > > There is a comprehensive reading list here: > https://christophermeiklejohn.com/crdt/2014/07/22/readings-in-crdts.html > > The combination of using Klarna?s (forthcoming) leveldb backend and a CRDT > eventual consistency layer on top would be an interesting start offering a > distributed transactional database with eventual consistency > > Gordon > > On 14 Oct 2015, at 02:16, Richard A. O'Keefe wrote: > > > On 14/10/2015, at 6:46 am, > wrote: > > > I asked Fred what it would it take to upgrade Mnesia for the 21st century > (or, at least, for the next decade). He didn't know. > > > There's one thing that strikes me. > > I could go to a shop today and buy a 1 TB external drive for > NZD 75, including Goods and Services Tax of 15%. (At least > that's what the ad I saw a couple of days ago said.) > That's almost exactly USD 50. This is a drive that fits in > a shirt pocket, with room left over for all sorts of junk. > > To make Mnesia a data base for the 2010s, it has to be able to > handle at least 1TB of data. Heck, I've got enough goodies-for- > research money left that I could get the department to buy me > ten of these gadgets, so let's say Mnesia > - should be able to handle a single table in the low TB > - should be able to handle a collection of tables in the > tens of TB > - where "handle" includes creating, populating, checking, > recovering, and accessing in "a reasonable time". > > That's a "single machine data base for the 2010s". > Of course there are multicore, cluster, and network issues as > well. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zxq9@REDACTED Wed Oct 14 11:02:05 2015 From: zxq9@REDACTED (zxq9) Date: Wed, 14 Oct 2015 18:02:05 +0900 Subject: [erlang-questions] Erlang in the Insurance Sector In-Reply-To: References: Message-ID: <13137327.OLXvpmNeDx@burrito> On Wednesday 14 October 2015 10:27:06 Laura M. Castro wrote: > http://dl.acm.org/citation.cfm?id=940884 > http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=1562879 > http://www.tandfonline.com/doi/abs/10.3166/jds.17.501-521 > > (if you don't have access to any of those, just let me know) I am very interested, but these are all behind rather steep paywalls. -Craig From vances@REDACTED Wed Oct 14 11:29:53 2015 From: vances@REDACTED (Vance Shipley) Date: Wed, 14 Oct 2015 14:59:53 +0530 Subject: [erlang-questions] Mnesia In-Reply-To: References:

<1444758374.470121944@apps.rackspace.com> <3880B774-DB69-487A-8BD7-6DA3E8DE6766@basho.com> Message-ID: On Wed, Oct 14, 2015 at 2:16 PM, Richard Carlsson wrote: > The simple view of Mnesia is that it's a transaction layer on top of ETS tables, with some varying forms of backing storage. I am careful not to use the term "database" when referring to mnesia as it leads to unfair comparisons and unreasonable expectations. I use terms such as "distributed tables" or "data store". It's a more low level tool than what most people think of as a "database". In training I describe a progression in scale out like this: single process - store data in StateData (i.e. gb_trees) multiple processes, single node - store data in ets tables multiple processes, multiple nodes - store data in mnesia ... and that's before we talk about persistence or transactions. -- -Vance From henrik.x.nord@REDACTED Wed Oct 14 12:45:58 2015 From: henrik.x.nord@REDACTED (Henrik Nord X) Date: Wed, 14 Oct 2015 12:45:58 +0200 Subject: [erlang-questions] Patch Package OTP 18.1.2 Released Message-ID: <561E3266.9000601@ericsson.com> Patch Package: OTP 18.1.2 Git Tag: OTP-18.1.2 Date: 2015-10-14 Trouble Report Id: OTP-13036 Seq num: System: OTP Release: 18 Application: ssh-4.1.1 Predecessor: OTP 18.1.1 Check out the git tag OTP-18.1.2, and build a full OTP system including documentation. Apply one or more applications from this build as patches to your installation using the 'otp_patch_apply' tool. For information on install requirements, see descriptions for each application version below. --------------------------------------------------------------------- --- ssh-4.1.1 ------------------------------------------------------- --------------------------------------------------------------------- The ssh-4.1.1 application can be applied independently of other applications on a full OTP 18 installation. --- Improvements and New Features --- OTP-13036 Application(s): ssh A new option max_channels limits the number of channels with active server-side subsystems that are accepted. Full runtime dependencies of ssh-4.1.1: crypto-3.3, erts-6.0, kernel-3.0, public_key-0.22, stdlib-2.3 --------------------------------------------------------------------- --------------------------------------------------------------------- --------------------------------------------------------------------- From gguthrie@REDACTED Wed Oct 14 12:48:27 2015 From: gguthrie@REDACTED (Gordon Guthrie) Date: Wed, 14 Oct 2015 11:48:27 +0100 Subject: [erlang-questions] Mnesia In-Reply-To: References:

<1444758374.470121944@apps.rackspace.com> <3880B774-DB69-487A-8BD7-6DA3E8DE6766@basho.com> Message-ID: <15F12F95-C171-4BCD-A3FB-3B365DD9A05E@basho.com> I always planned reliability for Mnesia as hot/cold A shared resilient network storage fabric underneath both a running Mnesia instance and a non-running one. Failover would then be losing a server and bringing up the other one. Obviously this can be expensive with large tables that need to be loaded fully into memory, etc, etc? Gordon > On 14 Oct 2015, at 09:46, Richard Carlsson wrote: > > The simple view of Mnesia is that it's a transaction layer on top of ETS tables, with some varying forms of backing storage. Transactions are pessimistic, based on locking. Some small optimizations have been done to the locking mechanism in recent years, but it's maybe more could be done in that area. It might also be possible to add built-in support for optimistic transactions based on timestamps or compare-and-swap, for certain usage patterns. The semantics of dirty reads/writes need to be better documented, and if possible cleaned up a bit, because the behaviour can depend on table type and whether or not the tables are local or remote. The table size problems can probably be considered to be solved by mnesia_ext with leveldb or other backends. None of this will make it less of a 90's database though. Adding eventual consistency (e.g. based on vector clocks) as alternative to transactions would make it more modern. > > The big thing that would help, as Gordon mentioned, is a new distribution/replication layer. The existing one basically assumes that tables are not huge and the network between nodes is fast and reliable with netsplits being rare, like in a small cluster in a telecom base station. We use Mnesia, but not in distributed mode - we have a custom distribution layer (stable, but very limited) on top of local Mnesia instances that are not directly aware of each other. > > /Richard > > > > /Richard > > On Wed, Oct 14, 2015 at 9:25 AM, Gordon Guthrie > wrote: > Mnesia is also a pain when it partitions - you have to write your own reconciliation programmes. > > There has been a lot of work done on eventual consistency in *the other* Erlang database - Riak (disclaimer I am working at Basho now) > > Riak implements eventual consistency - and post-partition self-healing using Consistent Replicated Data Types (or CRDTs) and the canonical set of standalone CRDT libraries is written in Erlang: > https://github.com/basho/riak_dt > > There is a comprehensive reading list here: > https://christophermeiklejohn.com/crdt/2014/07/22/readings-in-crdts.html > > The combination of using Klarna?s (forthcoming) leveldb backend and a CRDT eventual consistency layer on top would be an interesting start offering a distributed transactional database with eventual consistency > > Gordon > >> On 14 Oct 2015, at 02:16, Richard A. O'Keefe > wrote: >> >> >> On 14/10/2015, at 6:46 am, > > wrote: >>> >>> I asked Fred what it would it take to upgrade Mnesia for the 21st century (or, at least, for the next decade). He didn't know. >> >> There's one thing that strikes me. >> >> I could go to a shop today and buy a 1 TB external drive for >> NZD 75, including Goods and Services Tax of 15%. (At least >> that's what the ad I saw a couple of days ago said.) >> That's almost exactly USD 50. This is a drive that fits in >> a shirt pocket, with room left over for all sorts of junk. >> >> To make Mnesia a data base for the 2010s, it has to be able to >> handle at least 1TB of data. Heck, I've got enough goodies-for- >> research money left that I could get the department to buy me >> ten of these gadgets, so let's say Mnesia >> - should be able to handle a single table in the low TB >> - should be able to handle a collection of tables in the >> tens of TB >> - where "handle" includes creating, populating, checking, >> recovering, and accessing in "a reasonable time". >> >> That's a "single machine data base for the 2010s". >> Of course there are multicore, cluster, and network issues as >> well. >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gpadovani@REDACTED Wed Oct 14 13:06:04 2015 From: gpadovani@REDACTED (Gianluca Padovani) Date: Wed, 14 Oct 2015 13:06:04 +0200 Subject: [erlang-questions] Delete key doesn't work in erl/iex shell In-Reply-To: References:

Message-ID: Thank you :-) 2015-10-13 19:43 GMT+02:00 Pierre Fenoll : > Just wait a little :) > https://github.com/erlang/otp/commit/2c7e387961251af59f14bca39cbf8fbbe880383e > > > Cheers, > -- > Pierre Fenoll > > > On 13 October 2015 at 09:03, Gianluca Padovani > wrote: > >> Hi, >> I'm using Ubuntu 14.04 and in Elixir and Erlang shell the delete key >> doesn't work. I found this issue[1] on elixir repo but they suggest to fix >> this problem on erlang shell. Do you consider this issue a bug? Is it >> possible to fix it or find a workaround? >> >> Thank you very much >> Gianluca >> >> [1] https://github.com/elixir-lang/elixir/issues/3471 >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Wed Oct 14 13:31:44 2015 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 14 Oct 2015 13:31:44 +0200 Subject: [erlang-questions] New pull request for mnesia_ext Message-ID: I have created a new pull request for mnesia_ext, based on the fresh OTP 18.1.2 tag: https://github.com/erlang/otp/pull/858 /Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From magnus@REDACTED Wed Oct 14 14:02:42 2015 From: magnus@REDACTED (Magnus Henoch) Date: Wed, 14 Oct 2015 13:02:42 +0100 Subject: [erlang-questions] Mnesia In-Reply-To: Lloyd R. Prentice's message of "Tue\, 13 Oct 2015 20\:28\:38 -0400 $11 hours\, 25 minutes\, 10 seconds ago$" Message-ID: "Lloyd R. Prentice" writes: > Hi Craig, > > Among the most wise things I've heard on the topic so far. > > How can we get a crisp summary of your points and implications > of Martin's negatives high up in the official mnesia docs. (I > really don't know the process.) Would have saved me many hours > of uncertainty. The process for adding something to the documentation is submitting a pull request to the erlang/otp repository on Github. The Mnesia documentation is in the lib/mnesia/doc/src directory. You could either clone the repository, edit the documentation in your favourite text editor, commit and push your changes, and open a pull request from your branch, or you could edit the documentation through the Github web interface and submit the pull request from there. Perhaps the Mnesia overview page would be a good place for this: https://github.com/erlang/otp/blob/maint/lib/mnesia/doc/src/Mnesia_overview.xml Click the pen icon near the top to start editing. Regards, Magnus From carlsson.richard@REDACTED Wed Oct 14 15:56:43 2015 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 14 Oct 2015 15:56:43 +0200 Subject: [erlang-questions] If you make WhatsApp today... In-Reply-To: <22044.46629.802283.964778@gargle.gargle.HOWL> References:

<1F5C5B03-1CAB-47DB-AFEB-306584201D10@feuerlabs.com> <1444655676.545392.407872329.7AC25FB7@webmail.messagingengine.com> <22044.46629.802283.964778@gargle.gargle.HOWL> Message-ID: The Postgres backend to mnesia_ext is now available here: https://github.com/klarna/mnesia_pg /Richard On Tue, Oct 13, 2015 at 9:43 AM, Mikael Pettersson

Welcome to App